Multi-task learning has shown large benefits in Natural Language Understanding (NLU). However, current state-of-the-arts (SOTAs) like MT-DNN and MMoE do not model task relationships explicitly and fail to obtain effective task alignment. In this paper, we propose a Collaborative Multi-Task Representation (CMTR) framework to tackle this problem. We capture instance-level task relations through a task interaction layer, which helps guide the fusion of task-oriented representations into the final representation. Moreover, tailored loss functions are proposed to facilitate the learning of task alignment. Specifically, we leverage knowledge distillation as an auxiliary loss to assist the adaptation layers in generating task-oriented representations. We also introduce a regularization loss to learn better gating functions for multi-task fusion. Empirically, CMTR outperforms SOTA multi-task learning frameworks on most natural language understanding tasks in the GLUE benchmark. Furthermore, it achieves better task alignment and demonstrates good interpretability.