Mutual understanding between driver and vehicle is critically important to the design of intelligent vehicles and customized interaction interface. In this study, a unified driver behavior reasoning system toward multi-scale and multi-tasks behavior recognition is proposed. Specifically, a multi-scale driver behavior recognition system is designed to recognize both the driver's physical and mental states based on a deep encoder-decoder framework. This system can jointly recognize three driver behaviors with different time scales based on the shared encoder network. Driver body postures and mental behaviors include intention and emotion are studied and identified. The encoder network is designed based on a deep convolutional neural network (CNN), and several decoders for different driver states estimation are proposed with fully connected (FC) and long short-term memory (LSTM) based recurrent neural networks (RNN). The joint feature learning with the CNN encoder increases the computational efficiency and feature diversity, while the customized decoders enable an efficient multi-tasks inference. The proposed framework can be used as a solution to exploit the relationship between different driver states, and it is found that when drivers generate lane change intentions, their emotions usually keep neutral state and more focus on the task. Two naturalistic datasets are used to investigate the model performance, which is a local highway dataset, namely, CranData and one public dataset from Brain4Cars. The testing results on these two datasets show accurate performance and outperform existing methods on driver postures, intention, and emotion recognition.