The accurate evaluation of the quality of driving behavior is crucial for optimizing and implementing autonomous driving technology in practice. However, there is no comprehensive understanding of good driving behaviors currently. In this paper, we sought to understand driving behaviors from the perspectives of both drivers and passengers. We invited 10 expert drivers and 14 novice drivers to complete a 5.7-kilometer urban road driving task. After the experiments, we conducted semi-structured interviews with 24 drivers and 48 of their passengers (two passengers per driver). Through the analysis of interview data, we found passengers' assessing logic of driving behaviors, divers' considerations and efforts to achieve good driving, and gaps between these perspectives. Our research provided insights into a systematic evaluation of autonomous driving and the design implications for future autonomous vehicles.