Understanding and mitigating drivers' negative emotions, stress levels, and anxiety is of high importance for decreasing accident rates, and enhancing road safety. While detecting drivers' stress and negative emotions can significantly help with this goal, understanding what might be associated with increases in drivers' negative emotions and high stress level, might better help with planning interventions. While studies have provided significant insight into detecting drivers' emotions and stress levels, not many studies focused on the reasons behind changes in stress levels and negative emotions. In this study, by using a naturalistic driving study database, we analyze the changes in the driving scene, including road objects and the dynamical relationship between the ego vehicle and the lead vehicle with respect to changes in drivers' psychophysiological metrics (i.e., heart rate (HR) and facial expressions). Our results indicate that different road objects might be associated with varying levels of increase in drivers' HR as well as different proportions of negative facial emotions detected through computer vision. Larger vehicles on the road, such as trucks and buses, are associated with the highest amount of increase in drivers' HR as well as negative emotions. Additionally, shorter distances and higher standard deviation in the distance to the lead vehicle are associated with a higher number of abrupt increases in drivers' HR, depicting a possible increase in stress level. Our finding indicates more positive emotions, lower facial engagement, and a lower abrupt increase in HR at a higher speed of driving, which often happens in highway environments. This research collectively shows that driving at higher speeds happening in highways by avoiding certain road objects might be a better fit for keeping drivers in a calmer, more positive state.