Detecting driver distraction is a significant concern for future intelligent transportation systems. We present a new approach for identifying distracted driving behavior by evaluating a stimulus and response interaction with the brain signals in two ways. First, measuring the driver response through EEG by creating various types of distraction stimuli such as reading, texting, calling and using phone camera (risk odds ratio of these activities determined by NHTSA study). Second, using a survey, comparing driver's order/perception of severity of distraction with the derived distraction index from EEG bands. A 14 electrodes headset was used to record the brain signals while driving in the pilot study with two subjects and a single dry electrode headset with 13 subjects in the main study. We used a naturalistic driving study as opposed to a virtual reality driving simulator to perform the distracted driving maneuvers, consisting of over 100 short duration trials (three to five seconds) for a subject. We overcame a big challenge in EEG analysis - reducing the number of electrodes by isolating one electrode (FC5) from 14 electrode locations to identify certain distractions. Our machine learning methods achieved a mean accuracy (averaged over the subjects and tasks) of 91.54 +/- 5.23% to detect a distracted driving event and 76.99 +/- 8.63% to distinguish between the five distraction cases in our study (read, text, call, and snapshot) using a single electrode. The quantification of distracted driving detailed in this paper is necessary to guide future policies in road safety. Our system addresses the safety concerns resulting from driver distraction and aims to bring about behavioral changes in drivers.