The National Highway Traffic Safety Administration reported that the number of bicyclist fatalities has increased by more than 35% since 2010. One of the main reasons associated with cyclists' crashes is the adverse effect of high cognitive load due to distractions. However, very limited studies have evaluated the impact of secondary tasks on cognitive distraction during cycling. This study leverages an Immersive Virtual Environment (IVE) simulation environment to explore the effect of secondary tasks on cyclists' cognitive distraction through evaluating their behavioral and physiological responses. Specifically, by recruiting 75 participants, this study explores the effect of listening to music versus talking on the phone as a standardized secondary tasks on participants' behavior (i.e., speed, lane position, input power, head movement) as well as, physiological responses including participants' heart rate variability and skin conductance metrics. Our results show that (1) listening to high-tempo music can lead to a significantly higher speed, a lower standard deviation of speed, and higher input power. Additionally, the trend is more significant for cyclists who had a strong habit of daily music listening (> 4 hours/day). In the high cognitive workload situation (simulated hands-free phone talking), cyclists had a lower speed with less input power and less head movement variation. Our results indicate that participants' HRV (HF, pnni-50) and EDA features (numbers of SCR peaks) are sensitive to cyclists' cognitive load changes in the IVE simulator.