Cyberbullying and online harassment have serious negative psychological and emotional consequences for the victims, such as decreased life satisfaction, suicidal ideation, self-harming behaviors, depression, anxiety, and others. Most of the prior works assessed people's emotional responses via questionnaires, while social media platforms contain data that could provide valuable insights into users' emotions in real online discussions. Therefore, this data-driven study investigates the effect of toxicity on Twitter users' emotions and other factors associated with expressing anger, anxiety, and sadness in terms of account identifiability, activity, conversation structure, and conversation topic. To achieve this goal, we identified toxic replies in the large dataset consisting of 79,799 random Twitter conversations and obtained the emotions expressed in these conversations. Then, we performed propensity score matching and analyzed causal associations between toxicity and users' emotions. In general, we found that users receiving toxic replies are more likely to express emotions of anger, sadness, and anxiety compared to users who did not receive toxic replies. Finally, analysis results indicate that the conversation topic and users' account characteristics are likely to affect their emotional responses to toxicity. Our findings provide a better understanding of toxic replies' consequences on users' emotional states, which can potentially lead to developing personalized moderation methods that will help users emotionally cope with toxicity on social media.