Popularity bias is a well-known issue in recommender systems where few popular items are over-represented in the input data, while majority of other less popular items are under-represented. This disparate representation often leads to bias in exposure given to the items in the recommendation results. Extensive research examined this bias from item perspective and attempted to mitigate it by enhancing the recommendation of less popular items. However, a recent research has revealed the impact of this bias on users. Users with different degree of tolerance toward popular items are not fairly served by the recommendation system: users interested in less popular items receive more popular items in their recommendations, while users interested in popular items are recommended what they want. This is mainly due to the popularity bias that popular items are over-recommended. In this paper, we aim at investigating the factors leading to this user-side unfairness of popularity bias in recommender systems. In particular, we investigate two factors: 1) the relationship between this unfairness and users' interest toward items' categories (e.g., movie genres), 2) the relationship between this unfairness and the diversity of the popularity group in users' profile (the degree to which the user is interested in items with different degree of popularity). Experiments on a movie recommendation dataset using multiple recommendation algorithms show that these two factors are significantly correlated with the degree of popularity unfairness in the recommendation results.