Hazard ratios are prone to selection bias, compromising their use as causal estimands. On the other hand, the hazard difference has been shown to remain unaffected by the selection of frailty factors over time. Therefore, observed hazard differences can be used as an unbiased estimator for the causal hazard differences in the absence of confounding. However, in the presence of effect (on the hazard) heterogeneity, the hazard difference is also affected by selection. In this work, we formalize how the observed hazard difference (from a randomized controlled trial) evolves by selecting favourable levels of effect modifiers in the exposed group and thus deviates from the causal hazard difference of interest. Such selection may result in a non-linear integrated hazard difference curve even when the individual causal effects are time-invariant. Therefore, a homogeneous time-varying causal additive effect on the hazard can not be distinguished from a constant but heterogeneous causal effect. We illustrate this causal issue by studying the effect of chemotherapy on the survival time of patients suffering from carcinoma of the oropharynx using data from a clinical trial. The hazard difference can thus not be used as an appropriate measure of the causal effect without making untestable assumptions.