It is known that the hazard ratio lacks a useful causal interpretation. Even for data from a randomized controlled trial, the hazard ratio suffers from built-in selection bias as, over time, the individuals at risk in the exposed and unexposed are no longer exchangeable. In this work, we formalize how the observed hazard ratio evolves and deviates from the causal hazard ratio of interest in the presence of heterogeneity of the hazard of unexposed individuals (frailty) and heterogeneity in effect (individual modification). For the case of effect heterogeneity, we define the causal hazard ratio. We show that the observed hazard ratio equals the ratio of expectations of the latent variables (frailty and modifier) conditionally on survival in the world with and without exposure, respectively. Examples with gamma, inverse Gaussian and compound Poisson distributed frailty, and categorical (harming, beneficial or neutral) effect modifiers are presented for illustration. This set of examples shows that an observed hazard ratio with a particular value can arise for all values of the causal hazard ratio. Therefore, the hazard ratio can not be used as a measure of the causal effect without making untestable assumptions, stressing the importance of using more appropriate estimands such as contrasts of the survival probabilities.