Survival analysis starts with a question: how long until an event happens? A patient relapses, a customer churns, a borrower defaults on a loan, a prisoner is rearrested. Parametric models answer by assuming a shape for the hazard — Weibull, log-logistic, exponential — and estimating its parameters. The Cox model sidesteps the entire question. You get hazard ratios, survival curves, and covariate effects without ever specifying what the baseline hazard looks like.
In our Bayesian survival analysis post, we used PyMC to fit an accelerated failure time model with explicit distributional assumptions. The Cox model takes the opposite approach: it's semi-parametric, making no assumption about the baseline hazard. This is why it dominates applied survival analysis in medicine, criminal justice, and customer churn modelling.
By the end of this post, you'll fit a Cox model to real recidivism data, interpret hazard ratios, test the proportional hazards assumption, and extend the model with time-dependent covariates.
The Data: Recidivism After Prison
The Rossi recidivism dataset follows 432 male prisoners for one year after their release from prison. The primary question: does receiving financial aid reduce the risk of rearrest?
Each prisoner has seven baseline covariates (financial aid, age, race, work experience, marital status, parole, prior convictions) plus 52 weekly employment indicators. Of the 432 prisoners, 114 (26%) were rearrested within the year; the remaining 318 were censored (not rearrested during the observation period).
The Kaplan-Meier curve gives us a first look at the survival pattern:
About 74% of prisoners avoid rearrest through the full year. But the curve doesn't tell us which factors predict rearrest. That's where Cox regression comes in.
Quick Win: Cox Regression in Action
Let's fit a Cox model and see which covariates matter. Click the badge to run this yourself:
import numpy as np
import pandas as pd
from lifelines import CoxPHFitter, KaplanMeierFitter
from lifelines.datasets import load_rossi
rossi = load_rossi()
print(f"{len(rossi)} prisoners, {rossi['arrest'].sum()} rearrested")
432 prisoners, 114 rearrested
Fitting the Cox model is one line:
cph = CoxPHFitter()
cph.fit(rossi, duration_col="week", event_col="arrest")
cph.print_summary()
The forest plot shows the hazard ratios for each covariate:
Three findings jump out:
- Age (HR = 0.94, p = 0.01): each additional year of age reduces the hazard of rearrest by 6%. Older prisoners are less likely to reoffend.
- Prior convictions (HR = 1.10, p < 0.005): each additional prior conviction increases the hazard by 10%. Criminal history is the strongest risk factor.
- Financial aid (HR = 0.68, p = 0.05): receiving financial aid reduces the hazard by 32%. This is the key finding for the original study, though it's borderline significant.
The model's concordance is 0.64, meaning it correctly ranks pairs of prisoners by risk 64% of the time.
Now let's see the effect of financial aid on the survival curve, holding all other covariates at their sample means:
Prisoners who received financial aid (blue) have a visibly higher survival probability throughout the year. By week 52, the gap is roughly 7 percentage points: 80% vs 73% avoiding rearrest.
What Just Happened?
The Cox Model in One Equation
The Cox proportional hazards model expresses the hazard (instantaneous risk of the event) for individual $i$ at time $t$ as:
where $h_0(t)$ is the baseline hazard (shared by everyone) and the exponential term scales it up or down based on covariates. The key insight: we never need to estimate $h_0(t)$. Cox's partial likelihood eliminates it entirely, letting us estimate the $\beta$ coefficients from the data alone.
Hazard Ratios: The Language of Cox Models
The quantity $\exp(\beta)$ is the hazard ratio (HR). For a binary covariate like financial aid:
- HR < 1 means the covariate is protective (lower hazard)
- HR > 1 means the covariate increases risk
- HR = 1 means no effect
For our financial aid variable: HR = 0.68 means that, holding everything else constant, prisoners who received financial aid have 68% of the hazard of those who didn't. Equivalently, financial aid reduces the hazard by 32%.
For a continuous covariate like age: HR = 0.94 means each additional year of age multiplies the hazard by 0.94. A 30-year-old has $0.94^{10} = 0.54$ times the hazard of a 20-year-old (46% lower risk).
The Partial Likelihood Trick
The magic of the Cox model is the partial likelihood, introduced by Cox (1972). At each event time, we ask: "Given that someone in the risk set was about to fail, what's the probability it was this particular individual?" That probability depends only on the $\beta$ coefficients, not on $h_0(t)$:
where $R_j$ is the set of individuals still at risk just before time $t_j$. The baseline hazard cancels out in the ratio. This is what makes the Cox model semi-parametric: parametric in the covariate effects, non-parametric in the baseline hazard.
In lifelines, this is all handled internally:
cph = CoxPHFitter()
cph.fit(rossi, duration_col="week", event_col="arrest")
Reading the Output
The print_summary() output gives you:
| Column | Meaning |
|---|---|
coef |
The $\beta$ coefficient (log hazard ratio) |
exp(coef) |
The hazard ratio |
se(coef) |
Standard error of $\beta$
|
z |
Wald test statistic ($\beta / \text{SE}$) |
p |
p-value for $H_0: \beta = 0$
|
exp(coef) lower/upper 95% |
95% CI for the hazard ratio |
A hazard ratio whose 95% CI includes 1.0 is not statistically significant at the 5% level. In the forest plot, covariates in grey (race, work experience, marital status, parole) all span the HR = 1 line.
Going Deeper
Time-Dependent Covariates: When Risk Changes Over Time
The basic Cox model assumes each covariate is fixed at baseline. But what about employment, which changes week to week? The original R code addresses this by expanding the data into start-stop format: one row per person per week, with that week's employment status.
from lifelines import CoxTimeVaryingFitter
# Expand to one row per person-week with employment status
# (see notebook for full expansion code)
ctv = CoxTimeVaryingFitter()
ctv.fit(df_expanded, id_col="id", event_col="arrest",
start_col="start", stop_col="stop")
Employment turns out to be the strongest predictor: HR = 0.35, meaning employed weeks have only 35% of the rearrest hazard compared to unemployed weeks. But the causality is ambiguous: prisoners who avoid rearrest are also more likely to maintain employment. Being in jail prevents you from showing up to work.
The risk trajectories for three participants illustrate this:
The never-employed participant (blue, arrested) maintains consistently high risk. The intermittently employed participant (red) shows risk that jumps up and down as employment changes. The mostly-employed participant (green) has consistently low risk.
The Proportional Hazards Assumption
The Cox model assumes that the hazard ratio between any two individuals stays constant over time. This is the "proportional" in proportional hazards. If financial aid halves your hazard at week 1, it should also halve it at week 50.
We test this with Schoenfeld residuals. For each event, the Schoenfeld residual measures how much the covariate value at that event differs from what the model expected. If the residuals trend with time, the proportional hazards assumption is violated.
For financial aid (left), the LOWESS smoother is roughly flat: the PH assumption holds well (p = 0.98). For age (centre), there's a subtle downward trend: the protective effect of age may weaken over time (p = 0.01, borderline violation). For prior convictions (right), the smoother is close to flat (p = 0.38).
from lifelines.statistics import proportional_hazard_test
ph_test = proportional_hazard_test(cph_reduced, df_reduced, time_transform="rank")
When the PH assumption fails, options include: stratifying by the offending covariate, adding a time-interaction term, or switching to an accelerated failure time model (as in our Bayesian survival post).
The Baseline Hazard
Although the Cox model doesn't need the baseline hazard to estimate coefficients, we can recover it after fitting. The Breslow estimator gives a non-parametric estimate of the cumulative baseline hazard:
The roughly linear increase suggests an approximately constant baseline hazard rate: the risk of rearrest per unit time doesn't change much over the year, for someone with average covariates. This is consistent with an exponential baseline (a finding that would validate a Weibull or exponential parametric model, which is exactly what we used in Post 21).
Cox vs Bayesian AFT: When to Use Which
Our Bayesian survival analysis post used an accelerated failure time (AFT) model with PyMC. How does it compare to Cox?
| Feature | Cox PH | Bayesian AFT |
|---|---|---|
| Approach | Semi-parametric (no baseline assumption) | Parametric (Weibull, log-logistic, etc.) |
| Interpretation | Hazard ratios: "how much faster does the event happen?" | Time ratios: "how much longer until the event?" |
| Uncertainty | Frequentist CIs | Full posterior distributions |
| Flexibility | Time-dependent covariates straightforward | Hierarchical structure, custom priors |
| Assumption | Proportional hazards (testable) | Distributional form of baseline hazard |
Use Cox when you want an assumption-light, interpretable analysis that's the standard in your field (medicine, criminal justice). Use Bayesian AFT when you need uncertainty quantification, hierarchical structure, or when the PH assumption fails.
Hyperparameter Choices
| Parameter | Value | Why |
|---|---|---|
| Full model covariates | fin, age, race, wexp, mar, paro, prio | All baseline covariates from the original study |
| Reduced model | fin, age, prio | Only the statistically significant predictors |
| Time-dependent expansion | 52 weekly employment indicators | One row per person-week for start-stop format |
| Confidence level | 95% | Standard for medical/social science applications |
| PH test transform | Rank | More robust than identity transform for discrete event times |
Where This Comes From
Cox (1972): The Most-Cited Statistics Paper
David Cox introduced the proportional hazards model in his 1972 paper "Regression Models and Life-Tables", published in the Journal of the Royal Statistical Society. With over 50,000 citations, it's one of the most influential statistics papers ever written.
Cox's insight was that for many survival problems, we care about the relative effect of covariates, not the absolute hazard function. By conditioning on the set of individuals at risk at each event time, the baseline hazard cancels out:
"The model is formulated in a very general way and is not restricted to any particular parametric family of distributions."
-- Cox (1972)
The partial likelihood was initially controversial. It's not a full likelihood in the classical sense, and the asymptotic theory required new developments. Andersen and Gill (1982) provided the rigorous counting-process framework that established the mathematical foundations.
The Rossi Dataset
Our dataset comes from Rossi, Berk, and Lenihan (1980), Money, Work, and Crime: Experimental Evidence. The study was a randomised experiment: prisoners were randomly assigned to receive financial aid (or not) upon release, then tracked for one year. This experimental design makes the financial aid coefficient more interpretable than typical observational studies, though compliance was imperfect.
The R analysis we translated follows John Fox's companion chapter to Applied Regression Analysis, which has been a standard teaching resource for Cox models in R since 2002.
From Cox to Modern Survival Analysis
Cox's framework has been extended in many directions:
- Stratified Cox models: different baseline hazards for different groups (e.g., men vs women), but shared covariate effects
- Frailty models: random effects for unobserved heterogeneity, analogous to hierarchical models in Bayesian statistics
-
Time-varying coefficients: let
$\beta$change over time, relaxing the PH assumption - Competing risks: multiple possible event types (rearrest vs death vs emigration)
For Python practitioners, lifelines provides a mature, well-documented implementation. For Bayesian alternatives, PyMC can fit both AFT and piecewise-exponential Cox models (see our Post 21).
Further Reading
- The original paper: Cox (1972), "Regression Models and Life-Tables", JRSS Series B
- Mathematical foundations: Andersen & Gill (1982), "Cox's Regression Model for Counting Processes"
- The R companion chapter: Fox (2002), "Cox Proportional-Hazards Regression for Survival Data"
- Comprehensive textbook: Kalbfleisch & Prentice (2002), The Statistical Analysis of Failure Time Data, Wiley
- Python implementation: Davidson-Pilon (2019), lifelines documentation
Interactive Tools
- Kaplan-Meier Calculator — Estimate and compare survival curves interactively
- Medical Statistics Calculator — Compute diagnostic accuracy metrics for clinical data
Related Posts
- Bayesian Survival Analysis with PyMC: Modelling Customer Churn (Bayesian AFT alternative to Cox)
- Hierarchical Bayesian Regression with PyMC (hierarchical modelling, related to frailty models)
- Maximum Likelihood Estimation from Scratch (the MLE foundation that partial likelihood extends)
- Linear Regression Five Ways (regression fundamentals that Cox builds on)
Frequently Asked Questions
What does "semi-parametric" mean in the context of the Cox model?
The Cox model is parametric in how covariates affect the hazard (through the exponential term with beta coefficients) but non-parametric in the baseline hazard, which is left completely unspecified. This means you get interpretable covariate effects without having to assume the hazard follows a Weibull, exponential, or any other distribution.
How do I interpret a hazard ratio less than 1?
A hazard ratio below 1 means the covariate is protective. For example, a hazard ratio of 0.68 for financial aid means that prisoners receiving aid have 68% of the hazard of those who do not, or equivalently, a 32% reduction in the risk of rearrest at any given time point.
What happens if the proportional hazards assumption is violated?
If the hazard ratio between groups changes over time, the Cox model's estimates become averaged effects that may not accurately represent the relationship at any specific time point. Options include stratifying by the offending covariate, adding a time-interaction term, or switching to an accelerated failure time model that does not require proportional hazards.
Can the Cox model handle categorical covariates with more than two levels?
Yes. Categorical covariates with k levels are represented using k-1 dummy variables, just as in standard linear regression. Each dummy variable's hazard ratio is interpreted relative to the reference category. The lifelines library handles this automatically when you pass categorical columns.
What is the concordance index and what counts as a good value?
The concordance index (C-index) measures how well the model ranks individuals by risk. A value of 0.5 means the model is no better than random, while 1.0 means perfect ranking. In medical and social science applications, values between 0.6 and 0.7 are common and considered acceptable, while values above 0.8 are considered excellent.
Why use the Cox model instead of logistic regression for survival data?
Logistic regression ignores the time dimension entirely and cannot handle censored observations properly. If a prisoner was not rearrested during the study but might be rearrested later, logistic regression would treat them as a definitive non-event, wasting information. The Cox model uses the partial follow-up time from censored subjects to improve estimation.