We consider a time-average estimator $f_{k}$ of a functional of a Markov chain. Under a coupling assumption, we show that the expectation of $f_{k}$ has a limit $μ$ as the number of time-steps goes to infinity. We describe a modification of $f_{k}$ that yields an unbiased estimator $\hat f_{k}$ of $μ$. It is shown that $\hat f_{k}$ is square-integrable and has finite expected running time. Under certain conditions, $\hat f_{k}$ can be built without any precomputations, and is asymptotically at least as efficient as $f_{k}$, up to a multiplicative constant arbitrarily close to $1$. Our approach provides an unbiased estimator for the bias of $f_{k}$. We study applications to volatility forecasting, queues, and the simulation of high-dimensional Gaussian vectors. Our numerical experiments are consistent with our theoretical findings.