For a reversible and ergodic Markov chain $\{X_n,n\geq0\}$ with invariant distribution $π$, we show that a valid confidence interval for $π(h)$ can be constructed whenever the asymptotic variance $σ^2_P(h)$ is finite and positive. We do not impose any additional condition on the convergence rate of the Markov chain. The confidence interval is derived using the so-called fixed-b lag-window estimator of $σ_P^2(h)$. We also derive a result that suggests that the proposed confidence interval procedure converges faster than classical confidence interval procedures based on the Gaussian distribution and standard central limit theorems for Markov chains.