Let $X:=(X_t)_{t\geq 0}$ be an ergodic Markov process on $\real^d$, and $p>0$. We derive upper bounds of the $p$-Wasserstein distance between the invariant measure and the empirical measures of the Markov process $X$. For this we assume, e.g.\ that the transition semigroup of $X$ is exponentially contractive in terms of the $1$-Wasserstein distance, or that the iterated Poincaré inequality holds together with certain moment conditions on the invariant measure. Typical examples include diffusions and underdamped Langevin dynamics.