We provide a criterion for establishing lower bounds on the rate of convergence in $f$-variation of a continuous-time ergodic Markov process to its invariant measure. The criterion consists of novel super- and submartingale conditions for certain functionals of the Markov process. It provides a general approach for proving lower bounds on the tails of the invariant measure and the rate of convergence in $f$-variation of a Markov process, analogous to the widely used Lyapunov drift conditions for upper bounds. Our key technical innovation produces lower bounds on the tails of the heights and durations of the excursions from bounded sets of a continuous-time Markov process using path-wise arguments. We apply our theory to elliptic diffusions and Lévy-driven stochastic differential equations with known polynomial/stretched exponential upper bounds on their rates of convergence. Our lower bounds match asymptotically the known upper bounds for these classes of models, thus establishing their rate of convergence to stationarity. The generality of the approach suggests that, analogous to the Lyapunov drift conditions for upper bounds, our methods can be expected to find applications in many other settings.