It is shown that a seemingly harmless reordering of the steps in a block Gibbs sampler can actually invalidate the algorithm. In particular, the Markov chain that is simulated by the "out-of-order" block Gibbs sampler does not have the correct invariant probability distribution. However, despite having the wrong invariant distribution, the Markov chain converges at the same rate as the original block Gibbs Markov chain. More specifically, it is shown that either both Markov chains are geometrically ergodic (with the same geometric rate of convergence), or neither one is. These results are important from a practical standpoint because the (invalid) out-of-order algorithm may be easier to analyze than the (valid) block Gibbs sampler (see, e.g., Yang and Rosenthal [2019]).