The plug-in estimator of the squared Euclidean 2-Wasserstein distance is conservative, however due to its large positive bias it is often uninformative. We eliminate most of this bias using a simple centering procedure based on linear combinations. We construct a pair of centered plug-in estimators that decrease with the true Wasserstein distance, and are therefore guaranteed to be informative, for any finite sample size. Crucially, we demonstrate that these estimators can often be viewed as complementary upper and lower bounds on the squared Wasserstein distance. Finally, we apply the estimators to Bayesian computation, developing methods for estimating (i) the bias of approximate inference methods and (ii) the convergence of MCMC algorithms.