We consider an infinite-dimensional Gaussian regression model, equipped with a high-dimensional Gaussian prior. We address the frequentist validity of posterior credible sets for a vector of linear functionals. We specify conditions for a 'renormalized' Bernstein-von Mises theorem (BvM), where the posterior, centered at its mean, and the posterior mean, centered at the ground truth, have the same normal approximation. This requires neither a solution to the information equation nor a $\sqrt{N}$-consistent estimator. We show that our renormalized BvM implies that a credible ellipsoid, specified by the mean and variance of the posterior, is an asymptotic confidence set. For a single linear functional, we identify a credible ellipsoid with a symmetric credible interval around the posterior mean. We bound the diameter. We check our conditions for Darcy's problem, where the information equation has no solution in natural settings. For the Schrödinger problem, we recover an efficient semi-parametric BvM from our renormalized BvM.