We construct optimal low-rank approximations for the Gaussian posterior distribution in linear Gaussian inverse problems with possibly infinite-dimensional separable Hilbert parameter spaces and finite-dimensional data spaces. We first consider approximate posteriors in which the means vary and the posterior covariance is kept fixed, for all possible realisations of the data simultaneously. We give necessary and sufficient conditions for these approximating posteriors to be equivalent to the exact posterior. For such approximations, we measure the data-averaged approximation error with the Kullback-Leibler, Rényi and Amari $α$-divergences for $α\in(0,1)$, and the Hellinger distance. With the loss in Kullback-Leibler and Rényi divergences, we find the optimal approximations and formulate an equivalent condition for their uniqueness, extending the work in finite dimensions of Spantini et al. (SIAM J. Sci. Comput. 2015). We then consider joint low-rank approximation of the mean and covariance. For the reverse Kullback-Leibler divergence, the optimal approximations of the mean and of the covariance yield an optimal joint approximation of the mean and covariance. We interpret one such joint approximation in terms of an optimal projector in parameter space, and show that this approximation amounts to solving a Bayesian inverse problem with projected forward model. Extensive numerical examples demonstrate some of our theoretical findings.