Let $X=C+\mathrm{E}$ with a deterministic matrix $C\in\R^{M\times M}$ and $\mathrm{E}$ some centered Gaussian $M\times M$-matrix whose entries are independent with variance $σ^2$. In the present work, the accuracy of reduced-rank projections of $X$ is studied. Non-asymptotic universal upper and lower bounds are derived, and favorable and unfavorable prototypes of matrices $C$ in terms of the accuracy of approximation are characterized. The approach does not involve analytic perturbation theory of linear operators and allows for multiplicities in the singular value spectrum. Our main result is some general non-asymptotic upper bound on the accuracy of approximation which involves explicitly the singular values of $C$, and which is shown to be sharp in various regimes of $C$. The results are accompanied by lower bounds under diverse assumptions. Consequences on statistical estimation problems, in particular in the recent area of low-rank matrix recovery, are discussed.