A frequent and well-founded criticism of the maximum a posteriori (MAP) and minimum mean squared error (MMSE) estimates of a continuous parameter γtaking values in a differentiable manifold Γis that they are not invariant to arbitrary ``reparameterizations'' of Γ. This paper clarifies the issues surrounding this problem, by pointing out the difference between coordinate invariance, which is a sine qua non for a mathematically well-defined problem, and diffeomorphism invariance, which is a substantial issue, and then provides a solution. We first show that the presence of a metric structure on Γcan be used to define coordinate-invariant MAP and MMSE estimates, and we argue that this is the natural way to proceed. We then discuss the choice of a metric structure on Γ. By imposing an invariance criterion natural within a Bayesian framework, we show that this choice is essentially unique. It does not necessarily correspond to a choice of coordinates. In cases of complete prior ignorance, when Jeffreys' prior is used, the invariant MAP estimate reduces to the maximum likelihood estimate. The invariant MAP estimate coincides with the minimum message length (MML) estimate, but no discretization or approximation is used in its derivation.