Given a finite-dimensional inner product space $V$ and a group $G$ of isometries, we consider the problem of embedding the orbit space $V/G$ into a Hilbert space in a way that preserves the quotient metric as well as possible. This inquiry is motivated by applications to invariant machine learning. We introduce several new theoretical tools before using them to tackle various fundamental instances of this problem.