Motivated by Kloeckner's result on the isometry group of the quadratic Wasserstein space $\mathcal{W}_2\left(\mathbb{R}^n\right)$, we describe the isometry group $\mathrm{Isom}\left(\mathcal{W}_p (E)\right)$ for all parameters $0 < p < \infty$ and for all separable real Hilbert spaces $E.$ In particular, we show that $\mathcal{W}_p(X)$ is isometrically rigid for all Polish space $X$ whenever $0<p<1$. This is a consequence of our more general result: we prove that $\mathcal{W}_1(X)$ is isometrically rigid if $X$ is a complete separable metric space that satisfies the strict triangle inequality. Furthermore, we show that this latter rigidity result does not generalise to parameters $p>1$, by solving Kloeckner's problem affirmatively on the existence of mass-splitting isometries.