In this paper, we show the possibility of recovering a sum of Dirac measures on the rotation group $SO(3)$ from its low degree moments with respect to Wigner D-functions only. The main Theorem of the paper states, that exact recovery from moments up to degree $N$ is possible, if the support set of the measure obeys a separation distance of $\frac{36}{N+1}$. In this case, the sought measure is the unique solution of a total variation minimization. The proof of the uniqueness requires localization estimates for interpolation kernels and corresponding derivatives on the rotation group $SO(3)$ with explicit constants.