We propose a Bayesian approach to the problem of multi-reference alignment -- the recovery of signals from noisy, randomly shifted observations. While existing frequentist methods accurately recover the signal at arbitrarily low signal-to-noise ratios, they require a large number of samples to do so. In contrast, our proposed method leverages diffusion models as data-driven plug-and-play priors, conditioning these on the sample power spectrum (a shift-invariant statistic) enabling both accurate posterior sampling and uncertainty quantification. The use of an appropriate prior significantly reduces the required number of samples, as illustrated in simulation experiments with comparisons to state-of-the-art methods such as expectation--maximization and bispectrum inversion. These findings establish our approach as a promising framework for other orbit recovery problems, such as cryogenic electron microscopy (cryo-EM).