Let $A$ be a subset of a finite abelian group such that $A$ has a small difference set $A-A$ and the density of $A$ is small. We prove that, counter--intuitively, the smallness (in terms of $|A-A|$) of the Fourier coefficients of $A$ guarantees that $A$ is correlated with a large Bohr set. Our bounds on the size and the dimension of the resulting Bohr set are close to exact.