Let $B=(B_x)_{x\in\mathbb{R}^d}$ be a collection of $N(0,1)$ random variables forming a real-valued continuous stationary Gaussian field on $\mathbb{R}^d$, and set $C(x-y)=\mathbb{E}[B_xB_y]$. Let $\varphi:\mathbb{R}\to\mathbb{R}$ be such that $\mathbb{E}[\varphi(N)^2]<\infty$ with $N\sim N(0,1)$, let $R$ be the Hermite rank of $\varphi$, and consider $Y_t = \int_{tD} \varphi(B_x)dx$, $t>0$, with $D\subset \mathbb{R}^d$ compact. Since the pioneering works from the 80s by Breuer, Dobrushin, Major, Rosenblatt, Taqqu and others, central and noncentral limit theorems for $Y_t$ have been constantly refined, extended and applied to an increasing number of diverse situations, to such an extent that it has become a field of research in its own right. The common belief, representing the intuition that specialists in the subject have developed over the last four decades, is that as $t\to\infty$ the fluctuations of $Y_t$ around its mean are, in general (i.e. except possibly in very special cases), Gaussian when $B$ has short memory, and non Gaussian when $B$ has long memory and $R\geq 2$. We show in this paper that this intuition forged over the last forty years can be wrong, and not only marginally or in critical cases. We will indeed bring to light a variety of situations where $ Y_t $ admits Gaussian fluctuations in a long memory context. To achieve this goal, we state and prove a spectral central limit theorem, which extends the conclusion of the celebrated Breuer-Major theorem to situations where $C\not\in L^R(\mathbb{R}^d)$. Our main mathematical tools are the Malliavin-Stein method and Fourier analysis techniques.