The {\it number rigidity} of a stationary point process $\mathsf{P}$ entails that for a bounded set $A$ the knowledge of $\mathsf{P}$ on $A^{c}$ a.s. determines $\mathsf{P}(A)$; the $k$-order rigidity means the moments of $\mathsf{P}1_{A}$ up to order $k$ can be recovered. We show that $k$-rigidity occurs if the continuous component $\mathscr{s}$ of $\mathsf{P}$'s {\it structure factor} has a zero of order $k$ in $0$, by exploiting a connection with Schwartz's Paley-Wiener theorem for analytic functions of exponential type; these results apply to any random $L^{2}$ wide sense stationary measure on $\mathbb{R}^{d}$ or $\mathbb{Z}^{d}$. In the continuous setting, these local conditions are also necessary if $\mathscr{s}$ has finitely many zeros, or is isotropic, or at the opposite separable. This explains why no model seems to exhibit rigidity in dimension $d\geqslant 3$, and allows to efficiently recover many recent rigidity results about point processes. For a field on $\mathbb{Z} ^{d}$, these results hold provided $\# A >2k$. For a continuous Determinantal point process with reduced kernel $κ$, $k$-rigidity is equivalent to $(1- \widehat {κ^{2}})^{-1}$ having a zero of order $k$ in $0$, which answers questions on completeness and number rigidity. We also deduce some non-integrability results in the less tractable realm of Riesz gases. Finally, we are able to prove that random stationary quasicrystals are maximally rigid on any compact.