Let $f$ be a Gaussian random field on $\mathbb{R}^d$ and let $X$ be the number of critical points of $f$ contained in a compact subset. A long-standing conjecture is that, under mild regularity and non-degeneracy conditions on $f$, the random variable $X$ has finite moments. So far, this has been established only for moments of order lower than three. In this paper, we prove the conjecture. Precisely, we show that $X$ has finite moment of order $p$, as soon as, at any given point, the Taylor polynomial of order $p$ of $f$ is non-degenerate. We present a simple and general approach that is not specific to critical points and we provide various applications. In particular, we show the finiteness of moments of the nodal volumes and the number of critical points of a large class of smooth, or holomorphic, Gaussian fields, including the Bargmann-Fock ensemble.