For a large family of stationary continuous Gaussian fields $f$ on $\mathbb{R}^d$, including the Bargmann-Fock and Cauchy fields, we prove that there exists at most one unbounded connected component in the level set $\{f=\ell\}$ (as well as in the excursion set $\{f\geq\ell\}$) almost surely for every level $\ell\in \mathbb{R}$, thus proving a conjecture proposed by Duminil-Copin, Rivera, Rodriguez & Vanneuville. As the fields considered are typically very rigid (e.g.~analytic almost surely), there is no sort of finite energy property available and the classical approaches to prove uniqueness become difficult to implement. We bypass this difficulty using a soft shift argument based on the Cameron-Martin theorem.