Consider a real Gaussian stationary process $f_ρ$, indexed on either $\mathbb{R}$ or $\mathbb{Z}$ and admitting a spectral measure $ρ$. We study $θ_ρ^\ell=-\lim\limits_{T\to\infty}\frac{1}{T} \log\mathbb{P}\left(\inf_{t\in[0,T]}f_ρ(t)>\ell\right)$, the persistence exponent of $f_ρ$. We show that, if $ρ$ has a positive density at the origin, then the persistence exponent exists; moreover, if $ρ$ has an absolutely continuous component, then $θ_ρ^\ell>0$ if and only if this spectral density at the origin is finite. We further establish continuity of $θ_ρ^\ell$ in $\ell$, in $ρ$ (under a suitable metric) and, if $ρ$ is compactly supported, also in dense sampling. Analogous continuity properties are shown for $ψ_ρ^\ell=-\lim\limits_{T\to\infty}\frac{1}{T} \log\mathbb{P}\left(\inf_{t\in[0,T]}|f_ρ(t)|\le \ell\right)$, the ball exponent of $f_ρ$, and it is shown to be positive if and only if $ρ$ has an absolutely continuous component.