We derive a covariance formula for the number of excursion or level set components of a smooth stationary Gaussian field on $\mathbb{R}^d$ contained in compact domains. We also present two applications of this formula: (1) for fields whose correlations are integrable we prove that the variance of the component count in large domains is of volume order and give an expression for the leading constant, and (2) for fields with slower decay of correlation we give an upper bound on the variance which is of optimal order if correlations are regularly varying, and improves on best-known bounds if correlations are oscillating (e.g.\ monochromatic random waves).