We study the limiting fluctuations of the number of sign and level-set clusters of the Gaussian free field on $\mathbb{Z}^d$, $d \ge 3$, that are contained in a large domain. In dimension $d \ge 4$ we prove that the fluctuations are Gaussian at all non-critical levels, while in dimension $d=3$ we show that fluctuations may be Gaussian or non-Gaussian depending on the level. We also show that the sign clusters experience a form of Berry cancellation in all dimensions, that is, the fluctuations of the sign cluster count is suppressed compared to generic levels. Our proof is based on controlling the Weiner-Itô chaos expansion of the cluster count using percolation theoretic inputs; to our knowledge this is the first time that chaos expansion techniques have been applied to analyse a non-local functional of a strongly correlated Gaussian field.