Let $μ_t$ denote the critical derivative Gibbs measure of branching Brownian motion at time $t$. It has been proved by Madaule (Stochastic Process. Appl. 126 (2016), no. 2, 470--502) and Maillard and Zeitouni (Ann. Inst. Henri Poincaré Probab. Stat. 52 (2016), no. 3, 1144--1160) that $μ_t$ converges weakly to the random measure $Z_\infty \sqrt{2/π} x^2 e^{-x^2/2} \boldsymbol 1_{x >0} d x$, where $Z_\infty$ is the limit of the derivative martingale. In this paper, we are interested in the fluctuations that occur in this convergence and prove for a large class of functions $F$ that \begin{align*} \sqrt{t} \left( \int_{\mathbb R} F d μ_t - Z_\infty \int_0^\infty F(x) \sqrt{\frac{2}π} x^2 e^{-x^2/2} d x - \frac{c(F) \log t}{\sqrt{t}} Z_\infty \right) \to S(F), \end{align*} in law, as $t\to\infty$, where $c(F)$ is a constant depending on $F$ and, given $Z_\infty$, $S(F)$ has an explicit 1-stable distribution. Moreover, we extend this result to a functional convergence, and we identify precisely the particles responsible for the fluctuations. In particular, this proves the following result for the critical additive martingale $(W_t)_{t\geq 0}$: \[ \sqrt{t} \left( \sqrt{t} W_t - \sqrt{\frac{2}π} Z_\infty \right) \xrightarrow[t\to\infty]{} C Z_\infty, \quad \text{in law}, \] where here $C$ is a Cauchy variable independent of $Z_\infty$, confirming a conjecture by Mueller and Munier (Phys. Rev. E 90 (2014), 042143) in the physics literature.