We study $\mathrm{U}(N)$ invariant polynomials on the space of $N\times N$ matrices first introduced by Capitaine and Casalis, that are precursors of free cumulants in various respects. First, they are polynomials of deterministic matrices, that are not yet evaluated over some probability law, contrary to what is usually meant by cumulants. Secondly, they converge towards the algebraic expression of free cumulants in terms of moments as $N\to \infty$, with $1/N^2$ corrections expressed in terms of monotone Hurwitz numbers. Their most crucial property is their additivity with respect to averaging over sums of $\mathrm{U}(N)$ conjugacy orbits, providing a finite $N$ version of the well-known additivity of free cumulants in free probability. Finally, they extend several properties of free cumulants at finite $N$, including a Wick rule for their average over a Gaussian weight and their appearance in various matrix integrals. Building on the additivity property of these precursors, we also define and compute a coproduct describing the behaviour of general invariant polynomials with respect to the addition of $\mathrm{U}(N)$ conjugacy orbits, as well as their expectation values on sums of $\mathrm{U}(N)$-invariant random matrices. In our construction, a central role is played by the so-called HCIZ integral, both for the definition of the precursors and for the derivation of their properties.