We consider the $SO(d)$ and $O(d)$ invariant sectors of the bosonic $d$-matrix harmonic oscillator with $U(N)$ gauge symmetry. The micro-canonical degeneracy $\mathcal{Z}( N , d , k )$ for fixed energy $k$ is expressed as a pairing between an $N$-dependent vector and a $d$-dependent vector in the space of partitions of the integer $k$. This pairing formula is derived by counting invariant words in multi-matrix variables $X^i_{j,a}$, using properties of Clebsch-Gordan multiplicities (Kronecker coefficients) for the symmetric group $S_k$, Schur-Weyl duality and harmonic analysis on the homogeneous space $U(d)/SO(d)$. Analytic formulae for large $N$ and $k$ with $ k \le N $ are obtained using group integrals over $U(N)$ and $SO(d)$ (or $ O(d)$). The micro-canonical heat capacity in this regime is negative and turns positive, at a critical value $k_{\rm crit}$, due to finite $N$ modifications to the counting, thus forming what we denote as a characteristic caloric fold in the $ E $ versus $T$ curve. Data from the pairing formula is well fitted by $k_{\rm crit} \sim { N^2 \over 4 }$ for small values of $d$. A derivation of this large $N$ formula is given using a matrix model approximation and semi-classical analysis of the eigenvalue density. The large $N,d$ limit of the degeneracies reveals a key role for ribbon graph combinatorics. The caloric fold is also notably a property of black hole thermodynamics in anti-de-Sitter spaces. We propose the spherically symmetric \(SO(d)\) and \(O(d)\) invariant sectors of \(d\)-matrix quantum mechanics as tractable matrix systems for capturing key features of dual descriptions of black-hole thermodynamics.