We study the thermodynamic properties of the simplest gauged permutation invariant matrix quantum mechanical system of oscillators, for general matrix size $N$. In the canonical ensemble, the model has a transition at a temperature $T$ given by $x = e^{ -1/ T } \sim x_c=e^{-1/T_c}=\frac{\log N}{N}$, characterised by a sharp peak in the specific heat capacity (SHC), which separates a high temperature from a low temperature region. The peak grows and the low-temperature region shrinks to zero with increasing $N$. In the micro-canonical ensemble, for finite $N$, there is a low energy phase with negative SHC and a high energy phase with positive SHC. The low-energy phase is dominated by a super-exponential growth of degeneracies as a function of energy which is directly related to the rapid growth in the number of directed graphs, with any number of vertices, as a function of the number of edges. The two ensembles have matching behaviour above the transition temperature. We further provide evidence that these thermodynamic properties hold in systems with $U(N)$ symmetry such as the zero charge sector of the 2-matrix model and in certain tensor models. We discuss the implications of these observations for the negative specific heat capacities in gravity using the AdS/CFT correspondence.