The Quantum Random Energy Model (QREM) is a random matrix of Anderson-type which describes effects of a transversal magnetic field on Derrida's spin glass. The model exhibits a glass phase as well as a classical and a quantum paramagnetic phase. We analyze in detail the low-energy spectrum and establish a localization-delocalization transition for the corresponding eigenvectors of the QREM. Based on a combination of random matrix and operator techniques as well as insights in the random geometry, we derive next-to-leading order asymptotics for the ground-state energy and eigenvectors in all regimes of the parameter space. Based on this, we also deduce the next-to-leading order of the free energy, which turns out to be deterministic and on order one in the system size in all phases of the QREM. As a result, we determine the nature of the fluctuations of the free energy in the spin glass regime.