We revisit the long time dynamics of the spherical fully connected $p = 2$-spin glass model when the number of spins $N$ is large but {\it finite}. At $T=0$ where the system is in a (trivial) spin-glass phase, and on long time scale $t \gtrsim {\cal O}{(N^{2/3})}$ we show that the behavior of physical observables, like the energy, correlation and response functions, is controlled by the density of near-extreme eigenvalues at the edge of the spectrum of the coupling matrix $J$, and are thus non self-averaging. We show that the late time decay of these observables, once averaged over the disorder, is controlled by new universal exponents which we compute exactly.