We prove that $\frac{\log n}{n}$ is the sharp threshold for universality of the distribution of cokernels of random matrices over $\mathbb{Z}_p$. More precisely, let $α_n = \frac{c\log n}{n}$ for a constant $c>0$ and let $A(n)$ be an $α_n$-balanced random matrix over $\mathbb{Z}_p$. For non-symmetric, symmetric, and alternating matrix models, we prove that if $c>1$, then the limiting distribution of the cokernel of $A(n)$ coincides with the universal distribution of the corresponding symmetry type, whereas universality fails at the critical scale $c=1$. This improves earlier universality results, which required $α_n \gg \frac{\log n}{n}$, to the optimal threshold. As an application, we generalize the universality result for Sylow $p$-subgroups of sandpile groups of Erdős-Rényi random graphs to a broader class of Erdős-Rényi graph sequences. Our approach is based on a unified framework that simultaneously treats all symmetry types of random matrices as well as the random graph model, rather than handling each case separately.