In this paper, we study the distribution of the cokernels of random $p$-adic matrices with fixed zero entries. Let $X_n$ be a random $n \times n$ matrix over $\mathbb{Z}_p$ in which some entries are fixed to be zero and the other entries are i.i.d. copies of a random variable $ξ\in \mathbb{Z}_p$. We consider the minimal number of random entries of $X_n$ required for the cokernel of $X_n$ to converge to the Cohen--Lenstra distribution. When $ξ$ is given by the Haar measure, we prove a lower bound of the number of random entries and prove its converse-type result using random regular bipartite multigraphs. When $ξ$ is a general random variable, we determine the minimal number of random entries. Let $M_n$ be a random $n \times n$ matrix over $\mathbb{Z}_p$ with $k$-step stairs of zeros and the other entries given by independent random $ε$-balanced variables valued in $\mathbb{Z}_p$. We prove that the cokernel of $M_n$ converges to the Cohen--Lenstra distribution under a mild assumption. This extends Wood's universality theorem on random $p$-adic matrices.