We consider the singular numbers of a certain explicit continuous-time Markov jump process on $\mathrm{GL}_N(\mathbb{Q}_p)$, which we argue gives the closest $p$-adic analogue of multiplicative Dyson Brownian motion. We do so by explicitly classifying the possible dynamics of singular numbers of processes on $\mathrm{GL}_N(\mathbb{Q}_p)$ satisfying natural properties possessed by Brownian motion on $\mathrm{GL}_N(\mathbb{C})$. Computing the evolution of singular numbers explicitly, we find that the $N$-tuple of singular numbers in decreasing order evolves as a Poisson jump process on $\mathbb{Z}^N$, with ordering enforced by reflection off the walls of the positive type $A$ Weyl chamber. This contrasts with -- and provides a $p$-adic analogue to -- the behavior of classical Dyson Brownian motion, where ordering is enforced by conditioning to avoid the Weyl chamber walls.