Vladimirov defined an operator on balls in $\mathbb Q_p$, the $p$-adic numbers, that is analogous to the Laplace operator in the real setting. Kochubei later provided a probabilistic interpretation of the operator. This Vladimirov-Kochubei operator generates a real-time diffusion process in the ring of $p$-adic integers, a Brownian motion in $\mathbb Z_p$. The current work shows that this process is a limit of discrete time random walks. It motivates the construction of the Vladimirov-Kochubei operator, provides further intuition about the properties of ultrametric diffusion, and gives an example of the weak convergence of stochastic processes in a profinite group.