This paper studies rescaled images, under $\exp^{-1}_μ$, of the sample Fréchet means of i.i.d. random variables $\{X_k\vert k\geq 1\}$ with Fréchet mean $μ$ on a Riemannian manifold. We show that, with appropriate scaling, these images converge weakly to a diffusion process. Similar to the Euclidean case, this limiting diffusion is a Brownian motion up to a linear transformation. However, in addition to the covariance structure of $\exp^{-1}_μ(X_1)$, this linear transformation also depends on the global Riemannian structure of the manifold.