Given a planar curve, imagine rolling a sphere along that curve without slipping or twisting, and by this means tracing out a curve on the sphere. It is well known that such a rolling operation induces a local isometry between the sphere and the plane so that the two curves uniquely determine each other, and moreover, the operation extends to a general class of manifolds in any dimension. We use rolling to construct an analogue of a Gaussian process on a manifold starting from a Euclidean Gaussian process with mean $m$ and covariance $K$, and refer to it as a rolled Gaussian process parameterized by $m$ and $K$. The resulting model is generative, and is amenable to statistical inference given data as curves on a manifold. We identify conditions on the manifold under which the rolling of $m$ equals the Fréchet mean of the rolled Gaussian process, propose computationally simple estimators of $m$ and $K$, and derive their rates of convergence. We illustrate with examples on the unit sphere, symmetric positive-definite matrices, and with a robotics application involving 3D orientations.