We give a necessary condition of geometric nature for the existence of the $H$-fractional Brownian field indexed by a Riemannian manifold. In the case of the Lévy Brownian field ($H=1/2$) indexed by manifolds with minimal closed geodesics it turns out to be very strong. In particular we show that compact manifolds admitting a Lévy Brownian field are simply connected. We also derive from our result the nondegenerescence of the Lévy Brownian field indexed by hyperbolic spaces. These results stress the need for alternative kernels on nonsimply connected manifolds to allow for Gaussian modelling or kernel machine learning of functional data with manifold-valued entries.