Stein's method has been widely used to achieve distributional approximations for probability distributions defined in Euclidean spaces. Recently, techniques to extend Stein's method to manifold-valued random variables with distributions defined on the respective manifolds have been reported. However, several of these methods impose strong regularity conditions on the distributions as well as the manifolds and/or consider very special cases. In this paper, we present a novel framework for Stein's method on Riemannian manifolds using the Friedrichs extension technique applied to self-adjoint unbounded operators. This framework is applicable to a variety of conventional and unconventional situations, including but not limited to, intrinsically defined non-smooth distributions, truncated distributions on Riemannian manifolds, distributions on incomplete Riemannian manifolds, etc. Moreover, the stronger the regularity conditions imposed on the manifolds or target distributions, the stronger will be the characterization ability of our novel Stein pair, which facilitates the application of Stein's method to problem domains hitherto uncharted. We present several (non-numeric) examples illustrating the applicability of the presented theory.