We study a class of self-repelling diffusions on compact Riemannian manifolds whose drift is the gradient of a potential accumulated along their trajectory. When the interaction potential admits a suitable spectral decomposition, the dynamics and its environment are equivalent to a finite-dimensional degenerate diffusion. We show that this diffusion is a second-order lift of an Ornstein-Uhlenbeck process whose invariant law corresponds to the Gaussian invariant measure of the environment, and immediately obtain a general upper bound on the rate of convergence to stationarity using the framework of second-order lifts. Furthermore, using a flow Poincaré inequality, we develop lower bounds on the convergence rate. We show that, in the periodic case, these lower bounds improve upon those of Benaïm and Gauthier (Probab. Theory Relat. Fields, 2016), and even match the order of the upper bound in some cases.