In this paper we derive a quantitative dichotomy for the top Lyapunov exponent of a class of non-dissipative SDEs on a compact manifold in the small noise limit. Specifically, we prove that in this class, either the Lyapunov exponent is zero for all noise strengths, or it is positive for all noise strengths and that the decay of the exponent in the small-noise limit cannot be faster than linear in the noise parameter. As an application, we study the top Lyapunov exponent for the motion of a charged particle in randomly-fluctuating magnetic fields, which also involves an interesting geometric control problem.