Learning recurrent connectivity that supports memory over long intrinsic timescales is a basic problem in the theory of dynamical computation. While continuous attractor and integrator models describe how tuned recurrent circuits can maintain information, less is known about how such slow modes are acquired by gradient-based learning. Here we study this question in an analytically tractable setting: we build a mathematical theory of the learning dynamics of linear RNNs trained to integrate white noise. We show that when the initial recurrent weights are small, the dynamics of learning are described by a low-dimensional system that tracks a single outlier eigenvalue of the recurrent weights. This reveals the precise manner in which the long timescale associated with white noise integration is learned. We extend our analyses to RNNs learning a damped oscillatory filter, and find low-dimensional effective dynamical equations for the evolution of a conjugate pair of outlier eigenvalues. Taken together, our analyses build a rich mathematical framework for studying dynamical learning problems relevant to both machine learning and neuroscience.