Let $Z = (Z_t)_{t\in[0,\infty)}$ be an ergodic Markov process and, for every $n\in\mathbb{N}$, let $Z^n = (Z_{n^2 t})_{t\in[0,\infty)}$ drive a process $X^n$. Classical results show under suitable conditions that the sequence of non-Markovian processes $(X^n)_{n\in\mathbb{N}}$ converges to a Markov process and give its infinitesimal characteristics. Here, we consider a general sequence $(Z^n)_{n\in\mathbb{N}}$. Using a general result on stochastic averaging from [Kur92], we derive conditions which ensure that the sequence $(X^n)_{n\in\mathbb{N}}$ converges as in the classical case. As an application, we consider the diffusion limit of a Wright-Fisher model with fluctuating selection.