We consider a stochastically forced nonlinear oscillator driven by a stationary Gaussian noise that has an algebraically decaying covariance function. It is well known that such noise processes can be renormalized to converge to fractional Brownian motion, a process that has memory. In contrast, we show that the renormalized limit of the nonlinear oscillator driven by this noise converges to diffusion driven by standard (not fractional) Brownian motion, and thus retains no memory in the scaling limit. The proof is based on the study of a fast-slow system using the perturbed test function method.