Consider a discrete time Markov process $X^ε$ on $\mathbf R^d$ that makes a deterministic jump based on its current location, and then takes a small Gaussian step of variance $ε^2$. We study the behavior of the asymptotic variance as $ε\to 0$. In some situations (for instance if there were no jumps), then the asymptotic variance vanishes as $ε\to 0$. When the jumps are "chaotic", however, the asymptotic variance may be bounded from above and bounded away from $0$, as $ε\to 0$. This phenomenon is known as residual diffusivity, and we prove this occurs when the jumps are determined by certain expanding Bernoulli maps.