Consider a discrete time Markov process $X^\varepsilon$ on $\mathbb R^d$ that makes a deterministic jump prescribed by a map $\varphi \colon \mathbb R^d \to \mathbb R^d$, and then takes a small Gaussian step of variance $\varepsilon^2$. For certain chaotic maps $\varphi$, the effective diffusivity of $X^\varepsilon$ may be bounded away from $0$ as $\varepsilon \to 0$. This is known as residual diffusivity, and in this paper we prove residual diffusivity occurs for a class of maps $\varphi$ obtained from piecewise affine expanding Bernoulli maps.