Recently, a generalized Bernoulli process (GBP) was developed as a stationary binary sequence that can have long-range dependence. In this paper, we find the scaling limit of a random walk that follows GBP. The result is a new class of non-Markovian diffusion processes. The limiting processes include continuous-time stochastic processes with stationary increments whose correlation decays with an exponential rate, a power law, or an exponentially tempered power law. The limit densities solve a time tempered fractional diffusion equation or time fractional diffusion equation. The second-family of Mittag-Leffler distribution and exponential distribution arise as special cases of the limiting distributions. Subordinated processes are considered as time-changed Lévy processes, and the governing equations and dependence structure of the subordinated processes are discussed.