We study how changes in population size and fluctuating environmental conditions influence the establishment of seed banks in plants. Our model is a modification of the Wright-Fisher model with seed bank, introduced by Kaj, Krone and Lascoux. We distinguish between wild type individuals, producing only nondormant seeds, and mutants, producing seeds with dormancy. To understand how changing population size shapes the establishment of seed banks, we analyse the process under a diffusive scaling. The results support the biological insight that seed banks are favoured in a declining population, and disfavoured if population size is constant or increasing. The surprise is that this is true even when population sizes are changing very slowly -- over evolutionary timescales. We also investigate the influence of short-term fluctuations, such as annual variations in rainfall or temperature. Mathematically, our analysis reduces to a stochastic dynamical system forced onto a manifold by a large drift, which converges under scaling to a diffusion on the manifold. Inspired by the Lyapunov--Schmidt reduction, we derive an explicit formula for the limiting diffusion coefficients by projecting the system onto its linear counterpart. This provides a general framework for deriving diffusion approximations in models with strong drift and nonlinear constraints.