We examine to what extent the tempo and mode of environmental fluctuations matter for the growth of structured populations. The models are switching, linear ordinary differential equations $x'(t)=A(σ(ωt))x(t)$ where $x(t)=(x_1(t),\dots,x_d(t))$ corresponds to the population densities in the $d$ individual states, $σ(t)$ is a piece-wise constant function representing the fluctuations in the environmental states $1,\dots,N$, $ω$ is the frequency of the environmental fluctuations, and $A(1),\dots,A(n)$ are Metzler matrices. $σ(t)$ can either be a periodic function or correspond to a continuous-time Markov chain. Under suitable conditions, there is a Lyapunov exponent $Λ(ω)$ such that $\lim_{t\to\infty} \frac{1}{t}\log\sum_i x_i(t)=Λ(ω)$ for all non-negative, non-zero initial conditions $x(0)$ (with probability one in the random case). For both forms of switching, we derive analytical first-order and second-order approximations of $Λ(ω)$ in the limits of slow ($ω\to 0$) and fast ($ω\to\infty$) environmental fluctuations. When the order of switching and the average switching times are equal, we show that the first-order approximations of $Λ(ω)$ are equivalent in the slow-switching limit, but not in the fast-switching limit. We illustrate our results with applications to stage-structured and spatially-structured models. When dispersal rates are symmetric, the first order approximations suggest that population growth rates increase with the frequency of switching -- consistent with earlier work on periodic switching. In the absence of dispersal symmetry, we demonstrate that $Λ(ω)$ can be non-monotonic in $ω$. In conclusion, our results show how population growth rates depend on the tempo ($ω$) and mode (random versus deterministic) of the environmental fluctuations.