We consider a general class of Markovian models describing the growth in a randomly fluctuating environment of a clonal biological population having several phenotypes related by stochastic switching. Phenotypes differ e.g. by the level of gene expression for a population of bacteria. The time-averaged growth rate of the population, $Λ$, is self-averaging in the limit of infinite times; it may be understood as the fitness of the population in a context of Darwinian evolution. The observation time $T$ being however typically finite, the growth rate fluctuates. For $T$ finite but large, we obtain the variance of the time-averaged growth rate as the maximum of a functional based on the stationary probability distribution for the phenotypes. This formula is general. In the case of two states, the stationary probability was computed by Hufton, Lin and Galla \cite{HufLin2}, allowing for an explicit expression which can be checked numerically.