Drifts of asset returns are notoriously difficult to model accurately and, yet, trading strategies obtained from portfolio optimization are very sensitive to them. To mitigate this well-known phenomenon we study robust growth-optimization in a high-dimensional incomplete market under drift uncertainty of the asset price process $X$, under an additional ergodicity assumption, which constrains but does not fully specify the drift in general. The class of admissible models allows $X$ to depend on a multivariate stochastic factor $Y$ and fixes (a) their joint volatility structure, (b) their long-term joint ergodic density and (c) the dynamics of the stochastic factor process $Y$. A principal motivation of this framework comes from pairs trading, where $X$ is the spread process and models with the above characteristics are commonplace. Our main results determine the robust optimal growth rate, construct a worst-case admissible model and characterize the robust growth-optimal strategy via a solution to a certain partial differential equation (PDE). We demonstrate that utilizing the stochastic factor leads to improvement in robust growth complementing the conclusions of the previous study by Itkin et. al. (arXiv:2211.15628 [q-fin.MF], forthcoming in $\textit{Finance and Stochastics}$), which additionally robustified the dynamics of the stochastic factor leading to $Y$-independent optimal strategies. Our analysis leads to new financial insights, quantifying the improvement in growth the investor can achieve by optimally incorporating stochastic factors into their trading decisions. We illustrate our theoretical results on several numerical examples including an application to pairs trading.