We consider a robust asymptotic growth problem under model uncertainty in the presence of stochastic factors. We fix two inputs representing the instantaneous covariance for the asset price process $X$, which depends on an additional stochastic factor process $Y$, as well as the invariant density of $X$ together with $Y$. The stochastic factor process $Y$ has continuous trajectories but is not even required to be a semimartingale. Our setup allows for drift uncertainty in $X$ and model uncertainty for the local dynamics of $Y$. This work builds upon a recent paper of Kardaras & Robertson, where the authors consider an analogous problem, however, without the additional stochastic factor process. Under suitable, quite weak assumptions we are able to characterize the robust optimal trading strategy and the robust optimal growth rate. The optimal strategy is shown to be functionally generated and, remarkably, does not depend on the factor process $Y$. Our result provides a comprehensive answer to a question proposed by Fernholz in 2002. We also show that the optimal strategy remains optimal even in the more restricted case where $Y$ is a semimartingale and the joint covariation structure of $X$ and $Y$ is prescribed as a function of $X$ and $Y$. Our results are obtained using a combination of techniques from partial differential equations, calculus of variations, and generalized Dirichlet forms.