We study the convergence in rough path topology of a certain class of discrete processes, the hidden Markov walks, to a Brownian motion with an area anomaly. This area anomaly, which is a new object, keeps track of the time-correlation of the discrete models and brings into light the question of embeddings of discrete processes into continuous time. We also identify an underlying combinatorial structure in the hidden Markov walks, which turns out to be a generalization of the occupation time from the classical ergodic theorem in the spirit of rough paths.