The double-directional (DD) wireless channel model is important for realistic system design since it provides complete propagation information. While stochastic and deterministic channel models are widely adopted, and existing machine learning (ML) solutions mostly aim to align future channel realizations, these solutions are often limited to short time spans that may not be statistically significant. Moreover, because the number of multi-path components (MPCs) varies with spatial and temporal variation of the receiver (RX) and/or interacting objects (IOs), typical ML solutions that require fixed, predefined input and output shapes fall short. To curb these limitations, we propose a statistics-aided ML solution that relies on a fixed subset of MPCs selection. More specifically, we first select top-$M$ MPCs, where $M\in\mathbb{Z}^+$ is much smaller than the total number of MPCs, and construct learnable graphs to train our proposed hybrid TimesNet-TimeFilter (TNTF) model. We then use a channel statistics-aided training method to generate future top-M DD channel realizations such that the statistics calculated from these realizations matches closely with those of the actual statistics from the complete time-varying DD channel realizations. We validate the proposed solution using extensive simulations on both synthetic stochastic channel model (SCM)-based and deterministic ray-tracing-based datasets, and demonstrate its effectiveness relative to state-of-the-art baselines.