Modern 5G/6G deployments routinely face cross-configuration handovers--users traversing cells with different antenna layouts, carrier frequencies, and scattering statistics--which inflate channel-prediction NMSE by $37.5\%$ on average when models are naively fine-tuned. The proposed improvement frames this mismatch as a continual-learning problem and benchmarks three adaptation families: replay with loss-aware reservoirs, synaptic-importance regularization, and memory-free learning-without-forgetting. Across three representative 3GPP urban micro scenarios, the best replay and regularization schemes cut the high-SNR error floor by up to 2~dB ($\approx 35\%$), while even the lightweight distillation recovers up to $30\%$ improvement over baseline handover prediction schemes. These results show that targeted rehearsal and parameter anchoring are essential for handover-robust CSI prediction and suggest a clear migration path for embedding continual-learning hooks into current channel prediction efforts in 3GPP--NR and O-RAN. The full codebase can be found at https://github.com/ahmd-mohsin/continual-learning-channel-prediction.git.