Abstract:Large language models (LLMs) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks. Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization. However, these methods lack semantic awareness of internal knowledge distribution in LLMs. As a result, they cannot distinguish parameters that should be preserved or updated. We propose an attribution-guided continual fine-tuning framework for LLMs. Our method estimates task-specific, element-wise parameter importance in each Transformer layer and uses these scores to modulate gradients. Parameters important to previous tasks receive smaller updates, while less relevant ones remain plastic for learning new tasks. Experiments on continual learning benchmarks show that our method consistently outperforms baselines, achieving better retention of old tasks while maintaining competitive performance on new tasks.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.05285 [cs.LG] |
| (or arXiv:2605.05285v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.05285 arXiv-issued DOI via DataCite |
From: Yazheng Liu [view email]
[v1]
Wed, 6 May 2026 17:38:13 UTC (907 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。