Abstract:Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at this https URL.
| Comments: | Accepted by ICML 2026 |
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2603.00191 [cs.LG] |
| (or arXiv:2603.00191v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.00191 arXiv-issued DOI via DataCite |
From: Lingfeng He [view email]
[v1]
Fri, 27 Feb 2026 02:31:00 UTC (7,144 KB)
[v2]
Sat, 2 May 2026 01:48:44 UTC (7,411 KB)
[v3]
Tue, 12 May 2026 07:35:28 UTC (7,411 KB)
[v4]
Sun, 24 May 2026 03:29:33 UTC (7,413 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。