View PDF HTML (experimental)

Abstract:Continual learning (CL) with large pre-trained models aims to incrementally acquire knowledge without catastrophic forgetting. Existing LoRA-based Mixture-of-Experts (MoE) methods expand capacity by adding isolated new experts while freezing old ones, but still suffer from redundancy, interference, routing ambiguity, and consequent forgetting. We investigate the issues stemming from coarse-grained expert granularity. Coarse-grained experts (e.g., high-rank LoRA) encode low-specialty information, leading to expert duplication/interference and routing degradation/confusion as experts accumulate. In this work, we propose MoRAM (Mixture of Rank-1 Associative Memory). Grounded in the view that weight matrices act as linear associative memories, MoRAM achieves CL as gradual incrementing of reusable atomic rank-1 experts as memory. Each rank-1 adapter acts as a fine-grained MoE expert or an associative memory unit. By viewing rank-1 adapters as key-value memory pairs, we eliminate explicit MoE-LoRA routers with self-activation, where each memory atom evaluates its relevance via its intrinsic key. The inference process thus becomes a robust, content-addressable retrieval over the incrementally accumulated memory. Extensive experiments on CLIP and LLMs show that MoRAM significantly outperforms state-of-the-art methods, achieving a better plasticity-stability trade-off, stronger generalization, and reduced forgetting. Project Page: this https URL.

Submission history

From: Haodong Lu [view email]
[v1] Thu, 26 Jun 2025 06:19:05 UTC (3,609 KB)
[v2] Thu, 9 Oct 2025 05:43:44 UTC (3,636 KB)
[v3] Fri, 6 Feb 2026 02:59:44 UTC (3,713 KB)
[v4] Wed, 11 Feb 2026 00:58:13 UTC (3,714 KB)
[v5] Thu, 21 May 2026 00:15:19 UTC (3,734 KB)