

























Abstract:Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments.
| Subjects: | Machine Learning (cs.LG) |
| ACM classes: | I.2; I.2.6 |
| Cite as: | arXiv:2602.17363 [cs.LG] |
| (or arXiv:2602.17363v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.17363 arXiv-issued DOI via DataCite |
From: Gabriel Mongaras [view email]
[v1]
Thu, 19 Feb 2026 13:45:23 UTC (8,567 KB)
[v2]
Thu, 2 Apr 2026 02:07:56 UTC (20,578 KB)
[v3]
Fri, 15 May 2026 03:46:00 UTC (8,596 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。