Abstract:Standard sequence mixing layers used in language models struggle to balance efficiency and performance. Self-attention performs well on long context tasks but has expensive quadratic compute and linear memory costs, while linear attention and SSMs use only linear compute and constant memory but struggle with long context processing. In this paper, we develop a sequence mixing layer that aims to find a better compromise between memory-compute costs and long-context processing, which we call online vector-quantized (OVQ) attention. OVQ-attention requires linear compute costs and constant memory, but, unlike linear attention and SSMs, it uses a sparse memory update that allows it to greatly increase the size of its memory state and, consequently, memory capacity. We develop a theoretical basis for OVQ-attention based on Gaussian mixture regression, and we test it on a variety of synthetic long context tasks and on long context language modeling. OVQ-attention shows significant improvements over linear attention baselines and the original VQ-attention, on which OVQ-attention was inspired. It demonstrates competitive, and sometimes identical, performance to strong self-attention baselines up 64k sequence length, despite using a small fraction of the memory of full self-attention.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2602.03922 [cs.LG] |
| (or arXiv:2602.03922v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.03922 arXiv-issued DOI via DataCite |
From: Nicholas Alonso [view email]
[v1]
Tue, 3 Feb 2026 18:50:00 UTC (2,711 KB)
[v2]
Fri, 6 Feb 2026 20:08:46 UTC (2,710 KB)
[v3]
Thu, 14 May 2026 20:00:35 UTC (3,670 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。