Abstract:In this paper, we introduce Exact Flow Linear Attention~(EFLA), an exact-flow formulation of delta-rule linear attention. We show that the delta-rule update can be interpreted as an explicit Euler discretization of an underlying continuous-time system. EFLA replaces this first-order update with the exact closed-form flow. By exploiting the rank-1 structure of the dynamics matrix, both the matrix exponential and the input integral collapse to a simple update that preserves delta-rule linear attention's algebraic structure, parameter count, linear-time complexity, and chunkwise parallelism. This attention mechanism removes the Euler discretization error of the delta-rule dynamics without introducing additional parameters. Experiments on robustness tests, language modeling benchmarks, and the MAD synthetic benchmark show that EFLA improves stability under corrupted and high-energy inputs, reduces perplexity, and achieves stronger downstream performance compared to SSM and Euler-style baselines. These results establish exact-flow integration as a principled and scalable update mechanism for delta-rule linear attention.
| Comments: | 16 pages, 5 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2512.12602 [cs.LG] |
| (or arXiv:2512.12602v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.12602 arXiv-issued DOI via DataCite |
From: Jingdi Lei [view email]
[v1]
Sun, 14 Dec 2025 08:51:02 UTC (958 KB)
[v2]
Wed, 7 Jan 2026 08:08:46 UTC (961 KB)
[v3]
Sat, 7 Feb 2026 08:29:37 UTC (959 KB)
[v4]
Fri, 8 May 2026 15:47:57 UTC (689 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。