

























Abstract:Softmax attention struggles with long contexts due to structural limitations: the strict sum-to-one constraint forces attention sinks on irrelevant tokens, and probability mass disperses as sequence lengths increase. We tackle these problems with Threshold Differential Attention (TDA), a sink-free attention mechanism that achieves ultra-sparsity and improved robustness at longer sequence lengths without the computational overhead of projection methods or the performance degradation caused by noise accumulation of standard rectified attention. TDA applies row-wise extreme-value thresholding with a length-dependent gate, retaining only exceedances. Inspired by the differential transformer, TDA also subtracts an inhibitory view to enhance expressivity. Theoretically, we prove that TDA controls the expected number of spurious survivors per row to $O(1)$ and that consensus spurious matches across independent views vanish as context grows. Empirically, TDA produces $>99\%$ exact zeros and eliminates attention sinks while maintaining competitive performance on standard and long-context benchmarks.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2601.12145 [cs.LG] |
| (or arXiv:2601.12145v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.12145 arXiv-issued DOI via DataCite |
|
| Journal reference: | ACL 2026 |
From: Xingyue Huang [view email]
[v1]
Sat, 17 Jan 2026 19:41:23 UTC (1,399 KB)
[v2]
Thu, 16 Apr 2026 14:20:51 UTC (1,724 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。