Abstract:Transformers often display an attention sink: probability mass concentrates on a fixed, content-agnostic position. Are sinks a byproduct of the optimization/training regime? Or are they sometimes functionally necessary in softmax Transformers? We prove that, in some settings, it is the latter: computing a simple trigger-conditional behavior necessarily induces a sink in softmax self-attention models. Our results formalize a familiar intuition: normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state (e.g., when the model needs to ignore the input). We instantiate this with a concrete task: when a designated trigger token appears, the model must return the average of all preceding token representations, and otherwise output zero, a task which mirrors the functionality of attention heads in the wild (Barbero et al., 2025; Guo et al., 2024). We also prove that non-normalized ReLU attention can solve the same task without any sink, confirming that the normalization constraint is the fundamental driver of sink behavior. Experiments validate our predictions and demonstrate they extend beyond the theoretically analyzed setting: softmax models develop strong sinks while ReLU attention eliminates them in both single-head and multi-head variants.
| Comments: | 21 pages, 8 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2603.11487 [cs.LG] |
| (or arXiv:2603.11487v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.11487 arXiv-issued DOI via DataCite |
From: Yuval Ran-Milo [view email]
[v1]
Thu, 12 Mar 2026 03:13:28 UTC (1,582 KB)
[v2]
Sat, 14 Mar 2026 03:59:57 UTC (1,582 KB)
[v3]
Tue, 17 Mar 2026 01:50:38 UTC (1,582 KB)
[v4]
Thu, 9 Apr 2026 02:48:56 UTC (1,582 KB)
[v5]
Fri, 17 Apr 2026 08:24:12 UTC (1,582 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。