



















Abstract:The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit sphere idealizes layer normalization. In some weight settings the system can even be seen as a gradient flow for an explicit energy, and one can make sense of the infinite context length (mean-field) limit thanks to Wasserstein gradient flows. In this paper we study the effect of the perceptron block in this setting, and show that critical points are generically atomic and localized on subsets of the sphere.
| Subjects: | Machine Learning (cs.LG); Optimization and Control (math.OC) |
| Cite as: | arXiv:2601.21366 [cs.LG] |
| (or arXiv:2601.21366v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.21366 arXiv-issued DOI via DataCite |
From: Borjan Geshkovski [view email]
[v1]
Thu, 29 Jan 2026 07:47:46 UTC (33,176 KB)
[v2]
Wed, 13 May 2026 13:56:23 UTC (33,360 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。