






















Abstract:Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive token distributions toward clustering. However, existing mean-field analyses largely treat model parameters as prescribed, leaving open how training reshapes this clustering picture. We study this question in a noisy mean-field Transformer in which only a parameter-linear FFN is trained under $L^2$ regularization. We find and analyze a training-induced phase in the dynamics: after initially following attention-driven clustering, the token distribution can leave the clustered regime near the final layers. Our mathematical analysis is based on an entropy-regularized interaction energy that captures the clustering bias of attention. More broadly, our results point toward a training-aware mean-field theory of Transformer dynamics, in which training and inference dynamics are treated together.
| Comments: | 48 pages, 6 figures, comments are wellcome! |
| Subjects: | Machine Learning (cs.LG); Analysis of PDEs (math.AP); Dynamical Systems (math.DS); Optimization and Control (math.OC) |
| MSC classes: | 49N80, 35Q84, 49Q22, 68T07 |
| Cite as: | arXiv:2605.07772 [cs.LG] |
| (or arXiv:2605.07772v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07772 arXiv-issued DOI via DataCite (pending registration) |
From: Noboru Isobe [view email]
[v1]
Fri, 8 May 2026 14:12:13 UTC (4,294 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。