Abstract:Transformer-based models have achieved remarkable success across a wide range of domains, yet our understanding of their training dynamics remains limited. In this work, we identify a recurrent focus-dilution cycle in attention learning and provide a rigorous explanation in a one-layer Transformer setting for Markovian data via gradient-flow analysis. Using stage-wise linearization around critical points, we show that a single focus-dilution cycle can be decomposed into a sequence of distinct stages. First, embedding and projection rapidly condense to a rank-one structure, while attention parameters remain effectively frozen. Then, the attention parameters begin to increase, inducing a frequency-driven focus toward high-frequency tokens. As attention continues to evolve, it generates next-order perturbations in embeddings, leading to a mass-redistribution mechanism that progressively dilutes this focus. Finally, small asymmetries among low-frequency tokens lift a degenerate critical point, opening new embedding directions and initiating the next cycle. Experiments on synthetic Markovian data as well as WikiText and TinyStories corroborate the predicted stages and cyclical dynamics.
| Comments: | ICML 2026 spotlight |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.01199 [cs.LG] |
| (or arXiv:2605.01199v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.01199 arXiv-issued DOI via DataCite (pending registration) |
From: Zheng-An Chen [view email]
[v1]
Sat, 2 May 2026 02:30:52 UTC (2,382 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。