
























Abstract:We study how we can leverage only a handful of characteristics of a transformer's architecture to closely predict the number of different sequences it can output, both qualitatively and quantitatively. We provide an upper bound depending on the length of the prompt, which we show empirically to be tight up to a factor less than 10, across architectures and model sizes. Our analysis also provides a theoretical explanation for previously observed empirical failures of transformers on simple sequence tasks, such as copying and cramming. Formally, we prove that (i) the maximal length of accessible sequences (those that the transformer can output for some prompt) grows linearly with the prompt length, (ii) beyond a critical threshold, the proportion of accessible sequences decays exponentially with sequence length, and (iii) the linear coefficient relating prompt length to accessible sequence length admits a theoretical upper bound. Notably, these results hold even with unbounded context and computation time.
| Comments: | ICML 2026 Spotlight |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.22223 [cs.LG] |
| (or arXiv:2605.22223v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.22223 arXiv-issued DOI via DataCite (pending registration) |
From: Maxime Meyer [view email]
[v1]
Thu, 21 May 2026 09:26:37 UTC (1,289 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。