Abstract:We explore the expressive power of Transformers by establishing precise approximation error upper and lower bounds for Hölder class. Specifically, a new approximation upper bound is derived for the standard Transformer architecture equipped with Softmax operators, ReLU activation functions, and residual connections. We prove that a Transformer network composed of at most $\mathcal{O}(\varepsilon^{-{d_{0}}/{\alpha}})$ blocks can approximate any bounded Hölder function with $d_{0}$-dimensional input and smoothness $\alpha\in(0,1]$ under any accuracy $\varepsilon>0$. In the case of approximation lower bounds, leveraging the VC-dimension upper bound, we are the first to rigorously prove that Transformers demand for at least $\Omega(\varepsilon^{-{d_{0}}/({4\alpha})})$ blocks to achieve the $\varepsilon$ approximation accuracy. As a final step, we extend the derived results for standard Transformers to a general regression task and establish the corresponding excess risk rates demonstrating Transformers' empirical effectiveness in real-world settings.
| Comments: | 31 pages, 2 figures. Accepted by ICML2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.07463 [cs.LG] |
| (or arXiv:2605.07463v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07463 arXiv-issued DOI via DataCite (pending registration) |
From: Xin He [view email]
[v1]
Fri, 8 May 2026 09:10:31 UTC (352 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。