Abstract:Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its demonstrated efficacy across various domains, the theoretical understanding of ICL is still developing. Whereas most existing theory has focused on linear models, we study ICL in the nonlinear regression setting. Through the interaction mechanism in attention, we explicitly construct transformer networks to realize nonlinear features, such as polynomial or spline bases, which span a wide class of functions. Based on this construction, we establish a framework to analyze end-to-end in-context nonlinear regression with the constructed features. Our theory provides finite-sample generalization error bounds in terms of context length and training set size. We numerically validate the theory on synthetic regression tasks.
| Subjects: | Machine Learning (cs.LG); Numerical Analysis (math.NA) |
| Cite as: | arXiv:2605.05176 [cs.LG] |
| (or arXiv:2605.05176v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.05176 arXiv-issued DOI via DataCite (pending registration) |
From: Alexander Hsu [view email]
[v1]
Wed, 6 May 2026 17:42:07 UTC (2,857 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。