Abstract:Decoder-only transformers compute the conditional probability of the next token from a sequence of past observations. This paper derives, from first principles, inference architectures that solve the same prediction problem - and in doing so, recovers transformer-like layer operations as a consequence of optimal control theory. The framework is developed for two model classes: a nonlinear model of discrete-valued processes, directly motivated by the transformer, and a linear Gaussian model as a tractable baseline. For both model classes, the prediction objective is reformulated as an optimal control problem whose solution yields an explicit inference algorithm, the dual filter, with a layer structure that mirrors the layer structure of a decoder-only transformer. Numerical experiments provide a comparison of the optimal control to attention weights from a trained transformer. These experiments reveal that when the embedding dimension is insufficient, the transformer implicitly exploits non-Markovian structure.
| Comments: | Preprint |
| Subjects: | Machine Learning (cs.LG); Systems and Control (eess.SY) |
| Cite as: | arXiv:2605.15608 [cs.LG] |
| (or arXiv:2605.15608v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15608 arXiv-issued DOI via DataCite (pending registration) |
From: Aditya Kudre [view email]
[v1]
Fri, 15 May 2026 04:42:19 UTC (1,385 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。