

























Abstract:Generative Pre-trained Transformer (GPT) architectures are the most popular design for language modeling. Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape. We propose a minimal modification of the GPT setting to unify it with the EBM framework. The inference step of our model, which we call eNeRgy-GPT (NRGPT), is conceptualized as an exploration of the tokens on the energy landscape. We prove, and verify empirically, that under certain circumstances this exploration becomes gradient descent, although they don't necessarily lead to the best performing models. We demonstrate that our model performs well for simple language (Shakespeare dataset), algebraic ListOPS tasks, and richer settings such as OpenWebText language modeling. We also observe that our models may be more resistant to overfitting, doing so only during very long training.
| Comments: | Accepted to ICLR 2026 main conference |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2512.16762 [cs.LG] |
| (or arXiv:2512.16762v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.16762 arXiv-issued DOI via DataCite |
From: Benjamin Hoover [view email]
[v1]
Thu, 18 Dec 2025 16:59:10 UTC (2,849 KB)
[v2]
Wed, 25 Feb 2026 18:23:01 UTC (2,859 KB)
[v3]
Fri, 1 May 2026 17:39:23 UTC (2,841 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。