Abstract:We present a discrete diffusion-based language model using Glauber dynamics from statistical physics. Our main insight is that instead of trying to train a discrete state space diffusion model using Glauber dynamics with a uniform transition kernel as the forward process, one can set up an ``energy function'' based on pretrained causal/masked language models. When viewed as the stationary distribution, this energy function allows us to significantly improve the quality of the generated text. Incorporating UL2 as the pretrained model into our diffusion pipeline, we outperform prior diffusion based LMs and perform competitively with autoregressive models of comparable model sizes. Furthermore, our models are competitive with or outperform prior diffusion models and GPT-2 style auto-regressive models on zero-shot common sense reasoning tasks as well as planning and search tasks like Sudoku and Zebra puzzles.
| Comments: | To appear in ACL 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.04291 [cs.LG] |
| (or arXiv:2605.04291v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.04291 arXiv-issued DOI via DataCite (pending registration) |
From: Tarun Kathuria [view email]
[v1]
Tue, 5 May 2026 20:51:03 UTC (9,126 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。