Abstract:Diffusion Language Models (DLMs) decode multiple tokens in parallel, but aggressive multi-token decoding amplifies cross-token dependency errors and can sharply degrade generation quality. We propose BackPlay, a frozen-backbone self-correction framework that trains only a lightweight correction head on a finetuned DLM without updating any backbone or adapter parameters. Because the head is trained on errors produced by the same frozen generator used at inference time, its training distribution aligns with the error patterns of the deployed model. We further introduce Look-back Correction, a training mechanism that injects predictions from earlier, more corrupted denoising states into later, richer contexts, enabling the head to leverage later context to detect mistakes made in earlier generation steps. During inference, BackPlay periodically revisits previously generated tokens through selective remasking and regeneration to limit error accumulation. Across mathematical reasoning and code generation benchmarks, BackPlay improves the speed--quality trade-off of the underlying DLM under multi-token decoding.
| Comments: | 16 pages |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2601.06428 [cs.LG] |
| (or arXiv:2601.06428v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.06428 arXiv-issued DOI via DataCite |
From: Liming Liu [view email]
[v1]
Sat, 10 Jan 2026 05:04:33 UTC (75 KB)
[v2]
Wed, 4 Feb 2026 03:42:06 UTC (72 KB)
[v3]
Wed, 22 Apr 2026 22:08:24 UTC (77 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。