Abstract:Discrete diffusion language models (dLLMs) have recently emerged as a promising alternative to traditional autoregressive approaches, offering the flexibility to generate tokens in arbitrary orders and the potential of parallel decoding. However, existing heuristic sampling strategies remain inefficient: they choose only a small part of tokens to sample at each step, leaving substantial room for improvement. In this work, we study the problem of token sampling order optimization and demonstrate its significant potential for acceleration. Specifically, we find that fully leveraging correct predictions at each step can reduce the number of sampling iterations by an order of magnitude without compromising accuracy. Based on this, we propose Neural Indicator Sampling (NI Sampling), a general sampling order optimization framework that utilize a neural indicator to decide which tokens should be sampled at each step. We further propose a novel trajectory-preserving objective to train the indicator. Experiments on LLaDA and Dream models across multiple benchmarks show that our method achieves up to 14.3$\times$ acceleration over full-step sampling with negligible performance drop, and consistently outperforms confidence threshold sampling in the accuracy-step trade-off. Code is available at this https URL.
| Comments: | Accepted by ICLR 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.18471 [cs.LG] |
| (or arXiv:2604.18471v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.18471 arXiv-issued DOI via DataCite |
From: Enshu Liu [view email]
[v1]
Mon, 20 Apr 2026 16:22:59 UTC (729 KB)
[v2]
Sun, 26 Apr 2026 18:45:11 UTC (729 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。