Abstract:Reinforcement learning (RL) struggles to scale to large, combinatorial action spaces common in many real-world problems. This paper introduces a novel framework for training discrete diffusion models as highly effective policies in these complex settings. Our key innovation is an efficient online training process that ensures stable and effective policy improvement. By leveraging policy mirror descent (PMD) to define an ideal, regularized target policy distribution, we frame the policy update as a distributional matching problem, training the expressive diffusion model to replicate this stable target. This decoupled approach stabilizes learning and significantly enhances training performance. Our method achieves state-of-the-art results and superior sample efficiency across a diverse set of challenging combinatorial benchmarks, including DNA sequence generation, RL with macro-actions, and multi-agent systems. Experiments demonstrate that our diffusion policies attain superior performance compared to other baselines.
| Comments: | 22 pages, 10 figures. Haitong Ma and Ofir Nabati contributed equally to this paper |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2509.22963 [cs.LG] |
| (or arXiv:2509.22963v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.22963 arXiv-issued DOI via DataCite |
From: Haitong Ma [view email]
[v1]
Fri, 26 Sep 2025 21:53:36 UTC (323 KB)
[v2]
Wed, 1 Oct 2025 00:48:42 UTC (323 KB)
[v3]
Tue, 19 May 2026 22:38:20 UTC (268 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。