Abstract:Diffusion policy sampling enables reinforcement learning (RL) to represent multimodal action distributions beyond suboptimal unimodal Gaussian policies. However, existing diffusion-based RL methods primarily focus on offline settings for reward maximization, with limited consideration of safety in online settings. To address this gap, we propose Augmented Lagrangian-Guided Diffusion (ALGD), a novel algorithm for off-policy safe RL. By revisiting optimization theory and energy-based model, we show that the instability of primal-dual methods arises from the non-convex Lagrangian landscape. In diffusion-based safe RL, the Lagrangian can be interpreted as an energy function guiding the denoising dynamics. Counterintuitively, direct usage destabilizes both policy generation and training. ALGD resolves this issue by introducing an augmented Lagrangian that locally convexifies the energy landscape, yielding a stabilized policy generation and training process without altering the distribution of the optimal policy. Theoretical analysis and extensive experiments demonstrate that ALGD is both theoretically grounded and empirically effective, achieving strong and stable performance across diverse environments.
| Subjects: | Machine Learning (cs.LG); Systems and Control (eess.SY) |
| Cite as: | arXiv:2602.02924 [cs.LG] |
| (or arXiv:2602.02924v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.02924 arXiv-issued DOI via DataCite |
From: Xiaoyuan Cheng [view email]
[v1]
Mon, 2 Feb 2026 23:53:53 UTC (7,566 KB)
[v2]
Tue, 5 May 2026 18:05:36 UTC (3,789 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。