Abstract:Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs). However, GRPO is prone to advantage collapse, a failure mode where homogeneous rewards within a group (e.g., all correct or all incorrect answers) yield near-zero advantages and vanishing gradients. To address this, we introduce the Advantage Collapse Rate (ACR), the first diagnostic metric quantifying the proportion of training batches with ineffective gradients. Across models from 0.5B to 14B parameters on mathematical reasoning benchmarks, we show that ACR strongly predicts training stagnation and final performance. We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples, guided by real-time ACR monitoring, to enable learning from homogeneous groups without additional model rollouts. AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales, while maintaining generalization on the evaluated out-of-domain task. Code and datasets are available at this https URL.
| Comments: | 26 pages, 12 figures. Accepted at the International Conference on Machine Learning (ICML 2026) |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.21125 [cs.LG] |
| (or arXiv:2605.21125v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21125 arXiv-issued DOI via DataCite (pending registration) |
From: Xixiang He [view email]
[v1]
Wed, 20 May 2026 12:57:37 UTC (2,904 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。