Abstract:The advantage function is a central concept in RL that helps reduce variance in policy gradient estimates. For language modeling, Group Relative Policy Optimization (GRPO) was proposed to use the within-group sample mean as a baseline for advantage normalization. This estimator can be sensitive to small group size and rollout-level stochasticity, which may lead to suboptimal advantage estimates in some settings. In this paper, we propose Kalman Filter Enhanced Group Relative Policy Optimization (KRPO), a lightweight variant that treats per-group rewards as noisy observations of a latent prompt-level reward baseline and uses a 1D Kalman filter to estimate both the baseline and its uncertainty. KRPO introduces no additional learned parameters and can be integrated into GRPO with minimal computational overhead. On mathematical reasoning benchmarks, KRPO consistently improves training reward curves and final accuracy over GRPO. These results suggest that adaptive advantage estimation is a promising direction for critic-free reinforcement learning in language model reasoning. The code is available at this https URL.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2505.07527 [cs.LG] |
| (or arXiv:2505.07527v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.07527 arXiv-issued DOI via DataCite |
From: Hu Wang [view email]
[v1]
Mon, 12 May 2025 13:09:49 UTC (2,198 KB)
[v2]
Wed, 21 May 2025 08:49:01 UTC (1,279 KB)
[v3]
Wed, 24 Sep 2025 02:31:07 UTC (1,596 KB)
[v4]
Fri, 30 Jan 2026 09:30:41 UTC (4,836 KB)
[v5]
Tue, 21 Apr 2026 19:04:07 UTC (4,820 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。