Abstract:Recent advances in sparse reward policy gradient methods have enabled effective reinforcement learning (RL)-based language model post-training. However, for reasoning tasks such as mathematical problem solving, binarized outcome rewards provide limited feedback on intermediate reasoning steps. While some studies have attempted to address this issue by estimating overall reasoning quality, it remains unclear whether these rewards are reliable proxies for the quality of stepwise reasoning. In this study, we consider reasoning as a structured process and propose TACReward, the reward model that can be seamlessly integrated into sparse reward policy gradient methods without additional human annotation costs or architectural modifications. TACReward aggregates stepwise structural deviations between teacher and policy reasoning using process mining techniques, producing a scalar output reward range of [0, 1] to indicate reasoning quality. Experiments on multiple mathematical reasoning benchmarks demonstrate that integrating the TACReward into sparse reward frameworks encourages the policy model to improve the structural quality of reasoning. Consequently, this leads to consistent performance improvements over existing sparse reward frameworks. Our code and checkpoints are publicly available at this https URL and this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2510.25065 [cs.AI] |
| (or arXiv:2510.25065v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2510.25065 arXiv-issued DOI via DataCite |
From: Yongjae Lee [view email]
[v1]
Wed, 29 Oct 2025 01:07:45 UTC (332 KB)
[v2]
Tue, 27 Jan 2026 02:10:32 UTC (6,293 KB)
[v3]
Sun, 24 May 2026 05:29:01 UTC (6,310 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。