Abstract:Reward models in RLHF are trained to score only the final token of a response - a choice that discards rich signal from every intermediate position and produces models whose token-level outputs are noise. We argue this is a missed opportunity: a well-trained reward model's output at any token should represent the conditional expectation of the final reward given the response so far. We introduce Temporally Coherent Reward Modeling (TCRM), which induces this property via two regularization terms on top of the standard Bradley-Terry loss, with minimizers provably equal to conditional expectations. The regularizers correspond to Monte Carlo and TD value-learning objectives, establishing a direct connection to RL value functions. TCRM requires zero changes to architecture, data, or inference, yet unlocks three capabilities from one principle: interpretable token-level reward trajectories (middle-token pairwise accuracy improved from 50% to 88.9%, final-token accuracy preserved); state-of-the-art PRM performance on ProcessBench (44.9% average F1) among models trained only on outcome data; and unified reward/value modeling in PPO, reducing peak GPU memory by 27% and step time by 19% with matching LLM quality.
| Comments: | 27 pages, 14 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.22981 [cs.LG] |
| (or arXiv:2604.22981v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.22981 arXiv-issued DOI via DataCite (pending registration) |
From: Alex Nikulkov [view email]
[v1]
Fri, 24 Apr 2026 19:49:56 UTC (373 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。