Abstract:Reinforcement learning has become a powerful approach for enhancing large language model reasoning, but faces a fundamental dilemma: training on easy problems can cause overfitting and pass@k degradation, while training on hard problems often results in sparse rewards. Recent question augmentation methods address this by prepending partial solutions as hints. However, uniform hint provision may introduce redundant information while missing critical reasoning bottlenecks, and excessive hints can reduce reasoning diversity, causing pass@k degradation. We propose \textbf{PieceHint}, a hint injection framework that strategically identifies and provides critical reasoning steps during training. By scoring the importance of different reasoning steps, selectively allocating hints based on problem difficulty, and progressively withdrawing scaffolding, PieceHint enables models to transition from guided learning to independent reasoning. Experiments on six mathematical reasoning benchmarks show that our 1.5B model achieves comparable average performance to 32B baselines while preserving pass@k diversity across all $k$ values.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.15830 [cs.LG] |
| (or arXiv:2604.15830v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.15830 arXiv-issued DOI via DataCite (pending registration) |
From: Yangyi Fang [view email]
[v1]
Fri, 17 Apr 2026 08:34:51 UTC (1,263 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。