Abstract:While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.08706 [cs.LG] |
| (or arXiv:2604.08706v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.08706 arXiv-issued DOI via DataCite (pending registration) |
From: Charles Arnal [view email]
[v1]
Thu, 9 Apr 2026 18:56:12 UTC (4,258 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。