Abstract:Reinforcement learning with verifiable rewards (RLVR) has become the mainstream technique for training LLM agents. However, RLVR highly depends on well-crafted task queries and corresponding ground-truth answers to provide accurate rewards, which requires significant human effort and hinders the scaling of RL processes, especially in agentic scenarios. Although a few recent works explore task synthesis methods, the difficulty of generated agentic tasks can hardly be controlled to provide effective RL training advantages. To achieve agentic RLVR with higher scalability, we explore self-play training for deep search agents, in which the learning LLM utilizes multi-turn search engine calling and acts simultaneously as both a task proposer and a problem solver. The task proposer aims to generate deep search queries with well-defined ground-truth answers and increasing task difficulty. The problem solver tries to handle the generated search queries and output the correct answer predictions. To ensure that each generated search query has accurate ground truth, we collect all the searching results from the proposer's trajectory as external knowledge, then conduct retrieval-augmentation generation (RAG) to test whether the proposed query can be correctly answered with all necessary search documents provided. In this search self-play (SSP) game, the proposer and the solver co-evolve their agent capabilities through both competition and cooperation. With substantial experimental results, we find that SSP can significantly improve search agents' performance uniformly on various benchmarks without any supervision under both from-scratch and continuous RL training setups. The code is at this https URL.
| Comments: | Published as a conference paper at the Fourteenth International Conference on Learning Representations (ICLR 2026) |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.18821 [cs.LG] |
| (or arXiv:2510.18821v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.18821 arXiv-issued DOI via DataCite |
From: Yuhang Wen [view email]
[v1]
Tue, 21 Oct 2025 17:19:35 UTC (1,332 KB)
[v2]
Fri, 26 Dec 2025 14:47:24 UTC (1,811 KB)
[v3]
Tue, 19 May 2026 07:37:46 UTC (2,303 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。