Abstract:Recent work has explored optimizing LLM collaboration through Multi-Agent Reinforcement Learning (MARL). However, most MARL fine-tuning approaches rely on predefined execution protocols, which often require centralized execution. Decentralized LLM collaboration is more appealing in practice, as agents can run inference in parallel with flexible deployments. Also, current approaches use Monte Carlo methods for fine-tuning, which suffer from high variance and thus require more samples to train effectively. Actor-critic methods are prevalent in MARL for dealing with these issues; thus, we developed Multi-Agent Actor-Critic (MAAC) methods to optimize decentralized LLM collaboration. In this paper, we analyze when and why these MAAC methods are beneficial. We propose 2 MAAC approaches, \textbf{CoLLM-CC} with a \textbf{C}entralized \textbf{C}ritic and \textbf{CoLLM-DC} with \textbf{D}ecentralized \textbf{C}ritics. Our experiments across writing, coding, and game-playing domains show that Monte Carlo methods and CoLLM-DC can achieve performance comparable to CoLLM-CC in short-horizon and dense-reward settings. However, they both underperform CoLLM-CC on long-horizon or sparse-reward tasks, where Monte Carlo methods require substantially more samples and CoLLM-DC struggles to converge.
| Subjects: | Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2601.21972 [cs.AI] |
| (or arXiv:2601.21972v5 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2601.21972 arXiv-issued DOI via DataCite |
From: Shuo Liu [view email]
[v1]
Thu, 29 Jan 2026 16:50:30 UTC (3,524 KB)
[v2]
Wed, 4 Feb 2026 02:30:21 UTC (3,524 KB)
[v3]
Fri, 13 Feb 2026 23:02:28 UTC (3,524 KB)
[v4]
Sat, 2 May 2026 20:56:56 UTC (3,516 KB)
[v5]
Tue, 26 May 2026 15:41:11 UTC (3,497 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。