



























Abstract:Preference-based reinforcement learning (RL) is a key paradigm for aligning policies with human judgments, yet its theoretical behavior in distributed settings where preference data are fragmented across heterogeneous users remains poorly understood. Direct Preference Optimization (DPO) avoids explicit reward modeling but lacks convergence guarantees under federated and decentralized training, where communication constraints and non-IID preferences fundamentally alter optimization dynamics. We provide the first convergence and time-complexity analysis of DPO in distributed environments. Modeling personalized offline RL with user-specific preference distributions, we characterize the induced global optimization landscape. For federated DPO, we derive convergence rates that quantify the impact of client drift, communication frequency, and preference heterogeneity; for decentralized DPO, we establish convergence over general communication graphs and show how spectral connectivity governs optimization speed and consensus. Empirically, we corroborate our theoretical insights on standard alignment benchmarks, demonstrating that our proposed methods not only enjoy strong theoretical guarantees but also deliver robust and scalable performance in practice. The code base is available here.
| Comments: | 29 pages, 12 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.20696 [cs.LG] |
| (or arXiv:2605.20696v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.20696 arXiv-issued DOI via DataCite (pending registration) |
From: Zhanhong Jiang [view email]
[v1]
Wed, 20 May 2026 04:49:13 UTC (280 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。