Abstract:Direct Preference Optimization (DPO) has emerged as a cornerstone of reinforcement learning from human feedback (RLHF) due to its simplicity and efficiency. However, existing DPO-based methods typically treat all preference pairs equally, overlooking substantial variations in data quality and learning difficulty, which leads to inefficient data utilization and suboptimal performance. To address this limitation, we propose Uni-DPO, a unified dynamic preference optimization framework that jointly considers (a) the inherent quality of preference pairs and (b) the model's evolving performance during training. By adaptively reweighting samples based on both factors, Uni-DPO enables more effective use of preference data and achieves superior performance. Extensive experiments across models and benchmarks demonstrate the effectiveness and generalization of Uni-DPO. On textual tasks, Gemma-2-9B-IT fine-tuned with Uni-DPO surpasses the leading LLM, Claude 3 Opus, by 6.7 points on Arena-Hard. On mathematical and multimodal tasks, Uni-DPO consistently outperforms baseline methods across all benchmarks, providing strong empirical evidence of its effectiveness and robustness.
| Comments: | Accepted by ICLR 2026. Code & models: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2506.10054 [cs.LG] |
| (or arXiv:2506.10054v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.10054 arXiv-issued DOI via DataCite |
From: Senqiao Yang [view email]
[v1]
Wed, 11 Jun 2025 17:58:05 UTC (7,841 KB)
[v2]
Fri, 15 Aug 2025 15:40:50 UTC (7,320 KB)
[v3]
Wed, 11 Feb 2026 15:06:07 UTC (7,048 KB)
[v4]
Sat, 23 May 2026 09:10:36 UTC (7,048 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。