Abstract:Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.21883 [cs.CL] |
| (or arXiv:2605.21883v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21883 arXiv-issued DOI via DataCite |
From: Chengyu Huang [view email]
[v1]
Thu, 21 May 2026 01:43:09 UTC (364 KB)
[v2]
Tue, 26 May 2026 03:18:58 UTC (366 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。