





















Abstract:We focus on the problem of \emph{Answer-Level Fine-Tuning} (ALFT), where the goal is to optimize a language model based on the correctness or properties of its final answers, rather than the specific reasoning traces used to produce them. Directly optimizing answer-level objectives is computationally intractable due to the need to marginalize over the vast space of latent reasoning paths. To overcome this, we propose a general game-theoretical framework that lifts the problem to a \emph{Distributional Alignment Game}. We formulate ALFT as a two-player game between a Policy (the generator) and a Target (an auxiliary distribution). We prove that the Nash Equilibrium of this game corresponds exactly to the solution of the original answer-level optimization problem. This variational perspective transforms the intractable marginalization problem into a tractable projection problem. We demonstrate that this framework unifies recent approaches to diversity and self-improvement (coherence) and provide efficient algorithms compatible with Group Relative Policy Optimization (GRPO), such as Coherence-GRPO, yielding significant complexity gains in mathematical reasoning tasks.
| Subjects: | Machine Learning (cs.LG); Computer Science and Game Theory (cs.GT) |
| Cite as: | arXiv:2604.27166 [cs.LG] |
| (or arXiv:2604.27166v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.27166 arXiv-issued DOI via DataCite (pending registration) |
From: Jonathan Schneider [view email]
[v1]
Wed, 29 Apr 2026 20:15:37 UTC (126 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。