Abstract:LLMs are increasingly used to make or support high-stakes decisions under uncertainty, where alignment depends not only on factual accuracy but on how models weigh tradeoffs between different outcomes. We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize: we elicit the model's probability distribution over unknowns along with the choice it would make for the decision task and then fit a discrete choice model to recover the cost function that best rationalizes the model's decisions. We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way, whether they can verbalize a description of their objectives which matches their revealed decision policy, and whether prompting can reliably steer those policies to implement a user-specified cost function. We apply this evaluation across four medical diagnosis domains and multiple frontier and open-source models. We find that while many models have a nontrivial degree of internal coherence, they also have significant weaknesses in faithfully reporting or adopting preferences in response to user direction.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.08556 [cs.LG] |
| (or arXiv:2605.08556v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.08556 arXiv-issued DOI via DataCite (pending registration) |
From: Khurram Yamin [view email]
[v1]
Fri, 8 May 2026 23:26:35 UTC (1,207 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。