

















Abstract:When VLMs answer correctly, do they genuinely rely on visual information? We introduce a Tri-Layer Diagnostic Framework with three per-sample metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score, which disentangle perception, dependency, and alignment failures. Across 9 VLMs and 9,000 model-sample pairs under counterfactual blind, noise, and conflict interventions, 72.9% of samples exhibit Visual Sycophancy, a Split Beliefs pattern in which internal evidence is preserved yet a hallucinated answer is decoded, while zero samples show Robust Refusal, indicating that current alignment training has eliminated refusal as a decoding outcome. Scaling within the Qwen-VL family, both within- and across-generation, monotonically reduces Language Shortcuts but amplifies Visual Sycophancy, showing that scale and newer post-training alone cannot resolve the grounding problem. Diagnostic scores further enable a training-free selective-prediction strategy yielding up to +9.5 percentage points accuracy at 50% coverage.
| Comments: | 14 pages, 1 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.18373 [cs.CV] |
| (or arXiv:2603.18373v3 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2603.18373 arXiv-issued DOI via DataCite |
From: Rui Hong [view email]
[v1]
Thu, 19 Mar 2026 00:15:05 UTC (1,342 KB)
[v2]
Wed, 15 Apr 2026 21:46:07 UTC (1,343 KB)
[v3]
Tue, 26 May 2026 06:21:17 UTC (1,010 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。