
























Abstract:Reinforcement learning (RL) finetuning has become a key technique for enhancing large language models (LLMs) on reasoning-intensive tasks, motivating its extension to vision-language models (VLMs). While RL-tuned VLMs improve on visual reasoning benchmarks, they remain vulnerable to weak visual grounding, hallucinations, and over-reliance on textual cues. We show that simple, controlled textual perturbations, including misleading captions or incorrect chain-of-thought (CoT) traces, cause substantial drops in robustness and confidence, and that these effects are more pronounced when CoT consistency is taken into account across open-source multimodal reasoning models. In contrast, closed models exhibit similar failure modes but maintain markedly greater robustness and reasoning consistency, suggesting that the gap reflects a shortcoming in current open-source RL finetuning rather than an inherent limitation of the task. To better understand these vulnerabilities, we further analyze RL finetuning dynamics and uncover an accuracy-faithfulness trade-off: finetuning raises benchmark accuracy, but can simultaneously erode the reliability of the accompanying CoT and its robustness to contextual shifts. Although adversarial augmentation improves robustness, it does not by itself prevent faithfulness drift. Incorporating a faithfulness-aware reward can restore alignment between answers and reasoning, but when paired with augmentation, training risks collapsing onto shortcut strategies and robustness remains elusive. Together, these findings highlight the limitations of accuracy-only evaluations and motivate training and assessment protocols that jointly emphasize correctness, robustness, and the faithfulness of visually grounded reasoning.
| Comments: | ICML 2026 |
| Subjects: | Machine Learning (cs.LG) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2602.12506 [cs.LG] |
| (or arXiv:2602.12506v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.12506 arXiv-issued DOI via DataCite |
From: Rosie Zhao [view email]
[v1]
Fri, 13 Feb 2026 01:12:00 UTC (20,969 KB)
[v2]
Sat, 14 Mar 2026 17:24:06 UTC (19,936 KB)
[v3]
Thu, 21 May 2026 07:28:16 UTC (22,371 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。