Abstract:Multimodal sentiment analysis (MSA) aims to understand human emotions by integrating information from multiple modalities, such as text, audio, and visual data. However, existing methods often suffer from spurious correlations both within and across modalities, leading models to rely on statistical shortcuts rather than true causal relationships, thereby undermining generalization. To mitigate this issue, we propose a Multi-relational Multimodal Causal Intervention (MMCI) framework, which leverages the backdoor adjustment from causal theory to address the confounding effects of such shortcuts. Specifically, we first model the multimodal inputs as a multi-relational graph to explicitly capture intra- and inter-modal dependencies. Then, we apply an attention mechanism to separately estimate and disentangle the causal features and shortcut features corresponding to these intra- and inter-modal relations. Finally, by applying the backdoor adjustment, we stratify the shortcut features and dynamically combine them with the causal features to encourage MMCI to produce stable predictions under distribution shifts. Extensive experiments on several standard MSA datasets and out-of-distribution (OOD) test sets demonstrate that our method effectively suppresses biases and improves performance.
| Comments: | Corrected several hyperparameter settings. Updated some experimental results |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2508.04999 [cs.LG] |
| (or arXiv:2508.04999v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2508.04999 arXiv-issued DOI via DataCite |
From: Menghua Jiang [view email]
[v1]
Thu, 7 Aug 2025 03:24:04 UTC (716 KB)
[v2]
Wed, 20 May 2026 05:14:50 UTC (733 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。