Abstract:Foundation models, despite their robust zero-shot capabilities, remain vulnerable to spurious correlations and 'Clever Hans' strategies. Existing mitigation methods often rely on unavailable group labels or computationally expensive gradient-based adversarial optimization. To address these limitations, we propose Visual Disentangled Diffusion Autoencoders (DiDAE), a novel framework integrating frozen foundation models with disentangled dictionary learning for efficient, gradient-free counterfactual generation directly for the foundation model. DiDAE first edits foundation model embeddings in interpretable disentangled directions of the disentangled dictionary and then decodes them via a diffusion autoencoder. This allows the generation of multiple diverse, disentangled counterfactuals for each factual, much faster than existing baselines, which generate single entangled counterfactuals. When paired with Counterfactual Knowledge Distillation, DiDAE-CFKD achieves state-of-the-art performance in mitigating shortcut learning, improving downstream performance on unbalanced datasets.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2601.21851 [cs.LG] |
| (or arXiv:2601.21851v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.21851 arXiv-issued DOI via DataCite |
From: Sidney Bender [view email]
[v1]
Thu, 29 Jan 2026 15:25:37 UTC (11,472 KB)
[v2]
Wed, 6 May 2026 16:03:53 UTC (11,477 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。