Abstract:Multimodal deep neural networks enhance deep comprehension by integrating diverse data modalities. Data from different modalities are typically projected into a shared latent space for similarity computation, but this process is resource intensive due to large image encoders and equal processing of test data during prediction. Early exit methods reduce computational load by utilizing intermediate layers, saving time and memory. However, developing such methods is challenging for multimodal data like image-text pairs. This study investigates the semantic content distributions present in intermediate layers of encoders such as CLIP, which can be derived from textual descriptions. We introduce Text-Guided Exit Modules (T-GEMs) and a rate-based regularizer to control encoder usage costs while maintaining cross-modal understanding performance.
| Comments: | Accepted at ICASSP 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.17499 [cs.LG] |
| (or arXiv:2605.17499v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.17499 arXiv-issued DOI via DataCite (pending registration) |
From: Alberto Presta Mr [view email]
[v1]
Sun, 17 May 2026 15:18:20 UTC (6,734 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。