Abstract:Variational Autoencoders (VAEs) often exhibit a polarised regime in which latent variables separate into active, passive, and mixed subsets. Existing criteria for identifying active dimensions depend on a Gaussian prior, limiting their applicability to variational models and specific priors. We propose a simple information-theoretic classification of the polarised regime based on the entropy of the mean representation. We show theoretically how this entropy couples to KL minimisation through entropy--variance bounds, and we relate the resulting criterion to Bonheme's active/passive conditions. We also clarify a key limitation: entropy of the mean alone cannot reliably distinguish active from mixed dimensions without additional signals from the variance representation. Empirically, we evaluate the entropy criterion on $\beta$-VAEs, identifiable VAEs, Least-Volume Autoencoders, and L2-regularised autoencoders, and find that it consistently recovers a polarised regime when such a regime is present across the model classes studied. Finally, we show that passive dimensions can yield small but consistent improvements on downstream tasks when latent codes are appropriately normalised, suggesting that collapse is often a matter of scale rather than absolute information removal.
| Comments: | 13 pages, 17 figures, under review at Neural Networks |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.15965 [cs.LG] |
| (or arXiv:2605.15965v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15965 arXiv-issued DOI via DataCite (pending registration) |
From: Peter Clapham [view email]
[v1]
Fri, 15 May 2026 13:55:55 UTC (999 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。