






















Abstract:Sparse autoencoders (SAEs) have been used widely to decompose and interpret neural network activations, especially those of transformer language models. One key issue with SAEs is their inability to directly model multidimensional features. Instead, SAEs may tile such features by a set of independent directions that must be grouped together after the SAE training phase, impeding discoverability and interpretation of learned feature representations. We begin to address this issue by introducing the Sparse MIXture of Autoencoders (SMIXAE) architecture. Empirically, we provide evidence that SMIXAE models have success both in directly learning previously identified manifold structures, as well as finding novel structures, within the open source Gemma 2 2B and 9B models. Finally, we discuss several limitations and point towards areas for future work.
| Comments: | 20 pages, 10 figures, 11 tables. Submitted to Mechanistic Interpretability Workshop, ICML 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.09224 [cs.LG] |
| (or arXiv:2605.09224v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.09224 arXiv-issued DOI via DataCite (pending registration) |
From: Collin Francel [view email]
[v1]
Sat, 9 May 2026 23:48:27 UTC (4,838 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。