Abstract:Learning hierarchical features in Sparse Autoencoders (SAEs) is essential for capturing the structured nature of real-world data and mitigating issues like feature absorption or splitting. Existing works attempt to identify hierarchical relationships within independent feature sets by relying on activation coverage, the assumption that child feature should only activate when its parent feature activates. However, we demonstrate that this condition alone is insufficient; that is, it often produces false positives where parent and child concepts are semantically unrelated. To address this, we introduce a novel reconstruction condition that enforces a deeper functional link between hierarchical levels. By combining both activation and reconstruction constraints, we propose the Tree SAE, a model designed to learn hierarchical structures directly from within the feature set. Our results demonstrate that Tree SAEs significantly surpass the existing SAEs at learning hierarchical pairs while maintaining competitive performance to the state-of-the-art on several key benchmarks. Finally, we demonstrate the practical utility of our Tree SAE in mapping the geometry of child feature subspaces and uncovering the complex hierarchical concept structures encoded within large language models.
| Comments: | 21 pages |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.07922 [cs.LG] |
| (or arXiv:2605.07922v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07922 arXiv-issued DOI via DataCite (pending registration) |
From: Tue Minh Cao [view email]
[v1]
Fri, 8 May 2026 15:57:37 UTC (5,500 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。