Picture a tree.
Now make it small, a sapling; then tall, an oak; then enormous, a redwood; then tiny again, a bonsai. Turn its leaves summer green, then autumn red, then strip them bare. Change its species: birch, pine, willow. Each one is still a tree, but they differ along many axes of variation. The concept "tree" is not a one-dimensional idea, but a multidimensional space of variation.Even before the emergence of LLMs, many have argued we should think of concepts as regions in structured geometric spaces, e.g. Peter Gärdenfors in Conceptual Spaces: The Geometry of Thought.
Trees are just one example here; our world is full of intricate structure. So are the internals of neural networks! But the most popular methods we use to understand what's happening inside neural networks, like sparse autoencoders (SAEs) and transcoders, treat each internal concept as a single straight line – an assumption which is surprisingly powerful, but which nonetheless doesn't effectively capture the naturally curved, multidimensional structures we know exist in model representations.
In this post, we explain our new paper which introduces a family of techniques – Block-Sparse Featurizers (BSF) – that we think provide a simple, viable alternative. BSFs decompose model activations into subspaces, which empirically contain multidimensional shapes.
We believe this is a significant step forward in understanding neural networks on their own terms. Applied to vision models, we find that BSFs find interpretable, multidimensional features which better explain model activations; that those features enable fine-grained steering; and that most concepts in the models are multidimensional.
As part of interpretability's broader goal of understanding what goes on inside neural networks, we want to know what a model is "thinking" at any given moment. The primary way we do thisBut not the only way! E.g., see our work on decomposing weights. is by reading the activations flowing through the model and mapping them to meaningful concepts. In their raw form, though, those activations are just large vectors with no convenient labels. So how do we disentangle them into meaningful patterns?
That question can be reframed as a different one: how are a model's representations structured? If we know that, then we can choose the best method to disentangle them.
If you've been following this series of posts, you can probably guess where this is going. We've discussed the growing body of research, from Goodfire as well as many others, showing that concepts are often represented inside neural networks as manifolds – multidimensional, curved shapes. We've also discussed the mismatch between these complex representations and the tools we use to find them: the most popular methods, like SAEs and transcoders, look only for straight lines.This straight-line assumption has gotten SAEs quite far! We want the simplest possible building blocks for what a network computes, and a straight line is about as simple as a shape can be. Straight lines are also easy to work with computationally. But we think it's a fundamentally limiting assumption, if we want to fully understand model activations.
To get an intuition for the consequences of this mismatch, consider the toy model above. We constructed clean artificial activations by summing samples from predefined manifolds – implicitly stacking the manifolds in superposition. This forms a simplified model of how we think real neural network activations are structured. Then we use two different featurizer methods to try to recover the known underlying structure from the activations.
On the left are the ground-truth manifolds, which are what we're aiming to recover. In the middle column, we see that a standard SAE does a poor job of finding the underlying manifolds!In our previous work, we showed SAEs don't lose manifold geometry so much as fragment it by tiling manifolds across many redundant features that only post-hoc analysis (with an Ising-model and extra compute) can regroup. In contrast, BSFs recover these subspaces coherently during training, even when the SAE is given the advantage of selecting its best top-g directions against a single BSF block of size g.
But if we instead begin with the assumption that model activations are made of manifolds, that implies a different strategy, whose results are on the right. A Block-Sparse Featurizer (BSF) – the method this post describes – finds the ground-truth structures with much higher fidelity!In the full paper, we test not only an SAE and vanilla BSF, but also the two other variants of BSF we propose, as well as two concurrent featurizer methods (MFA and SMixAE). Applying the block-sparsity principle allows us to greatly improve the ability of MFA and SMixAE to faithfully recover our toy manifolds!
So: how do BSFs work, and why are they effective at finding manifolds?
We want a method that respects neural geometry. So instead of looking for one-dimensional lines as the fundamental "atoms" of neural computations, we look for multidimensional subspaces. Since each subspace is multidimensional, it can far better capture the shape taken by a concept's representations.
How do we disentangle these subspaces? The key insight is that a block – the set of directions that span a subspace – should tend to "fire" together on the same input. Unlike SAEs, which assume that a small (sparse) number of 1D features are firing at any given time, BSFs assume that a small (sparse) number of blocks are firing at any given time. That is, we enforce sparsity at the block level.
This principle is what we call block-sparsity, and it is not new: it belongs to the older family of structured sparsity, a long-studied topic in dictionary learningE.g., Yuan & Lin, 2006; Eldar & Bolksei, 2009; Jenatton et al., 2009..
In the course of this research, we found that the principle of block-sparsity matters more than the particular implementation. We built several methods that follow the general principle, yet differ in how they impose that structure,Though removing the ReLU turns out to matter across all of them! See the paper for more details. and every one of them recovers concept manifolds. The same principle even lets us improve several existing methods, as we show in the full paper.
The diagram below shows the architecture of a generic BSF:
We propose three different BSF variants: vanilla BSF, Grassmannian BSF, and group-lasso BSF. All three share the generic architecture in the diagram above, but they differ in their choice of encoder/decoder and activation function. For example, the vanilla BSF uses a standard linear encoder $\mathbf{xW} + \mathbf{b}$ and a block-wise TopK (based on the L2 norm of each block) as its activation function. All BSF methods we propose are trained to minimize reconstruction error. For more details on the three variants and their theoretical grounding, see the full paper.
The real test of any featurizer is the quality of the features that it recovers from models that are actually used in production: do the features correspond to coherent concepts? How well do they explain the model's activations? And what do the features allow you to do with the model?
We trained BSFs on two off-the-shelf open-source models: a vision model, DINOv3, and a text-to-image model, Stable Diffusion XL (SDXL). Unlike in the toy model from earlier, the structure of the model's representations is not known in advance.
Quantitatively, we find that BSFs yield a more faithful and efficient account of the model: they better explain the original model activations with a lower minimum description length.See sections 3.2 and 3.3 of the paper for more. Qualitatively, the features look quite interesting!
In these vision models, the activations of a BSF block rarely form a line, instead forming dense, connected clouds. A standard SAE squashes each cloud down to a single direction, shown as the black lines in the figure below. A BSF keeps the whole cloud instead, treating that entire region as the concept.
This shows up clearly in DINOv3. Take the "arch feature" below: the BSF finds a region with interpretable internal structure. Moving through the region traces out the different parts of an arch. Whether in a viaduct or a church's nave arcade, one part of the region (visualized in red) activates on the bottom of an arch, smoothly transitioning to another part of the region (visualized in green) which activates across the top of an arch.
Because a BSF block gives us a region of activations corresponding to a concept, rather than a single point, we can move around inside it. This lets us steer along the concept, sweeping through its internal variations rather than turning a one-dimensional knob up or down.
We can see this directly by laying a grid over the region and walking from point to point. Each step moves us through the range of examples that belong to one concept, so we can explore the space and watch the many kinds of pretzels the model can draw: the "pretzel manifold".
Below, we show two more examples, simplifying the multidimensional structure into a 2D grid. We can see high-level concepts varying smoothly across tiles: the hat's brim angle, crown shape, and orientation, or the coffee's milk pattern, spoon location, or cup design.
We now revisit a feature the interpretability community has studied for years, using BSFs to examine it from a new angle.
Curve detectors in InceptionV1 are a classic case study in interpretability. Cammarata et al. (2020)Cammarata et al., "Curve Detectors," Distill (2020). showed that a small group of neurons in the model's early layers acted as "orientation detectors", meaning each neuron specialized in recognizing one particular curve orientation. Building on this, Gorton (2024)Gorton, "The Missing Curve Detectors of InceptionV1" (2024). found curve detectors that were "missing", meaning they had no neuron associated with them, and showed that SAE features could recover them.
Using our BSFs, we showed that both the neurons and the SAE features actually give us a fragmented view of a continuous space that smoothly captures the rotation of curves!
But it goes deeper than that: beyond recovering orientation as a continuum in a single subspace, the BSF also surfaces other features in that same block, which turn out to be higher-order Fourier harmonics - a property of the representation that has not been documented before. These harmonics are symmetries the model has learned: the first harmonic encodes the full 360 degrees of possible orientations (peaking at 0°), the second activates for curves that are the same when flipped around (peaking at 0° and 180°), and so on for the higher harmonics.
This also explains an observation from Olah et al. (2020)Olah et al., "An Overview of Early Vision in InceptionV1," Distill (2020).: certain families of curve-detecting neurons wrap around every 180° instead of a full 360°. It turns out these neurons are simply reading off the second harmonic we found. So even in a well-studied setting, and without targeting it directly, block-sparsity had something new to reveal.
With a featurizer that looks for subspaces rather than single directions, we can ask a question that was hard to even approach before: how often are concepts actually multidimensional in neural network activations?
In vision models, at least, most concepts appear to be multidimensional – and specifically, two- to four-dimensionalAverage of stable rank across 32,000 concepts found in DINOv3. See section 3.4 of the paper for more detail.. That is perhaps less surprising than it sounds, since the scenes these images capture are themselves projections of the three-dimensional world. We would not call this a final answer, but it may be the first time the question admits something close to one.
Our earlier work kept finding concepts with rich, multidimensional structure, which our tools struggled to capture. So we adjusted the tools to match the geometry we were seeing. The new featurizer does what it was designed to do, recovering most concepts as multidimensional representations. We believe the principle of block-sparsity matters more than any single way we implement it, and we expect sharper implementations of it to come.
Other settings will likely call for their own variants. For example, vision-language-action models (used in robotics) lay their activations out along a temporal trajectory. Each domain may need a featurizer shaped to its own geometry, and block-sparsity is unlikely to be the right prior everywhere.
That is really the broader point. A featurizer is a hypothesis about how a model's representations are structured, and part of the job of interpretability is to work out what those building blocks should be in the first place. Block-Sparse Featurizers are one answer to this question, and one step in a longer agenda of reading the geometry of a model and letting it tell us what the next answer should be.
For prior work, theoretical grounding, more experiments, and details on training BSFs, read the full paper. Our code to train BSFs is available on GitHub.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。