Can dictionary learning scale to production language models?

Eight months after sparse autoencoders recovered monosemantic features from a one-layer transformer, the open question was whether the method scales — if it cannot reach state-of-the-art models, it cannot contribute to safety. This work answers it: dictionary learning extracts high-quality features from Claude 3 Sonnet, a medium-sized production model. The approach rests on two hypotheses worth stating because they are load-bearing: the linear representation hypothesis (concepts are directions in activation space) and the superposition hypothesis (networks use almost-orthogonal directions to pack more features than dimensions). Sparse autoencoders are the dictionary-learning approximation that exploits this.

The recovered features are notable on three dimensions. They are abstract — features for famous people, countries and cities, type signatures in code. They are multilingual and multimodal — the same feature responds to a concept across languages and in both text and images. And they span abstract and concrete instantiations of one idea (code with security vulnerabilities and abstract discussion of security vulnerabilities fire the same feature). Most consequentially, the features are not merely correlational: they both respond to and behaviorally cause the relevant behaviors — clamping a feature steers the model.

The significance for the vault is that interpretability is tractable at production scale, not just in toy models — a precondition for any feature-level safety or steering work. It sits against Do standard analysis methods hide nonlinear features in neural networks?: SAEs recover an impressive feature diversity, but that caution remains live — what dictionary learning surfaces may still over-represent the linearly accessible.