Starting this post, I follow Anthropic’s A Mathematical Framework for Transformer Circuits (2021) section by section. This paper is the foundation of the whole Transformer Circuits thread. Toy Models of Superposition and the induction heads paper both stand on the vocabulary defined here.
The paper’s stated motivation is safety. Language models keep revealing capabilities and problem behaviors that even their creators did not know about, years after training. If a model’s internal computations can be read the way a compiled binary is reverse engineered into source code, a systematic approach becomes possible: explaining current safety problems, finding new ones, and perhaps anticipating the problems of powerful future models that have not been built yet.
The strategy: model organisms
Just as biology started with E. coli and fruit flies, this paper starts with the smallest transformers. Attention-only models with the MLPs removed, at zero, one, and two layers. The goal is not performance but complete understanding. If a small model can be reverse engineered to the end, the concepts gained there become the language for looking at large models.
The strategy actually worked. The induction head, discovered in a two-layer toy model, later becomes the leading candidate for explaining in-context learning in large models.
A map of the key results
Here is the paper’s summary in one page. Each item gets its own post later.
The residual stream is a communication channel. Every layer of a transformer reads from the residual stream and adds back into it. The stream itself does no computation: it is a linear conduit. Because of this, even distant layers can be analyzed as if directly wired together, through products called virtual weights.
Attention heads are independent and additive. The heads in a layer look entangled, but mathematically they decompose into independent terms, each computing on its own and adding into the residual stream.
A head splits into two circuits. The QK circuit decides where to attend; the OV circuit decides what to move from the attended position. The two can be analyzed independently. Attention is information movement.
Each added layer brings a qualitatively new capability. A zero-layer model learns only bigram statistics. A one-layer model adds skip-trigrams. From two layers on, head composition becomes possible, and induction heads appear. An induction head repeats patterns seen earlier in the context: the prototype of in-context learning.
The tool is path expansion. Rewrite the whole model as a sum over paths from input to output, and each path becomes one interpretable term. Most of the math in this paper is the work of justifying and using this expansion.
Why read this paper first
Many papers in mechanistic interpretability say “here is what we found.” This paper first decides “here is what to look with.” Residual stream, QK/OV, composition, induction heads: these are now the standard vocabulary of the field. Learn the vocabulary first and the later papers read much faster.
Next post, we redraw the transformer the way this paper does.
Source: A Mathematical Framework for Transformer Circuits (Elhage, Nanda, Olsson et al., Anthropic, 2021).