The paper’s closing account. Standing at the two-layer toy model and looking back, the gains and the remainder line up side by side.
What was secured
For attention-only models, the results are clear. A way of seeing (the residual stream and virtual weights), a unit of analysis (QK/OV), a computational tool (path expansion). And evidence that the frame actually works: the induction head. Cases like the skip-trigram bug, where an error was predicted from structure, show that this understanding is not post-hoc storytelling but understanding with predictive power.
Above all, a vocabulary now exists. Interpretability research after this paper speaks in its terms.
And the gains are not confined to toy models. Ordinary transformers also contain circuits mediated primarily by attention, because even with MLPs present, attention heads still operate on the residual stream and interact directly with each other and with the embeddings. We may not understand the whole model, but those portions can be reverse engineered right now with the tools at hand.
The wall: MLPs
The list of what remains effectively converges to one item: the MLP layers. Two thirds of a standard transformer’s parameters live in MLPs, and this paper’s clean analysis stood on the attention-only simplification. The situation is worse than that, the paper adds: since many attention heads interact with the MLP layers, the share understandable without them is even smaller than the remaining third.
MLPs are hard because of nonlinearity. The power of path expansion came from linearity. The model could be split into a sum of independent terms because the residual stream and attention’s weighted sums are all linear. The MLP’s nonlinear activation blocks exactly this decomposition.
There is an empirical difficulty too. MLP neurons rarely show the tidy picture of one neuron per concept. Polysemanticity, one neuron responding to several meanings, is common. The team’s later answer to why is the superposition hypothesis, and the paper that digs into it is Toy Models of Superposition: the next paper this series will read.
Figure 1. The structure of an MLP layer. Reading and writing are still linear, but the GeLU nonlinearity in the middle blocks path expansion.
The announced next steps
The paper leaves two directions open.
One extends upward: checking whether the induction head found at two layers really is the mechanism of in-context learning in genuinely large models. That question becomes the following year’s In-context Learning and Induction Heads paper.
The other extends sideways: building theory that can handle MLPs. The superposition hypothesis, and the dictionary-learning-based feature decomposition (SAE) work years later, sit on this branch.
How to read this section
Read it as the interim report of a research program. Not the proclamation of a finished theory, but an honest record of how far things got and which wall stands ahead. And because the wall is named precisely, it is clear what the follow-up papers are attacking. A good research program produces not only answers but the next questions.
Takeaways
- For attention-only models, predictive understanding and a standard vocabulary were secured.
- The remaining wall is the MLP: nonlinearity blocks path expansion, and polysemanticity blocks neuron-level reading.
- Large-model verification of induction heads and the superposition hypothesis are announced as the next papers.
The last post covers Related Work: the research landscape this paper stands on.
Source: the Where Does This Leave Us? section of A Mathematical Framework for Transformer Circuits. The figure in this post is from the original paper.