The closing post of the Framework reading. Following the paper’s Related Work section, here is the landscape this work stands on. It is not an isolated invention but a confluence of several lines.
Direct ancestry: Circuits in vision models
The most direct ancestor is the Circuits thread the same authors ran on Distill: reverse engineering features like curve detectors in CNNs such as InceptionV1, and the circuits connecting them. The belief that neural networks can be read at the level of weights, and the methodological attitude, came from there. This paper transplants that program to transformers. The differences between vision and language required new tools, and the results are the residual-stream view and the QK/OV split.
It also leaves an interesting scaling outlook. Attention-only circuits can be read in large, mostly linear chunks: heads. Even very large models have only a few thousand heads, a scale where looking at every single one does not seem implausible. The paper also records a contrast: studying tiny attention-only transformers went far more smoothly than small vision models, where uninterpretable neurons blocked the way.
The lineage of attention analysis
There was already plenty of work peering into transformer attention: studies observing that BERT’s heads tend to track syntactic relations, pruning studies showing most heads can be removed with little performance loss, and the debate over whether attention weights count as explanations.
The difference between that lineage and the Framework is the size of the ambition. Much prior work observes attention patterns and reports statistical tendencies. The Framework tries to read algorithms out of the weights themselves: not what the pattern looks like, but why that pattern gets computed, shown as matrix products. It is the difference between observation and reverse engineering.
On the debate over whether attention weights count as explanations in particular, the framework offers a concrete answer. When a head operates in isolation as a first-order term, its pattern can be read at face value. In higher-order terms where composition is involved, that reading becomes a misreading. The induction head is the object lesson: the pattern alone explains a lot, but without knowing it is built by K-composition with a previous-token head, one completely misunderstands the previous-token head’s role. In effect, the paper provides a typology of when attention is an explanation and when it is not.
The logit lens and the residual-stream view
The community’s logit-lens family of experiments deserves mention too: apply the unembedding early to intermediate residual streams, and you can watch the model’s tentative prediction being refined layer by layer. It shares the intuition of the residual stream as the model’s workspace, and the Framework can be seen as giving that intuition mathematical standing.
A culture of public verification
Not part of the paper body, but worth recording: the Transformer Circuits thread attaches external researchers’ reviews and replications to its documents as Comments & Replications. Keeping claims and their verification on the same page has become a good tradition of the field.
What the series reads next
That concludes the Framework reading. Summarizing the seven posts: a language was built for expanding transformers into sums of paths, the smallest models were read completely in that language, a real mechanism called the induction head was discovered, and the next problem, the MLP, was named precisely.
Next up is the frontal assault on that MLP problem: Toy Models of Superposition.
Source: the Related Work section of A Mathematical Framework for Transformer Circuits.