This is the climax of the paper. The moment there are two attention layers, something becomes possible that was impossible in principle at one layer: the output of an earlier head becoming the input of a later head. Composition.
Three kinds of composition
Recall the picture from part 2. Heads read from and write to a shared board, the residual stream. With two layers, something new becomes possible: what an earlier head wrote on the board, a later head can read off it. And as we saw in part 2, a later head reads from the board in three places: when it builds its search query, when it builds its index (key), and when it builds its contents (value). Depending on which of the three the earlier result flows into, composition comes in three kinds.
- Q-composition: the earlier output influences the later head’s query. Where to look now depends on the earlier layer’s computation.
- K-composition: the earlier output influences the later head’s keys. How much each position gets attended to now depends on the earlier layer’s computation.
- V-composition: the earlier output influences the later head’s values. The moved content itself is something the earlier layer processed.
Applying path expansion to two layers multiplies the terms: the direct path, paths through a single head, and paths through an earlier head then a later head in sequence. In particular, the term for two heads chained by V-composition collapses to the form $(A^{h_2} A^{h_1}) \otimes (W_{OV}^{h_2} W_{OV}^{h_1})$: pattern multiplied with pattern, OV multiplied with OV, effectively one new head. The paper calls these virtual attention heads. Why only V-composition folds up like this can be said in words too: composing information movement with information movement is still information movement. Q- and K-composition instead change the attention pattern itself, creating something new that does not reduce to a single head.
Figure 1. Path expansion of the two-layer logits. The direct path term handles bigrams, the individual head terms match the one-layer case, and the final virtual attention head terms correspond to V-composition.
The induction head
Analyzing trained two-layer models, one clear pattern that actually uses composition shows up again and again: the induction head.
Its job fits in one rule. If [A][B] appeared in the context and the current token is [A] again, predict [B] next. Continue the pattern seen before. If the token it is looking for is not in the context, it attends to the start token and does nothing: a kind of resting position.
Figure 2. Attention pattern and logit effect of induction head 1:8 on the first paragraph of Harry Potter. The current token’s query attends to the key of the token right after the previous occurrence, and pushes up that token’s logit.
The mechanism is a collaboration of two heads, and the crux is K-composition. Let’s follow it on a concrete case. “Harry Potter” appeared in the context, and the current token is “Harry” again. The next token should be “Potter.”
- A previous-token head in the earlier layer does the preparation: at each position, it reads the token just before and writes it into the residual stream. The position of “Potter” now carries a sticky note saying “the token before me was Harry.”
- The induction head in the later layer builds its index (key) from that note. This is K-composition. So the index at “Potter”’s position becomes “a place whose predecessor was Harry.” The search query of the current token “Harry” looks for exactly such a place, and attention locks onto “Potter.”
- Its OV circuit is plain copying. It pushes the attended “Potter” up as the next-token prediction.
Generalized: [A][B] … [A] → [B]. In part 2’s language, the QK has become a much smarter search thanks to the earlier layer’s notes, finding “the spot right after my previous occurrence,” while the OV is part 4’s copying, unchanged. Not a new part; a new combination.
Figure 3. The QK circuit expanded over tokens instead of heads. Key and query intensity show how much each token raises the attention score. Because the key is computed from tokens shifted one back, a query searching for the same token lands on the position right after it.
Compare with the one-layer copying head and the qualitative difference is visible. One-layer copying says “tokens that appeared tend to appear again.” The induction head implements “patterns that appeared continue,” and it does not care what the pattern is. It works even on repeated sequences of completely random tokens, input far off the training distribution. It has not memorized a statistic; it has learned an algorithm that reads rules out of the context.
The paper verifies the theory from two directions. One is weight verification. If the algorithm is right, an induction head’s OV circuit must be a copying matrix, and the QK term created by K-composition must be a matching matrix that looks for the same token. Measuring both axes with last post’s eigenvalue yardstick, the induction heads alone cluster in the extreme positive-positive corner. The other is ablation. Term importance analysis, which deletes expansion terms order by order, shows that the large share of performance sits in the second-layer head terms while the virtual-head terms contribute little.
Figure 4. Second-layer heads placed by OV-circuit eigenvalue positivity (x) and the K-composed QK term’s positivity (y). Only the induction heads gather in the top-right corner, extreme in both copying and matching.
Composition is sparser than you’d think
One empirical finding to note. In their two-layer model, composition was not everywhere but sparse. Most heads barely compose and compute skip-trigrams as if they were in a one-layer model. The model as a whole largely behaves like a big one-layer model with induction heads on top. The new capability arrived not as a flood filling the layer, but as a few thin streams.
One erratum. After publication, a bug was found in the authors’ library used for this composition measurement, and a corrected figure was added to the document. The real model had more composition than it first appeared, and some induction heads also composed with a head that looks at the last several tokens, not just the previous-token head. The core conclusion stands, though: K-composition with a previous-token head remains the backbone of induction heads. The thread’s culture of attaching claims and corrections to the same document returns in the final post.
Figure 5. The erratum’s composition diagrams: the buggy original above, the corrected version below (teal: induction heads, red: previous token heads). More composition appears in the corrected plot, but the K-composition backbone stands. Line widths are not directly comparable between the two.
The same goes for virtual attention heads. In theory, virtual heads can exist in numbers multiplying across head counts, a potential source of enormous expressivity in large models. One hint the paper offers: heads attending to the previous token are common, while heads attending two tokens back are rarely found, perhaps because that predictive power can be had by virtually composing two previous-token heads. And for jobs that move very little information, like person or tense, a virtual head is more economical than a full-sized regular head. But in the small models this paper analyzes, the clearly large role belongs to induction heads, and the virtual heads’ share was small. The paper’s care in separating signs of possibility from confirmed fact is worth learning from.
Takeaways
- From two layers on, Q-, K-, and V-composition become possible, and V-composed paths become new terms called virtual heads.
- The induction head implements [A][B] … [A] → [B] through K-composition with a previous-token head.
- This is an in-context learning algorithm that works on arbitrary patterns, and it becomes the central subject of the later large-model work.
- Observed composition is sparse, though: the two-layer model largely behaves like a big one-layer model plus induction heads.
The next post is the paper’s closing account: how far this framework got, and what remains.
Source: the Two-Layer Attention-Only Transformers section of A Mathematical Framework for Transformer Circuits. All figures in this post are from the original paper.