https://3rdlayer.uk/tags/transformer-circuits/Recent content in Transformer Circuits on 3rd layerHugo -- 0.157.0en-USSat, 09 Aug 2025 00:00:00 +0000https://3rdlayer.uk/posts/framework-07-related-work/Sat, 09 Aug 2025 00:00:00 +0000https://3rdlayer.uk/posts/framework-07-related-work/Reading the Framework paper, final part. Whose shoulders this paper stands on, and how its approach differs from the lines of work before it.https://3rdlayer.uk/posts/framework-06-where-does-this-leave-us/Sat, 26 Jul 2025 00:00:00 +0000https://3rdlayer.uk/posts/framework-06-where-does-this-leave-us/Reading the Framework paper, part 6. What the framework secured and what it did not: the wall called MLP, and the follow-up research this paper announces.https://3rdlayer.uk/posts/framework-05-two-layer/Sat, 12 Jul 2025 00:00:00 +0000https://3rdlayer.uk/posts/framework-05-two-layer/Reading the Framework paper, part 5. With two layers, heads compose with heads: Q-, K-, and V-composition, and the paper’s biggest discovery, the mechanism of the induction head.https://3rdlayer.uk/posts/framework-04-one-layer/Sat, 28 Jun 2025 00:00:00 +0000https://3rdlayer.uk/posts/framework-04-one-layer/Reading the Framework paper, part 4. Fully expanding a one-attention-layer model into paths: skip-trigrams, copying heads and positive eigenvalues, and the characteristic bugs the factored structure produces.https://3rdlayer.uk/posts/framework-03-zero-layer/Sat, 14 Jun 2025 00:00:00 +0000https://3rdlayer.uk/posts/framework-03-zero-layer/Reading the Framework paper, part 3. What does a transformer with no attention at all learn? Bigram statistics, and that answer turns out to be the key to the direct path in large models.https://3rdlayer.uk/posts/framework-02-transformer-overview/Sat, 31 May 2025 00:00:00 +0000https://3rdlayer.uk/posts/framework-02-transformer-overview/Reading the Framework paper, part 2. Redrawing the residual stream as a communication channel and attention heads as independent information movers, through virtual weights and the QK/OV split.https://3rdlayer.uk/posts/framework-01-summary/Sat, 17 May 2025 00:00:00 +0000https://3rdlayer.uk/posts/framework-01-summary/Reading the Framework paper, part 1. Treating tiny attention-only models as model organisms for fully reverse engineering transformers, and a map of the key results.https://3rdlayer.uk/posts/mech-interp-resources/Sat, 10 May 2025 00:00:00 +0000https://3rdlayer.uk/posts/mech-interp-resources/Reverse-engineering neural networks, circuit by circuit. Anthropic’s Transformer Circuits thread and a few companion resources are all free. Where to start, to open this series.