We start with the smallest transformer: zero attention layers, just an embedding and an unembedding.

The whole model is one table

What this model does is exactly what phone autocomplete does. It sees no context at all; it looks at the one word that just appeared and suggests the next.

As a formula, this is everything:

$$T = W_U W_E$$

A token comes in, gets embedded ($W_E$), and is immediately unembedded ($W_U$) into logits for the next token: a score assigned to each candidate next token. No position, no context, so each token must predict the next from itself alone.

Under that constraint, the best the model can learn is fixed: “given the current token A, the distribution of the next token,” that is, bigram statistics. Feed in “Barack” and out comes a score table like this.

Obama    very high
said     middling
banana   very low

With a 50,000-token vocabulary, $W_U W_E$ is a low-rank compression of that 50,000 x 50,000 table. Open up a trained zero-layer model and exactly this kind of statistic is what you find.

Why this toy matters

The zero-layer model itself is trivial. The real use of this section shows up when looking at large models.

However many layers a model has, the direct path from embedding straight to unembedding, $W_U W_E$, always exists, because the residual stream is a linear conduit. And that direct path is structurally identical to a zero-layer transformer: it cannot see context, and can only push the next token from the current one.

So a natural picture follows. Even the deepest model has one autocomplete built in, and bigrams are its share of the work. The paper chooses its words carefully, though. Other paths also predict parts of the bigram distribution, so what the direct path holds in a large model is not the full bigram table but its residual. Neighbor relations that no general rule like grammar explains, things you simply have to memorize, such as “Obama” after “Barack,” are what remains here. Seen as a sum over paths, the model divides labor by what each path is capable of.

Takeaways

  • A zero-layer transformer is one table ($W_U W_E$): context-free autocomplete, i.e., bigram statistics, and nothing more.
  • The same direct path exists in every deep model and takes the same share of the work.
  • Complete understanding of the smallest model transplants directly into understanding one piece of a large model. The first payoff of the model-organism strategy.

Next post: what changes when one attention layer appears. The answer is skip-trigrams.

Source: the Zero-Layer Transformers section of A Mathematical Framework for Transformer Circuits.