A single attention layer, with no help from anything deeper, can already learn a surprising trick: if it saw A somewhere earlier and the current token is B, boost token C — skipping over everything between. ‘keep’ … ‘in’ → ‘mind’. These skip-trigrams are literally all a one-layer attention-only transformer can compute. Slide through them and watch the leap.
Anthropic’s mathematical framework (Elhage et al. 2021) showed a one-layer attention-only transformer implements exactly SKIP-TRIGRAMS: the current token B attends back to a matching source A and, through the OV circuit, raises the logit of a specific completion C — a [A]…[B]→[C] rule that leaps over the intervening tokens. The instrument runs a small table of them. A fail-loud self-check throws unless each configured [A]…[B] pair boosts its correct C (e.g. keep…in→mind) — the exact one-layer computation.
A lookup demonstration of the skip-trigram FUNCTION; a real head stores these in continuous QK/OV weight matrices, so they are approximate and come with the famous side effect (the same head that does keep…in→mind also weakly does keep…at→mind — skip-trigrams can’t be sculpted independently). The table shown is exact.