A language model never reads letters. The first thing that happens to your text is that it is cut into tokens — sub-word pieces drawn from a fixed vocabulary. Common words are a single token; rare words shatter into several; even the space before a word is part of it. Before meaning, before prediction, there is the cut.
The Four Natures
each piece emerges by one of four natures
The Idea
the three-beat story
Before a model can do anything with your text, it slices it into tokens — not words, but sub-word fragments from a vocabulary fixed in advance. 'token' is one piece; 'tokenizer' might be 'token' + 'izer'; a rare word can break into half a dozen. The model's entire world is built from these pieces.
Why not whole words, or single letters? Whole words make the vocabulary impossibly large and leave no way to spell something new; single letters make every sequence painfully long. Sub-word tokenization (byte-pair encoding and its kin) is the trade: a fixed vocabulary that can still spell anything by combining pieces — common words staying whole, rare ones breaking down.
The cut is literal and unforgiving. The space before a word is usually part of the token; a capital letter makes a different token; a number may be one token or several. This is why models miscount letters and stumble on rhymes — they never see the letters at all, only the pieces the tokenizer handed up.
Text Becomes Tokens
a model never sees letters — first the text is cut into tokens, sub-word pieces from a fixed vocabulary. Common words are one token; rare words shatter into several; the space before a word (shown ·) is part of it. Click a phrase. An illustration of sub-word tokenization (real BPE is learned by merge frequency).
The Reckoning
the stage, and the honesty about it
Pipeline · Stage One
the thread
- The first arrow of the inference layer: stream → tokenize → embed → transformer layers → next-token distribution. Everything downstream sees only tokens, never your raw text.
- >Feeds the embedding (stage two); a sibling of the next prediction (the final stage).
Render, Not Invent
honest about BPE
- The live splitter above is an illustration of sub-word tokenization; a real tokenizer's vocabulary is learned by merge frequency (byte-pair encoding) over a huge corpus, not hand-listed.
- The behavior shown — common words whole, rare words fragmented, spaces and punctuation as their own tokens — is the genuine thing.
Why It Matters
the consequences
- Token boundaries set the bill (you pay per token), the memory (context limits are measured in tokens), and the quirks (letter-counting, spelling, penalties on rare scripts).
- The first cut decides what the model can even perceive.
The Roster
the parts and concepts of this stage as ACI .agents — each a birth certificate & a nature (10)
