Transformer Inference · Instrument 01 · Tokens & Embeddings

Turning words into vectors

Before a model can reason, it has to convert language into numbers it can work with. Text is chopped into tokens, each token becomes a learned vector, and those vectors are arranged so that meaning lives in their geometry. Position gets folded in too — because order changes everything.

Tokenization

Embedding space

Semantic geometry

Positional encoding

Byte-pair merges

merges applied0

Tokenizers are learned. Byte-pair encoding starts from characters and repeatedly fuses the most frequent adjacent pair into a new symbol. Common chunks become single tokens; the merges also tokenize unseen words from familiar pieces.

BPE training

Pick a token

Each token ID indexes a row of the embedding matrix — its vector. The model learns these so related meanings sit close together. Select a word to see its vector and its nearest neighbors in the space.

embedding space

Analogy

Relationships become directions. The vector from man to king is the same as from woman to queen — so adding it to woman lands on queen. The model never stored "queen is royal female"; it fell out of the geometry.

analogy arithmetic

Position

position5

Embeddings alone are a bag of vectors — "dog bites man" and "man bites dog" would look identical. Positional encodings give each slot a unique fingerprint from sinusoids of many frequencies, added to the token vector so the model knows where as well as what.

positional encoding