Once the text is tokens, each token is replaced by a vector — a long list of numbers — and those numbers are coordinates. Every token becomes a point in a high-dimensional space of meaning, where words used in similar ways sit near each other. The text stops being a stream and becomes a place. Meaning, made geometry.
The Four Natures
each piece emerges by one of four natures
The Idea
the three-beat story
Each token is looked up in an embedding table and replaced by a vector — hundreds or thousands of numbers. Those numbers are not random: they are coordinates in a space the model learned during training. The token is now a point.
Because the coordinates were learned from how words are actually used, words with similar meaning end up near each other — colors cluster, animals cluster, numbers cluster. The model can now ask 'how related are these two tokens?' and get an answer as simple as distance.
Stranger still, directions in the space mean things. The step from 'man' to 'king' is roughly the step from 'woman' to 'queen' — so king minus man plus woman lands near queen. Meaning has not just a position but a structure you can do arithmetic on.
Tokens Become a Map of Meaning
each token is replaced by a vector — a list of numbers that are coordinates. Every token becomes a point in a space where similar meanings sit near each other. Pick a phrase; hover a settled word for its nearest neighbours; try the analogy. An illustration (a real space has hundreds of dimensions; coordinates here are hand-placed so the clusters read).
The Reckoning
the stage, and the honesty about it
Pipeline · Stage Two
the thread
- The second arrow: tokens → vectors. Everything the transformer does next is geometry on these points.
- >Receives tokens from the tokenizer; hands a sequence of vectors to the transformer stack.
Render, Not Invent
honest about dimensions
- The live map hand-places a small lexicon so the clusters are legible. A real model learns the coordinates from data, in hundreds to thousands of dimensions, projected down to two for a picture.
- The behavior — similar meanings near each other, analogies as directions — is the real, observed phenomenon (word2vec, and the embedding layer of every transformer).
Why It Matters
the seam
- This is where text becomes math. From here on the model never reasons about words, only about positions and directions in a learned space.
- Add position information so order matters, and the sequence of vectors is ready for the layers.
The Roster
the parts and concepts of this stage as ACI .agents — each a birth certificate & a nature (10)
