A language model does one thing: predict the next token. But the path from your words to that prediction runs through a remarkable machine — text becomes numbers, numbers become vectors, vectors trade information through attention, refine across dozens of layers, and finally collapse into a single guess. This is that forward pass, end to end, plus the backward pass that taught the machine in the first place.
A model can't read letters — it reads tokens, chunks of text drawn from a fixed vocabulary. Common words are single tokens; rarer ones split into subwords. Each token is just an integer ID, an index into the vocabulary.
Each token ID is looked up in a table, becoming a vector — a list of numbers, a point in a high-dimensional space. The model learns to place related meanings near each other, so much so that directions carry meaning: the step from man to woman is the same step as king to queen.
A word's meaning depends on its neighbors — "bank" by a river isn't "bank" with money. Attention lets every token look at every other and pull in what's relevant, weighting each by how much it matters. Those weights always sum to one: a token spends its full attention, distributed.
The vectors flow upward through a stack of identical layers, each one running attention and a small neural network, each writing its result back into a shared residual stream. Information accumulates: early layers catch grammar, later ones catch sense. Modern models stack dozens to hundreds.
At the top, the final vector is projected onto the whole vocabulary, scoring every possible next token — the logits. A softmax turns those scores into probabilities that sum to one, and the model samples one. That token is appended, and the whole process runs again for the next.
None of this works until the weights are right — and they start random. During training the model predicts, measures how wrong it is with a loss, then sends that error backward through every layer, nudging each weight in the direction that reduces it. Repeat across trillions of tokens and the machine above emerges.