Feed a machine a story and ask only ‘what word tends to follow these?’ and it will babble new sentences that sound like the source without meaning a thing. It is the oldest text generator, and the shadow the large models grew from. Raise the memory (or tap) from gibberish to eerie fluency.
An order-n Markov chain over a source text. Build a table mapping each n-word prefix to the words that actually follow it in the source, then generate by sampling a next word from that table, sliding the window forward. At order 1 (only the previous word matters) the output is grammatical-ish gibberish; raise the order and it reproduces longer real phrases, approaching the source itself. Crucially, EVERY generated transition existed in the source — the chain never invents a word-pair it did not see (it can only recombine). A fail-loud self-check throws unless every generated bigram is a real source bigram.
A tiny source text (one Aesop fable) and word-level n-grams (real language models are sub-word neural nets that DO generalise beyond seen n-grams); higher order here just copies longer runs. The transition table and the sampling are computed exactly.