Knowing that an LLM is "a large model trained to predict the next word" is enough to use one. Knowing roughly how it predicts the next word is enough to stop being surprised by either its capabilities or its failures. This post is the second.
There is no maths in here. There are also no analogies that fall apart if you push them. Just a working picture of the path your prompt takes through the model, in five stages.
Stage 1: tokenisation
Your prompt arrives as a string. The first thing the model does is split it into tokens: small chunks of about 3 to 4 characters each.
Take "I love M-Pesa". A typical tokeniser splits this into roughly:
["I", " love", " M", "-", "Pesa"]
Five tokens. Common English words are usually one token each. Less common words split into pieces. Punctuation gets its own token. Whitespace clings to the next word.
Two consequences worth knowing:
- You pay per token. API pricing is "X dollars per million tokens". Long contexts cost more.
- Tokenisation is lossy. The model never sees the literal string "I love M-Pesa". It sees five integer IDs that point at entries in a vocabulary list. Word identity is preserved; spelling is not. This is why models sometimes miscount letters in a word: from the inside, "strawberry" is two or three tokens, not eleven characters.
Each token gets converted into an integer. The string is now a list of integers.
Stage 2: embedding
The integers are not yet anything the maths can work with. The model's first internal layer is a giant lookup table called the embedding matrix. It maps each token ID to a vector of about 4,000 to 12,000 floating-point numbers, depending on the model.
These numbers are not random. During training, the model arranged them so that semantically similar tokens have similar vectors. The vector for "cat" sits close to "kitten" and far from "regression". This is why embedding-based search works: the geometry of the space encodes meaning.
After this stage, each input token has become a vector of thousands of numbers. The list of integers has become a sequence of vectors.
Stage 3: attention, the part that matters
This is where LLMs do their real work. The architecture is called transformer, and the operation at its heart is called attention.
The intuition: at every position in the sequence, the model asks "which other positions should I pay attention to in order to predict what comes next?". For the prompt "The cat sat on the", when predicting the next word, the model probably wants to attend to "cat" (subject) and "on" (preposition pattern), more than to "the" (less informative).
Attention is computed as a weighted sum: for each token, look at every other token, decide how much it matters, sum them up. Repeat across many "attention heads" in parallel, each of which has learned to pay attention to a different kind of relationship (subject-verb, opening-closing brackets, anaphora, code structure, etc).
This happens at every layer of the model. A 70-layer model does this 70 times in sequence. By the end, each position's vector has absorbed information from every other position, weighted by relevance.
This is the answer to "how does the model handle long-range dependencies". It does not have memory in the human sense. It computes attention across the whole context, every time, from scratch.
Stage 4: prediction
After all those layers of attention and computation, the model arrives at a vector for the final position in the sequence. That vector is multiplied by another matrix that produces a score for every token in the model's vocabulary (which can be 100,000+ tokens long).
These scores are converted into probabilities (softmax, but you do not need the maths). The token with the highest probability is the model's best guess for "what comes next".
Crucially, the model does not just pick the highest-probability token. It samples from the distribution. The temperature setting controls how much the sampling respects the probabilities:
- Temperature 0. Always pick the single most likely token. Deterministic.
- Temperature 1. Sample proportionally to the probabilities. Some randomness, often mostly likely tokens.
- Temperature 2. Sampling is heavily flattened. Even unlikely tokens come up. Output gets weird.
For factual or code tasks, low temperature is right. For brainstorming, higher temperature gives variety.
Stage 5: do it again
The model picks one token. Then... it stops. To produce a longer response, the system feeds the prompt plus the new token back in as input, and the whole pipeline runs again to predict the next token. And again. And again.
This is why streaming is "tokens appearing one at a time". Each token requires a full pass through every layer of the model.
It is also why long outputs cost more (more tokens generated, more compute spent) and why each token of latency adds up. A 200-token response is 200 sequential model passes.
Why "predict the next token" is enough
Spend three paragraphs on this because it is the most counter-intuitive part.
A model trained on enough text learns enormous structure implicitly. To accurately predict the next word in a sentence about Kenyan tax law, it has to encode patterns about: English grammar, legal vocabulary, Kenyan jurisdiction specifics, document conventions, common typos, the way humans hedge claims, the structural difference between an example and a definition, and so on. None of these were taught explicitly. They are all there because the next-token-prediction objective rewards getting them right.
By the time a model has 70 billion parameters and has trained on most of the internet, the learned structure is rich enough that you can ask it to do tasks the training never explicitly showed it. Translate a poem. Refactor code. Reason about a legal hypothetical. None of these are next-token-prediction, exactly, but they all decompose into sequences of next-token decisions where the right answer is consistent with everything the model has internalised about how language and reasoning work.
This is the whole game. The model is not "thinking" the way humans do, but the kinds of patterns it has compressed are rich enough to look like thinking on a wide range of tasks. The phrase you will see for this is emergent capabilities: behaviours the model learned by accident as a side effect of getting better at predicting words. Nobody knows in advance which capabilities will emerge at which scale.
What this explains about LLM failure modes
Once you have this picture, the failures stop being mysterious.
Hallucinations. The model does not have a "I do not know" mode. It always picks the next token. If your prompt asks for something the model has no real information about, the most-likely next tokens are still going to feel like a confident answer, because that is what its training data looks like. The fix is structural, not stylistic: provide the source, or let the model use a tool to look things up.
Counting and exact maths. Tokenisation breaks numbers up unpredictably ("8729" might be two tokens, "8723" three). The model never sees the digits clearly. Plus, arithmetic is not really a "predict the next word" pattern. Fix: have the model write code or use a calculator tool.
Forgetting things mid-conversation. The context window is finite. When older messages drop out of the input, attention literally cannot reach them. Fix: longer-context models, or summarise earlier turns.
Confidently wrong dates. The training cutoff is hard. Anything after it is the model guessing what should plausibly come next based on what it knew when it stopped learning. Fix: give it search results, or note the cutoff in the system prompt.
Repetition and looping. At low temperatures with certain inputs, the highest-probability next token is sometimes "repeat the most recent thing". Fix: nudge temperature up, or use repetition penalty if available.
What this does not explain
Two important things this picture leaves out, which you can ignore for now but should be aware of:
- Reinforcement learning fine-tuning (RLHF and friends). After pre-training, modern models go through additional training that rewards "helpful, harmless, honest" outputs. This is why ChatGPT feels different from a raw next-token-predictor. Without RLHF, you would get a model that is happy to predict the next word in a Reddit shitpost.
- Multi-modal models. Newer models accept images, audio, video. The same attention machinery, but the inputs are tokens of pixels or audio frames instead of text. Same five stages.
What to do with this
You should now be able to:
- Explain why an LLM might confidently make up a citation (hallucination = highest-probability tokens that are not constrained by any source).
- Explain why long contexts cost more (tokens, multiplied by layers, multiplied by output length).
- Explain the difference between training and inference, and why one costs millions and the other costs cents.
- Read an LLM paper title without flinching.
You do not need to know more about the internals than this to use AI well. The next post in the series goes back outward to the practical side: which tools we use day to day, and which ones we have stopped reaching for.