Deep dive E · Level 2, optional · 10 min read

How a language model writes: the full pipeline

In one sentence: To write an answer, the model runs the same loop again and again: score every possible next token, turn the scores into chances, pick one, add it to the text, and go round again, and several clever tricks keep that loop fast.

Level 2, optional. You need Chapters 5, 6 and 7 first. This deep dive follows one answer through every stage.

The whole path

Here is one question, "How tall is Everest?", going through all eight stages.

Eight stages in a column from your text to back to words, with CPU or GPU tags and a loop from the pick back to the transformer layers.

1. Your text

The question is just a string of characters. Nothing clever has happened yet.

2. The tokenizer (on the CPU)

The text is cut into pieces and each piece is looked up in a fixed list to get an ID number. For example (the split and the numbers are made up for illustration): "How" "tall" "is" "Ever" "est" "?" might become the IDs 4438, 9061, 318, 18902, 395 and 30. Notice that "Everest" became two pieces. This is why token counts never match word counts. This step is string handling, not arithmetic, so it runs on the ordinary processor.

3. Embeddings

Each ID is replaced by its list of numbers (Chapter 5). In a typical model this list has several thousand numbers. So six tokens become a table with six rows and several thousand columns.

4. The transformer layers

The table passes through the stack of blocks from Deep dive D. In each block, every token looks at the tokens before it (attention), then is processed by a small network. While doing this, each layer saves the keys and values of every token. That saved store is the KV cache, and we return to it below.

5. Scores

At the top, a final layer turns the last token's list into one score for every token the model knows. The vocabulary is large, tens of thousands up to over a hundred thousand tokens. These raw scores are called logits.

6. Chances

Softmax (Deep dive D) turns the scores into chances that add up to 100%. This is the only place where the model "decides" anything: the shape of this list of chances.

7. Pick one

A rule called sampling chooses one token from the chances. We look at the rules in a moment.

8. Back to words

The chosen token is turned back into text and sent to your screen. It is also added to the text so far, and the loop goes back to stage 4 to choose the next token. It stops when the model picks a special "end" token, or reaches a length limit.

Two phases: reading and writing

The first time through, the model must process all the tokens in your question. After that, it only has to process one new token at a time.

A timeline. First, one wide block: prefill, reading the whole prompt at once. Then a first token appears. Then eight small blocks, one token each, in a row: decode, one at a time.

Prefill is the reading phase. All the prompt tokens are processed together, in parallel. This keeps the chip's arithmetic units busy, so the limit is calculation power (we say the phase is compute-bound). Prefill decides time to first token: how long you wait for the first word.
Decode is the writing phase. Each new token needs the one before it, so they come out one at a time. Each step has to read the model's numbers and the saved keys and values from memory, but does little arithmetic. So the limit is how fast memory can be read (we say the phase is memory-bound). Decode decides how quickly the rest of the answer appears.

These two phases have different bottlenecks, which is why serving systems treat them separately, and why a long prompt and a long answer cost in different ways.

Choosing the token: sampling with real numbers

Suppose the model is finishing "The sky is" and has five candidates with these scores: *blue* 3.0, *clear* 1.9, *cloudy* 1.7, *grey* 1.1, *falling* −1.0.

Softmax turns the scores into chances. At the normal setting (temperature 1) they are about: blue 56%, clear 19%, cloudy 15%, grey 8%, falling 1%.

Temperature divides the scores before softmax. This reshapes the chances.

Temperature 0.5 (lower): the favourite pulls ahead: blue 83%, clear 9%, cloudy 6%, grey 2%, falling 0%. The output is steady and predictable.
Temperature 2 (higher): the chances flatten: blue 38%, clear 22%, cloudy 20%, grey 15%, falling 5%. The output is more varied, and more likely to wander.

Five bar charts of the same five words. Raw chances; low temperature, where blue dominates; high temperature, where chances are more even; top-k equals 2, which keeps only blue and clear; and top-p equals 0.9, which keeps blue, clear and cloudy.

Two more rules cut the list down before choosing:

Top-k = 2 keeps only the two best, blue and clear. Their chances (56% and 19%) are rescaled to add up to 100%: 75% and 25%.
Top-p = 0.9 keeps the smallest group of top words whose chances add up to at least 90%. Here blue (56%) + clear (19%) + cloudy (15%) = 90%, so those three stay, rescaled to 62%, 21% and 17%. When the model is very sure, this group is small. When it is unsure, the group is large.

Finally, the token is drawn at random according to the remaining chances. "Always take the best" is called greedy decoding, and it is the same as a very low temperature. This random draw is why the same question can give different answers.

The KV cache

Look again at stage 4. To write token number 100, the model needs the keys and values of all 99 tokens before it. Rebuilding them from scratch at every step would waste enormous effort.

Two staircases of boxes over six steps. Without a cache, every box is rebuilt at every step. With a cache, earlier boxes are kept and only one new box is built at each step.

So the model stores them as it goes. That store is the KV cache. Each new token only has to build its own keys and values and add them to the store.

How much work does it save? Without a cache, step 1 builds 1 set, step 2 builds 2, and so on. For an answer of 1,000 tokens, that adds up to 1 + 2 + … + 1,000 = 500,500 sets. With a cache, it is 1,000. (Simplified, but the shape is right: the work grows with the square of the length without a cache, and in a straight line with it.)

The price is memory. For each token, the cache stores two lists (a key and a value) for every layer. The size per token is:

2 × layers × key-value heads × size of each head × bytes per number

For an illustrative mid-sized model with 32 layers, 8 key-value heads, a head size of 128 and 2 bytes per number, that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, about 128 KB per token. A conversation of 8,000 tokens then needs about 1 GB of cache, for a single user. With many users at once, the cache, not the model itself, can fill the chip's memory. (Check a real model's published settings for exact numbers.)

Making it faster

Because the writing phase is limited by memory, many speed-ups aim at moving less data or doing more useful work per trip. Three you will hear about:

Continuous batching: serve many users' requests together, adding new ones as others finish, so the chip is never idle.
Paged cache: store the cache in small blocks, as an operating system does with memory, so little space is wasted and blocks can be shared between requests.
Fused attention: reorganise the attention calculation so less data travels back and forth between slow and fast memory. The answer is the same, only quicker.

A fourth is worth a closer look, because it is clever.

Speculative decoding

Let a small, fast model guess several tokens ahead, and let the big model check the whole batch in one pass.

A small draft model guesses four tokens: Paris, is, a, small. The big model checks all four at once. It agrees with the first three and disagrees with the fourth, where it prefers large.

After "The capital of France is", the draft guesses "Paris", "is", "a", "small". The big model checks all four at once. It agrees with the first three, and would have written "large" instead of "small". So three tokens are accepted, and the big model supplies the fourth. That is four tokens for about the cost of one big-model step.

Importantly, the result is the same as if the big model had written it alone. The draft only saves time. If the draft is often wrong, the gain shrinks.

Another family of speed-ups makes the model's numbers smaller (for example, using fewer bytes per number), so less has to be read from memory. That is called quantisation.

Why this matters

Cost. You are usually charged by tokens, in and out. Long prompts and long answers cost more.
Delay. Time to first token depends on the length of the prompt. The speed of the rest depends on memory.
Capacity. How many people one chip can serve depends mostly on the KV cache.
Quality control. Temperature, top-k and top-p change how often the model says something unexpected.

The third book in this library, *Production AI Architecture*, is planned to take these ideas into sizing real hardware.

Back to Asha

Asha notices her helper's first word takes a second to appear when she pastes in her whole price list, but the rest flows quickly. That is prefill, reading a long prompt, and then decode. She shortens the pasted text, and the first word comes sooner. For her cheerful captions she raises the temperature, and for her price answers she sets it low.

Words explained

Logits: the raw scores for every possible next token.
Softmax: turns scores into chances that add up to 100%.
Sampling: choosing one token according to the chances.
Temperature, top-k, top-p: ways to reshape or cut down the chances before choosing.
Greedy decoding: always taking the single most likely token.
Prefill, decode: reading the prompt all at once; writing one token at a time.
Time to first token: the wait before the first word appears.
KV cache: the store of keys and values for earlier tokens, so they are not rebuilt.
Compute-bound, memory-bound: limited by arithmetic speed; limited by how fast data can be read.
Speculative decoding: a small model drafts, the big model checks.
Quantisation: storing the model's numbers with fewer bytes.

The limits of this chapter

Token splits, IDs and the model settings in the examples are invented for illustration. Check a real model's published details.
The pipeline is the common design for modern chat models. Some models differ, for example in how they handle images or very long inputs.
Serving systems combine many more tricks than the ones shown.

For engineers

Sampling pipeline: logits / T, then optional top-k and top-p (nucleus) truncation on the sorted probabilities, then renormalise and sample. Repetition and frequency penalties adjust logits before softmax. Greedy decoding is argmax; beam search is used for some tasks.
Prefill cost scales with prompt length and is dominated by matrix multiplications (high arithmetic intensity). Decode is dominated by reading weights and the KV cache each step (low arithmetic intensity), so throughput is largely set by memory bandwidth and batch size. Larger batches reuse each read of the weights for more users.
KV cache bytes per token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element. Grouped-query and multi-query attention reduce n_kv_heads and so shrink the cache. Cache memory grows linearly with sequence length and with the number of concurrent sequences.
Speculative decoding preserves the target model's output distribution through an accept-or-reject rule on each drafted token. Speed-up depends on the draft's acceptance rate and relative cost.
Related techniques: continuous batching, paged attention, prefix caching of shared prompt starts, fused attention kernels, quantisation of weights and cache, and disaggregating prefill and decode onto separate pools. These are covered in depth in the planned *Production AI Architecture*.
The sizes quoted here are only to show the arithmetic. Work out real numbers from the model's configuration file.

Check yourself

Why is the first word slower to arrive after a very long prompt? Answer: The model must read the whole prompt (prefill) before it can produce the first token, and that takes longer the longer the prompt is.
What does the KV cache save, and what does it cost? Answer: It saves rebuilding the keys and values of earlier tokens at every step. It costs memory, which grows with the length of the text and the number of users.
Why does a lower temperature give steadier answers? Answer: It sharpens the chances so the most likely token wins far more often, leaving little room for surprising choices.

Plain-words explanations in the author's own words. The examples are invented, and no real company, product or customer is described. Simplified on purpose: where a detail matters, the "For engineers" part says so. Disclaimers