Text generation in a language model doesn’t work the way it looks from the outside. Behind every request, the work splits into two phases with fundamentally different characteristics: prefill and decode. Understanding the difference between them explains why optimising inference is so difficult.
Why generation is sequential
Language models generate text token by token, with each new token depending on all the tokens before it. You cannot produce the next token before the current one, because each prediction changes the subsequent context. For this reason, generation is inherently sequential and cannot be parallelised.
Prefill: compute-bound
During the prefill phase, the model processes the entire input prompt at once, in parallel. All positions are computed simultaneously, producing large matrix multiplications that saturate the GPU’s compute cores. This phase is compute-bound: the workload is heavy computation, and core utilisation is very high — close to 95 percent in the source’s example.
Decode: memory-bandwidth-bound
During the decode phase, the model produces just one token at a time. The input shrinks to a small vector, and the matrix multiplications become tiny. The primary cost shifts elsewhere: at every step, the entire cache must be read from memory. This phase is memory-bandwidth-bound. Gigabytes of data move per step, but very little computation occurs on that data. In the source’s example, core utilisation in this phase drops to around 10 percent, while memory bandwidth nears saturation.
Why the cache exists
If there were no cache, every step would have to recompute all past tokens from scratch — a hugely wasteful process. When a new token is added, the data required from past tokens for the attention computation has not changed, because the past does not depend on the future. It is therefore enough to compute those values once and store them, preventing them from being rebuilt at every subsequent step. This is the key-value cache (KV cache).
This cache carries a cost, growing linearly with the length of the text. In the source’s example, for a model with 32 layers and a hidden dimension of 4,096 at half precision, each token requires about 512 KB of cache. That translates to roughly 2 GB for a 4,096-token sequence — and that is for a single request. This is why the cache, rather than the raw size of the model, usually dictates how many concurrent users a system can serve.
What this means in practice
In a typical request, the decode phase consumes most of the execution time, because each token requires a separate step. Since this phase is bound by memory rather than computation, the most effective way to raise throughput is to batch several requests together. By spreading the cost of reading the weights and the memory across multiple requests, the system’s total throughput increases considerably.