Serving a language model in production raises an immediate question: how many concurrent requests can you handle on the same hardware? The answer depends heavily on two infrastructure techniques, both rooted in a simple principle — never leave resources idle.
The problem with simple batching
The simplest serving method is static batching: waiting for a set of requests to gather, processing them synchronously, and only moving to the next batch when every request in the current one completes. The flaw here is that generation lengths vary. If one request finishes early while another continues, that compute slot sits idle until the longest sequence concludes. Expensive GPU cycles are spent waiting.
Continuous batching
Continuous batching removes this waste. Rather than waiting for the entire batch to finish, the system ejects a request the moment it completes, immediately slotting a new request into its place. The batch remains full, and no compute slot sits idle. The impact on throughput is significant — often several times higher than static batching, particularly when generation lengths vary widely.
The KV-cache problem
Compute is only one bottleneck; the second is the KV cache. Every request requires memory to store its context, and this requirement scales with sequence length. The naive approach is to allocate a single, contiguous block of memory per request, sized for the maximum possible length. Because most requests finish well before reaching that maximum, a large portion of the reserved allocation sits empty. In practice, this method wastes roughly half the available memory.
PagedAttention
The solution borrows a core concept from operating systems: PagedAttention. Just as an OS partitions memory into small, non-contiguous pages, PagedAttention divides the KV cache into fixed-size blocks (typically sixteen tokens per block). A request claims only the blocks it actively needs, and those blocks do not need to be contiguous in memory. Memory fragmentation effectively disappears, and utilisation rises above ninety percent.
This architecture yields a secondary benefit. If multiple requests share a common prefix — such as an identical system prompt — they can share the same physical blocks rather than maintaining duplicate copies. For workloads with heavy, static prompts, this reduces memory overhead significantly.
Why these two matter together
Continuous batching keeps compute slots saturated, while PagedAttention frees memory to fit more concurrent requests. The two mechanisms reinforce each other: reclaimed memory allows for a larger batch size, and a larger batch drives higher overall throughput. This compound effect is why both techniques form the foundation of modern language-model serving infrastructure.
Putting it together
Crucially, these optimisations do not alter the underlying model; they simply manage physical resources with greater discipline. The identical model, running on the same hardware, can serve significantly more users strictly because it ceases to waste compute and memory. In production, it is often this resource management — rather than the model itself — that dictates the fundamental unit economics of serving.