Table of Contents
- The Utilisation Illusion
- Two Phases With Opposite Bottlenecks
- Why Batching Is Nearly Free
- Continuous Batching Changes Everything
- The KV Cache Dominates Memory
- Quantisation Trade-Offs
- Choosing What to Optimise
- Measuring the Right Things
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: During token generation, the GPU spends most of its time waiting for weights to arrive from memory rather than computing. This is why batching multiplies throughput at almost no latency cost, and why memory bandwidth is the number that matters when choosing hardware.
The Utilisation Illusion
You are serving a language model. GPU utilisation reads 90 percent. Throughput is disappointing. Adding another GPU helps less than expected.
The utilisation metric is misleading you. It reports the fraction of time at least one kernel was executing — not whether the compute units were doing useful work. A GPU stalled waiting for data from memory registers as busy.
For token generation, that stall is most of the time. Modern accelerators have enormous arithmetic capability relative to their memory bandwidth, and generating one token requires reading the entire model’s weights from memory to perform a comparatively small amount of arithmetic.
The consequence is a specific and useful insight. Generation is memory-bandwidth-bound, not compute-bound. Every optimisation follows from that: anything that increases work done per byte read from memory increases throughput, and anything that only adds compute does not.
This is also why the intuition from training does not transfer. Training is compute-bound. Inference is not. Optimisations that help one frequently do nothing for the other.
Two Phases With Opposite Bottlenecks
Inference has two distinct phases with fundamentally different characteristics, and conflating them causes most confusion about performance.
Prefill processes the input prompt. All input tokens are available simultaneously, so their computation parallelises fully. This phase is genuinely compute-bound and uses the GPU’s arithmetic capability efficiently. It determines time to first token.
Decode generates output tokens one at a time. Each token depends on the previous one, so parallelisation across the sequence is impossible. The full weight matrix must be read from memory to produce a single token. This phase is memory-bandwidth-bound and determines tokens per second after the first.
| Property | Prefill | Decode |
|---|---|---|
| Parallelism | Across all input tokens | None within a request |
| Bottleneck | Compute | Memory bandwidth |
| Scales with | Input length | Output length |
| Affects | Time to first token | Tokens per second |
| Batching benefit | Moderate | Very large |
The practical implications diverge sharply. A long prompt costs prefill time roughly proportional to its length. A long output costs decode time per token regardless of prompt length. Optimising the wrong phase produces no improvement, which is why teams sometimes report that a change made no difference — it addressed a bottleneck they did not have.
Why Batching Is Nearly Free
Here is the counterintuitive consequence of memory-bound decode.
Generating one token for one request requires reading the full weight matrix. Generating one token for eight requests simultaneously requires reading the same weight matrix once and performing eight times the arithmetic on it.
The memory read — the expensive part — is amortised across the batch. The additional arithmetic is nearly free, because arithmetic was never the constraint.
The result is that throughput scales close to linearly with batch size while per-request latency barely increases, up to the point where memory capacity runs out or the arithmetic finally becomes the constraint.
batch=1 → 40 tokens/sec total (40 per request)
batch=8 → 290 tokens/sec total (36 per request)
batch=32 → 980 tokens/sec total (31 per request)
batch=64 → 1400 tokens/sec total (22 per request)Illustrative figures, and the shape is what matters: total throughput rises steeply while per-request throughput declines gently. A 32× larger batch delivers roughly 24× the total throughput at 78 percent of the single-request speed.
This makes batching the highest-return optimisation available for serving, by a wide margin. It is also why per-token pricing from providers is viable — they are batching your request with many others, and the marginal cost of adding your tokens to an existing batch is small.
Continuous Batching Changes Everything
Naive batching wastes most of the available benefit, and understanding why explains the value of modern serving frameworks.
With static batching, you collect requests, run them together, and wait for all to finish before starting the next batch. The problem is that requests have wildly different output lengths. A batch containing one request generating 800 tokens and seven generating 40 tokens runs at the pace of the longest — seven slots sit idle for most of the batch’s duration.
Continuous batching, sometimes called in-flight batching, removes that waste. As each sequence finishes, a queued request immediately takes its slot. The batch is continuously refilled rather than processed in lockstep.
The improvement is substantial — commonly two to four times the throughput of static batching on realistic workloads with variable output lengths, and the gap widens as length variance increases.
This is not something you implement yourself. It is a property of the serving framework, and it is the primary reason to use a purpose-built inference server rather than writing a loop around a model’s generate function. If your serving stack does static batching, changing that is likely your largest available improvement.
The KV Cache Dominates Memory
Attention requires the keys and values from all previous tokens. Recomputing them for every new token would be prohibitively expensive, so they are cached — and that cache becomes the binding memory constraint.
Cache size grows with the number of sequences, their lengths, the layer count, and the hidden dimension. Its important property is that it grows per request and per token generated, which means memory consumption rises as sequences lengthen.
This produces the central constraint of language model serving: the KV cache limits batch size, and batch size determines throughput. Anything that shrinks the cache permits larger batches, which increases throughput.
Techniques that help:
Paged attention. Allocate cache in fixed-size blocks rather than contiguous per-sequence reservations. This eliminates the internal fragmentation that otherwise wastes a large fraction of cache memory when actual lengths fall short of the reserved maximum. The gain is substantial and it is a framework feature rather than a model change.
Grouped-query attention. Share key and value projections across several attention heads, reducing cache size by the sharing factor. This is a model architecture property — you get it by choosing a model that has it.
Cache quantisation. Store keys and values at lower precision. Roughly halves cache size for a small quality cost.
Prefix sharing. Requests sharing a common prompt prefix — a system prompt, few-shot examples — can share the corresponding cache entries. For applications where every request carries the same long preamble, this is a large saving.
Bounding maximum sequence length. Unbounded generation means unbounded cache growth. A firm cap protects capacity for everyone.
Quantisation Trade-Offs
Reducing numerical precision shrinks both weights and cache, which directly increases the batch size you can fit — and since decode is memory-bound, smaller weights also mean less to read per token.
| Precision | Memory vs FP16 | Typical quality impact | Notes |
|---|---|---|---|
| FP16 / BF16 | baseline | none | Standard serving precision |
| FP8 | ~50% | negligible | Needs recent hardware support |
| INT8 | ~50% | small | Broadly supported |
| INT4 | ~25% | noticeable, task-dependent | Large throughput gain |
The pattern worth internalising is that quantisation helps inference more than it might appear, because it addresses the actual bottleneck twice — less memory read per token, and more batch capacity within the same memory.
Two cautions. Quality degradation from quantisation is task-dependent and not uniformly distributed: it tends to affect long-context reasoning and rare-token generation more than short straightforward responses. And it must be evaluated on your task rather than trusted from published benchmarks, because aggregate benchmark scores can hold while specific capabilities degrade.
The practical recommendation is INT8 or FP8 as a default for serving, with INT4 reserved for cases where you have measured the quality impact on your own evaluation set and found it acceptable.
Choosing What to Optimise
Different symptoms call for different interventions, and matching them correctly avoids wasted effort:
Time to first token is too high. This is prefill. Reduce prompt length, enable prompt caching for stable prefixes, or improve prefill parallelism. Batching does not help much here.
Tokens per second is too low for a single user. This is decode with batch size one, which is the worst case for memory-bound work. Options are a smaller model, quantisation, or speculative decoding. Batching cannot help a single request.
Total throughput is too low. This is where the largest wins live. Increase batch size, adopt continuous batching, shrink the KV cache, quantise.
Requests are being rejected or queued excessively. Memory capacity is the limit. Shrink the cache through paged attention or quantisation, or add capacity.
Cost per token is too high. Improve utilisation before adding hardware. An underbatched GPU is the most common cause of high per-token cost.
Speculative decoding deserves a note as the one technique that improves single-request latency. A small draft model proposes several tokens which the large model verifies in one forward pass. Because verification of several tokens costs little more than generating one — memory-bound again — accepted drafts come nearly free. It helps latency and does not help aggregate throughput, since it consumes extra compute that batching would otherwise use.
Measuring the Right Things
Metrics that reflect what users experience and what you pay for:
Time to first token, at p50 and p99. Perceived responsiveness. Driven by prefill and queueing.
Inter-token latency. Smoothness of streaming output. Driven by decode and batch size.
Total throughput in tokens per second. What determines your cost per token.
Batch size achieved, over time. The single most diagnostic number for throughput problems. If your average batch is 3 on hardware that could run 40, that is your entire problem.
KV cache utilisation. Approaching capacity means batch size is memory-limited.
Queue depth and wait time. Distinguishes a capacity problem from an efficiency problem.
Note that GPU utilisation percentage is absent from this list. It reports whether kernels were running, not whether they accomplished anything, and it reads high on a memory-stalled GPU. Achieved batch size and tokens per second tell you what utilisation cannot.
Common Pitfalls
Trusting GPU utilisation percentage. It reads high while the GPU waits on memory. Measure throughput instead.
Static batching. Wastes slots on length variance. Continuous batching is frequently a multiple-fold improvement.
Unbounded maximum sequence length. Reserves cache for a worst case that rarely occurs, shrinking batch size for everyone.
Optimising prefill when the problem is decode. Match the intervention to the phase.
Quantising without task-specific evaluation. Aggregate benchmarks can hold while your specific capability degrades.
Adding GPUs before improving batching. An underutilised GPU multiplied is an underutilised fleet.
Ignoring prefix sharing. Applications with a long shared system prompt are leaving a large saving unclaimed.
Conclusion
Language model inference is memory-bandwidth-bound during generation, and nearly every serving optimisation is a consequence of that single fact.
Batching multiplies throughput at minimal latency cost because it amortises the expensive memory read across requests. Continuous batching captures the benefit that static batching wastes on length variance. Shrinking the KV cache — through paged attention, grouped-query attention, quantisation, and prefix sharing — permits larger batches, which is the same lever again. And quantisation helps twice, reducing both bytes read per token and memory consumed per request.
Diagnose by phase before optimising: time to first token is prefill, tokens per second is decode, and total throughput is batching. Then measure achieved batch size rather than GPU utilisation, because utilisation will tell you the GPU is busy while it waits.
Frequently Asked Questions
Why does adding GPUs not proportionally increase throughput? Usually because each GPU is underbatched. Throughput comes from batch size, and splitting the same request rate across more GPUs reduces the batch each one achieves. Improve batching before adding hardware.
What batch size should be targeted? As large as memory permits while meeting your latency requirement. Continuous batching handles this dynamically; the useful number to monitor is the average batch actually achieved.
Does quantisation reduce quality noticeably? INT8 and FP8, rarely. INT4, sometimes and unevenly — long-context reasoning suffers more than short responses. Evaluate on your own task rather than trusting aggregate benchmarks.
Should I use a purpose-built inference server? For anything beyond low-volume experimentation, yes. Continuous batching and paged attention are difficult to implement well and account for most of the achievable throughput.
How much memory does a model actually need to serve? Weights plus KV cache plus activation overhead. The cache scales with concurrent sequences and their lengths, and frequently exceeds the weight memory at high concurrency — which surprises people sizing hardware from parameter count alone.
Is speculative decoding worth implementing? For latency-sensitive single-user workloads, yes. For maximising aggregate throughput, no — it consumes compute that batching would use more efficiently.
How does prompt caching relate to the KV cache? Prompt caching persists the KV cache entries for a stable prefix across requests, so the prefill for that prefix is not repeated. It reduces both cost and time to first token, and requires the prefix to be byte-identical.
