Key takeaway: Dynamic batching trades individual request latency for aggregate throughput, and the trade is invisible until someone investigates why identical requests report inconsistent response times.
Why Serving Batches Requests at All
GPU inference is far more efficient processing several requests together than one at a time, because the cost of loading model weights into compute is largely fixed regardless of batch size. Serving requests one by one wastes most of the available throughput.
Dynamic batching accumulates incoming requests for a short window, processes them together, and returns individual responses. This is what makes production LLM serving economically viable at scale — without it, per-request cost would be substantially higher for the same hardware.
The Latency Trade That Follows
A request arriving at the start of a batching window can wait the full window duration before processing begins, even though the model itself would have answered it instantly in isolation. A request arriving just as a batch closes gets processed almost immediately.
| Batch window setting | Aggregate throughput | Worst-case added latency |
|---|---|---|
| No batching | Lowest | None |
| Short window (10–20 ms) | Moderate improvement | Small, consistent |
| Long window (100+ ms) | Highest | Noticeable, inconsistent |
This is why two functionally identical requests can report meaningfully different response times purely based on arrival timing relative to the batch window — a fact that surprises teams debugging what looks like a random latency spike but is actually the batching mechanism working exactly as configured.
Setting the Window Deliberately
The right window length depends entirely on what the application actually needs. An interactive chat interface where users perceive delay directly should use a short window, accepting lower throughput efficiency in exchange for consistent responsiveness. A backend batch-processing job with no human waiting can use a much longer window and capture the full throughput benefit.
Applying one global batching configuration across workloads with different latency sensitivity is the common mistake — the setting that is correct for a background enrichment job actively harms a user-facing chat feature sharing the same serving infrastructure.
Continuous Batching for Variable-Length Output
Simple batching groups requests and waits for the whole batch to complete before returning any response, which is wasteful when output lengths vary — a short response finishes and then sits idle waiting for a longer one in the same batch.
Continuous batching, now standard in modern serving frameworks, allows individual requests to complete and leave the batch independently while new requests join dynamically. This substantially improves both throughput and latency consistency for workloads with variable output length, which describes almost all language model serving.
Diagnosing Latency Complaints
When users report inconsistent response times for what should be identical requests, check the batching configuration before assuming a model or infrastructure problem. Log the actual batch size and queue wait time per request, separate from total processing time — this separation is what reveals whether latency variance comes from batching behaviour or from something else entirely, and without it the two are easy to conflate.
The Bottom Line
Set batch window length based on the specific workload’s latency sensitivity rather than one global default, use continuous batching where output length varies, and log queue wait time separately from processing time so latency complaints can be diagnosed against the actual mechanism rather than guessed at.
