Batching and the throughput trade

Why does serving more users at once make each individual one wait longer?

There is a claim that sounds like it must be a mistake: the way to make a model server efficient is to make each user wait. Not as a side effect to be minimised — as the mechanism. Group requests together, and the machine does far more total work per second; leave them alone, and it idles.

That should bother you, because we usually think of contention as pure loss. If two people share a road, both go slower and nothing is gained. So what is different about a GPU that makes sharing produce something rather than merely dividing it?

The answer is in a detail of how the hardware spends its time, so let us look there.

When a model generates one token for one request, the arithmetic involved is small — a handful of multiplications against each weight. But to do that arithmetic, every weight in the model must be read out of memory and brought to the arithmetic units. For a large model that is tens of gigabytes moved to produce a single token. So which resource is the bottleneck: the multipliers, or the pipe from memory?

The pipe, overwhelmingly. And here is the consequence that makes the whole thing work: once the weights have been fetched, they can be used for many requests at almost no extra cost. Fetching the weights to serve one sequence and fetching them to serve thirty-two costs nearly the same in memory traffic. The arithmetic multiplies; the expensive part does not.

So the machine has been sitting there with its arithmetic units idle, waiting on memory, and batching fills that idle capacity with other people's work. That is why it is not like the shared road. The road was saturated; the GPU was not.

Now, what does the individual pay? Two separate costs, and it is worth keeping them apart.

The first is waiting to be grouped. If the server collects requests for fifty milliseconds before starting, everyone pays up to fifty milliseconds. That cost is pure delay, added before any work happens.

The second is sharing the step. Once the batch is large enough that the arithmetic units are actually busy, adding another request no longer rides along free — it competes. Each step now takes measurably longer, so every member of the batch produces tokens more slowly. Throughput is still climbing, but the curve is bending over, and each request's own experience is degrading.

There is a third limit that decides where you can stop. Every sequence in flight needs its attention keys and values kept in memory — the KV cache — and that memory grows with batch size and with each sequence's length. Long before the arithmetic units are the constraint, you can simply run out of room to hold everyone's context.

So what shape does the trade have? Throughput rising steeply, then flattening; per-request latency flat, then rising. Where would you choose to sit?

The naive version of this trade is harsher than it needs to be, and the fix changed how these systems are built.

If you form a fixed batch, run it to completion, and only then start the next, you have static batching — and it has a nasty property. The batch finishes when its longest member finishes. A request wanting twenty tokens sits in the machine while a neighbour generates two thousand, contributing nothing and occupying a slot. The short request's latency is set by the longest sequence it was unlucky enough to be grouped with.

Continuous batching — introduced by the Orca system in 2022 and made widely available by vLLM in 2023 — schedules at the level of individual decode steps instead. When a sequence finishes, its slot is immediately given to a waiting request; new arrivals join the very next step rather than the next batch. This removes the head-of-line blocking almost entirely, which is why the naive picture of "large batches mean long waits" is much less true of a modern server than of a 2021 one.

Two misconceptions to correct. First, batching does not make any single request faster — it never has and never can. It makes the fleet cheaper, and cheaper capacity is what lets you keep queues short at a given load. The user-visible benefit is indirect. Second, the trade is not between latency and throughput in the abstract; it is between latency and cost, because throughput divided by hardware is what appears on the invoice.

And Little's law sits quietly behind all of it: push the arrival rate up without adding capacity and the number of requests resident grows whatever your batching policy is. Batching does not repeal queueing theory; it moves the point at which the queue starts to grow.

• Prefill and decode have different bottlenecks, which is why some servers schedule them separately or on different machines.
• How KV-cache memory management — paged allocation in particular — sets the real ceiling on batch size.
• Little's law and queueing behaviour near saturation, where waiting time stops rising gently and starts rising sharply.

Every term the collection defines is gathered in the glossary.