What is continuous batching in LLM serving?

Also called in-flight batching, iteration-level scheduling, iteration-level batching

Definition

Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.

Continuous batching is a scheduling method for LLM serving in which the server decides which sequences to run at every decode iteration instead of once per batch of requests. Output lengths vary widely: one response might end after 20 tokens and another after 2,000. With continuous batching, a finished sequence leaves the batch after its last step and a queued request takes its slot at the next one, so the GPU keeps working on a full batch. vLLM, TensorRT-LLM, and llama.cpp's server all implement it. Because throughput depends on how full each batch is, a single request on an idle inference engine says little about what the same engine delivers under load.

How continuous batching works#

Continuous batching runs the model one iteration at a time over whichever sequences are active. After each iteration, the scheduler removes sequences that produced an end token or hit their length limit, then admits queued requests into the freed slots, as long as there is KV cache memory for them and room in the per-step token budget.

Four decode iterations of one batch: request A finishes after iteration 2, queued request D takes its slot in iteration 3, and E replaces C in iteration 4. Decode iteration and the requests in its batchIteration 1A, B, and C decodeABCIteration 2A emits its end tokenABCIteration 3Queued D takes A's slotDBCIteration 4C finished; queued E joinsDBE
Figure 1: Continuous batching refills a finished request's slot at the next iteration. A static batch would hold D and E until its longest request finished.

The idea was introduced as iteration-level scheduling in Orca (Yu et al., OSDI 2022). Orca paired it with selective batching, which applies batching to only a selected set of operations so that requests of different lengths can share an iteration. The authors report 36.9 times higher throughput than NVIDIA FasterTransformer at the same latency on a GPT-3 175B model.

Two details decide how well continuous batching works in practice. First, every admitted request needs KV cache space, and PagedAttention allocates that space in small blocks on demand, so more sequences fit. Second, a new request must run its prefill before it can decode. TensorRT-LLM's documentation, which calls the feature in-flight batching, states that sequences in the context (prefill) phase can be processed together with sequences in the generation phase. vLLM batches pending decodes first and fills the remaining per-step budget with chunks of new prompts.

Continuous batching vs static batching#

Static batching groups a fixed set of requests and runs them together until the longest one finishes. Continuous batching changes the batch membership at every iteration. Under static batching, slots sit idle after short requests finish and new arrivals wait for the whole batch, which costs both throughput and queueing time.

Static batching Continuous batching
Scheduling unit A whole request One decode iteration
A request finishes early Its slot stays idle until the batch ends Its slot is refilled at the next iteration
A new request arrives Waits for the current batch to complete Joins at the next iteration if memory allows

Larger batches raise total throughput but make each decode step slower, so continuous batching trades inter-token latency for throughput as concurrency grows. vLLM's optimization guide exposes that trade-off through max_num_batched_tokens: smaller values improve inter-token latency, and larger values improve time to first token and throughput. Pick the setting against a latency target measured at realistic concurrency.

The LLM inference optimization guide explains how continuous batching combines with chunked prefill, prefix caching, and paged KV memory.

Cite this definition

The AI Engineering Editors. "What is continuous batching in LLM serving?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/continuous-batching

Title
What is continuous batching in LLM serving?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM inference and serving hub · Updated