---
title: "What is continuous batching in LLM serving?"
description: "Continuous batching rebuilds the LLM batch at every decode step, so finished requests leave and queued ones join at once. How it works and vs static batching."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/continuous-batching"
markdown_url: "https://theaiengineering.com/glossary/continuous-batching.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Continuous batching","in-flight batching","iteration-level scheduling","iteration-level batching"]
citation: "The AI Engineering Editors. \"What is continuous batching in LLM serving?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/continuous-batching"
---

# What is continuous batching in LLM serving?

Also called in-flight batching, iteration-level scheduling, iteration-level batching.

> **Definition of Continuous batching:** Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.

Continuous batching is a scheduling method for LLM serving in which the server decides which sequences to run at every decode iteration instead of once per batch of requests. Output lengths vary widely: one response might end after 20 tokens and another after 2,000. With continuous batching, a finished sequence leaves the batch after its last step and a queued request takes its slot at the next one, so the GPU keeps working on a full batch. vLLM, TensorRT-LLM, and llama.cpp's server all implement it. Because throughput depends on how full each batch is, a single request on an idle [inference engine](https://theaiengineering.com/glossary/inference-engine) says little about what the same engine delivers under load.

## How continuous batching works

Continuous batching runs the model one iteration at a time over whichever sequences are active. After each iteration, the scheduler removes sequences that produced an end token or hit their length limit, then admits queued requests into the freed slots, as long as there is KV cache memory for them and room in the per-step token budget.

**Figure 1: Continuous batching refills a finished request's slot at the next iteration. A static batch would hold D and E until its longest request finished.**

Four decode iterations of one batch: request A finishes after iteration 2, queued request D takes its slot in iteration 3, and E replaces C in iteration 4.

Layers, top to bottom:

1. **Iteration 1**: A, B, and C decode (A, B, C)
2. **Iteration 2**: A emits its end token (A, B, C)
3. **Iteration 3**: Queued D takes A's slot (D, B, C)
4. **Iteration 4**: C finished; queued E joins (D, B, E)

The idea was introduced as iteration-level scheduling in [Orca](https://www.usenix.org/conference/osdi22/presentation/yu) (Yu et al., OSDI 2022). Orca paired it with selective batching, which applies batching to only a selected set of operations so that requests of different lengths can share an iteration. The authors report 36.9 times higher throughput than NVIDIA FasterTransformer at the same latency on a GPT-3 175B model.

Two details decide how well continuous batching works in practice. First, every admitted request needs KV cache space, and [PagedAttention](https://theaiengineering.com/glossary/paged-attention) allocates that space in small blocks on demand, so more sequences fit. Second, a new request must run its prefill before it can decode. [TensorRT-LLM's documentation](https://nvidia.github.io/TensorRT-LLM/latest/features/paged-attention-ifb-scheduler.html), which calls the feature in-flight batching, states that sequences in the context (prefill) phase can be processed together with sequences in the generation phase. vLLM batches pending decodes first and fills the remaining per-step budget with chunks of new prompts.

## Continuous batching vs static batching

Static batching groups a fixed set of requests and runs them together until the longest one finishes. Continuous batching changes the batch membership at every iteration. Under static batching, slots sit idle after short requests finish and new arrivals wait for the whole batch, which costs both throughput and queueing time.

| | Static batching | Continuous batching |
|---|---|---|
| Scheduling unit | A whole request | One decode iteration |
| A request finishes early | Its slot stays idle until the batch ends | Its slot is refilled at the next iteration |
| A new request arrives | Waits for the current batch to complete | Joins at the next iteration if memory allows |

Larger batches raise total throughput but make each decode step slower, so continuous batching trades [inter-token latency](https://theaiengineering.com/glossary/inter-token-latency) for throughput as concurrency grows. [vLLM's optimization guide](https://docs.vllm.ai/en/latest/configuration/optimization/) exposes that trade-off through `max_num_batched_tokens`: smaller values improve inter-token latency, and larger values improve [time to first token](https://theaiengineering.com/glossary/time-to-first-token) and throughput. Pick the setting against a latency target measured at realistic concurrency.

The [LLM inference optimization guide](https://theaiengineering.com/articles/llm-inference-optimization) explains how continuous batching combines with chunked prefill, prefix caching, and paged KV memory.

## Related terms

- [PagedAttention](https://theaiengineering.com/glossary/paged-attention): PagedAttention is an attention algorithm, introduced with vLLM, that stores each sequence's KV cache in fixed-size blocks placed anywhere in GPU memory and tracked by a block table, an idea taken from operating-system paging.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.
- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.

## Guides that cover continuous batching

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is continuous batching in LLM serving?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/continuous-batching

- Title: What is continuous batching in LLM serving?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/continuous-batching
- Markdown: https://theaiengineering.com/glossary/continuous-batching.md
