What is inter-token latency (ITL) in LLM serving?
Also called ITL, time per output token, TPOT
Definition
Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
Inter-token latency (ITL) is the average gap between output tokens once a streaming response has started. NVIDIA's NIM benchmarking guide defines it as the average time between consecutive tokens, notes that it is also known as time per output token (TPOT), and computes it as end-to-end latency minus time to first token, divided by the number of output tokens minus one. ITL comes from the decode phase, so it reflects how long one forward pass takes at the current batch size. For long outputs, such as reasoning traces or generated code, ITL accounts for most of the end-to-end latency.
How inter-token latency is measured#
Inter-token latency is measured per request from token timestamps: subtract time to first token from end-to-end latency, then divide by the output token count minus one. Benchmarks report percentiles such as p50 and p95 across requests. Tools differ in the details, so check whether a reported figure includes the first token before comparing numbers.
For example, a response with 201 output tokens, a time to first token of 0.4 seconds, and an end-to-end latency of 2.0 seconds has an ITL of (2.0 - 0.4) / 200, or 8 ms. NVIDIA's guide notes that its AIPerf tool excludes time to first token from ITL and that other tools differ on this point.
Some engines track the two names separately. vLLM's metrics design records vllm:inter_token_latency_seconds as the gap between successive streamed outputs and vllm:request_time_per_output_token_seconds as the per-request average computed with the formula above. The two diverge when one streamed chunk carries several tokens, which happens with speculative decoding.
ITL rises with batch size, with model size relative to memory bandwidth, and when long prefills are scheduled alongside decodes. Chunked prefill and running prefill and decode on separate GPUs protect it from prefill interference, while quantization shortens each decode step and speculative decoding emits more tokens per step.
Inter-token latency vs throughput#
Inter-token latency describes one user's experience, while throughput describes the server's total output. Raising concurrency usually raises total tokens per second until the GPU saturates, and each request's ITL grows because every decode step processes more sequences. Serving targets therefore pair a throughput goal with an ITL limit.
| Inter-token latency | Throughput | |
|---|---|---|
| Measures | Gap between one request's tokens | Tokens or requests per second across all users |
| Unit | Milliseconds per token | Tokens per second |
| Effect of a larger batch | Usually worse | Usually better, until the GPU saturates |
| Matters most for | Interactive and streaming use | Cost per token and capacity planning |
NVIDIA's guide states that total tokens per second rises with concurrency until the system saturates its compute, while tokens per second per user falls as latency increases. For long outputs, per-user speed approaches 1/ITL, so an 8 ms ITL is about 125 tokens per second for that user. The DistServe paper builds on this pairing with goodput: the highest request rate a deployment can serve while meeting both its time-to-first-token and time-per-output-token targets.
In engines that use continuous batching, batch limits are the main control for this trade-off. Track ITL percentiles in production with LLM observability tooling, and see the LLM inference optimization guide for techniques that lower ITL and the ones that trade it for throughput.
Related terms
Guides that cover inter-token latency
Cite this definition
The AI Engineering Editors. "What is inter-token latency (ITL) in LLM serving?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/inter-token-latency
- Title
- What is inter-token latency (ITL) in LLM serving?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated