---
title: "What is inter-token latency (ITL) in LLM serving?"
description: "Inter-token latency (ITL), or TPOT, is the average gap between streamed LLM tokens after the first. How it is measured and how it trades off against throughput."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/inter-token-latency"
markdown_url: "https://theaiengineering.com/glossary/inter-token-latency.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Inter-token latency","ITL","time per output token","TPOT"]
citation: "The AI Engineering Editors. \"What is inter-token latency (ITL) in LLM serving?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/inter-token-latency"
---

# What is inter-token latency (ITL) in LLM serving?

Also called ITL, time per output token, TPOT.

> **Definition of Inter-token latency:** Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.

Inter-token latency (ITL) is the average gap between output tokens once a streaming response has started. [NVIDIA's NIM benchmarking guide](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html) defines it as the average time between consecutive tokens, notes that it is also known as time per output token (TPOT), and computes it as end-to-end latency minus [time to first token](https://theaiengineering.com/glossary/time-to-first-token), divided by the number of output tokens minus one. ITL comes from the decode phase, so it reflects how long one forward pass takes at the current batch size. For long outputs, such as reasoning traces or generated code, ITL accounts for most of the end-to-end latency.

## How inter-token latency is measured

Inter-token latency is measured per request from token timestamps: subtract time to first token from end-to-end latency, then divide by the output token count minus one. Benchmarks report percentiles such as p50 and p95 across requests. Tools differ in the details, so check whether a reported figure includes the first token before comparing numbers.

For example, a response with 201 output tokens, a time to first token of 0.4 seconds, and an end-to-end latency of 2.0 seconds has an ITL of (2.0 - 0.4) / 200, or 8 ms. NVIDIA's guide notes that its AIPerf tool excludes time to first token from ITL and that other tools differ on this point.

Some engines track the two names separately. [vLLM's metrics design](https://docs.vllm.ai/en/latest/design/metrics/) records `vllm:inter_token_latency_seconds` as the gap between successive streamed outputs and `vllm:request_time_per_output_token_seconds` as the per-request average computed with the formula above. The two diverge when one streamed chunk carries several tokens, which happens with [speculative decoding](https://theaiengineering.com/glossary/speculative-decoding).

ITL rises with batch size, with model size relative to memory bandwidth, and when long prefills are scheduled alongside decodes. Chunked prefill and running [prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode) on separate GPUs protect it from prefill interference, while quantization shortens each decode step and speculative decoding emits more tokens per step.

## Inter-token latency vs throughput

Inter-token latency describes one user's experience, while throughput describes the server's total output. Raising concurrency usually raises total tokens per second until the GPU saturates, and each request's ITL grows because every decode step processes more sequences. Serving targets therefore pair a throughput goal with an ITL limit.

| | Inter-token latency | Throughput |
|---|---|---|
| Measures | Gap between one request's tokens | Tokens or requests per second across all users |
| Unit | Milliseconds per token | Tokens per second |
| Effect of a larger batch | Usually worse | Usually better, until the GPU saturates |
| Matters most for | Interactive and streaming use | Cost per token and capacity planning |

NVIDIA's guide states that total tokens per second rises with concurrency until the system saturates its compute, while tokens per second per user falls as latency increases. For long outputs, per-user speed approaches 1/ITL, so an 8 ms ITL is about 125 tokens per second for that user. The [DistServe paper](https://arxiv.org/abs/2401.09670) builds on this pairing with goodput: the highest request rate a deployment can serve while meeting both its time-to-first-token and time-per-output-token targets.

In engines that use [continuous batching](https://theaiengineering.com/glossary/continuous-batching), batch limits are the main control for this trade-off. Track ITL percentiles in production with [LLM observability](https://theaiengineering.com/glossary/llm-observability) tooling, and see the [LLM inference optimization guide](https://theaiengineering.com/articles/llm-inference-optimization) for techniques that lower ITL and the ones that trade it for throughput.

## Related terms

- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.
- [Continuous batching](https://theaiengineering.com/glossary/continuous-batching): Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.
- [Speculative decoding](https://theaiengineering.com/glossary/speculative-decoding): Speculative decoding is an inference technique where a cheap draft proposes a few tokens and the target model verifies them in one forward pass, emitting several tokens per step without changing the output distribution.
- [LLM observability](https://theaiengineering.com/glossary/llm-observability): LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.
- [LLM streaming](https://theaiengineering.com/glossary/llm-streaming): LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.

## Guides that cover inter-token latency

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is inter-token latency (ITL) in LLM serving?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/inter-token-latency

- Title: What is inter-token latency (ITL) in LLM serving?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/inter-token-latency
- Markdown: https://theaiengineering.com/glossary/inter-token-latency.md
