What is time to first token (TTFT) in LLMs?
Also called TTFT, first token latency
Definition
Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
Time to first token is the latency metric closest to what a user perceives in a streaming interface: how long before text starts to appear. NVIDIA's NIM benchmarking guide defines TTFT as the time from query submission to the first received token, including request queuing, prefill, and network latency. Teams usually track TTFT as percentiles such as p50 and p95, because tail latency under load can differ sharply from the median.
What affects time to first token#
- Prompt length. The server runs prefill over the entire input and builds the KV cache before it can produce the first token, so longer prompts raise TTFT.
- Queuing and load. Under heavy traffic, a request waits for capacity before its prefill starts.
- Cache hits. Prompt caching skips prefill for a previously seen prefix, which makes it one of the most direct ways to cut TTFT for applications with long system prompts.
- Model size and hardware. Larger models and slower accelerators take longer to complete prefill.
- Reasoning. A reasoning model can generate many reasoning tokens first, so the first visible answer token arrives later than the first generated token.
- Network hops. Each proxy or gateway between client and model adds round-trip time.
Time to first token vs latency#
TTFT measures how quickly a response starts. Inter-token latency (ITL), also called time per output token (TPOT), measures the average gap between subsequent tokens. End-to-end latency is the time to the complete response.
Interactive chat and voice applications are most sensitive to TTFT. Batch jobs that only use the final output care more about end-to-end latency and throughput. Serving engine choice also affects both metrics; see vLLM vs Ollama.
In production, track TTFT per model, provider, and route with LLM observability tooling. The OpenTelemetry GenAI semantic conventions define a gen_ai.server.time_to_first_token histogram metric for this purpose, and the LLM inference optimization guide discusses ways to reduce it.
Related terms
Guides that cover time to first token
- LLM inference optimization: a practical guide for engineers
- LLM observability: what to capture, which metrics, and tools
- vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use
- What is an AI gateway? How it works and when you need one
- Reasoning models: how they work and when to use them
- RAG vs fine-tuning: how to choose for your LLM application
Cite this definition
The AI Engineering Editors. "What is time to first token (TTFT) in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/time-to-first-token
- Title
- What is time to first token (TTFT) in LLMs?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated