---
title: "What is time to first token (TTFT) in LLMs?"
description: "Time to first token (TTFT) is the delay between sending an LLM request and receiving the first output token. What affects TTFT and how it differs from latency."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/time-to-first-token"
markdown_url: "https://theaiengineering.com/glossary/time-to-first-token.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Time to first token","TTFT","first token latency"]
citation: "The AI Engineering Editors. \"What is time to first token (TTFT) in LLMs?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/time-to-first-token"
---

# What is time to first token (TTFT) in LLMs?

Also called TTFT, first token latency.

> **Definition of Time to first token:** Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.

Time to first token is the latency metric closest to what a user perceives in a [streaming](https://theaiengineering.com/glossary/llm-streaming) interface: how long before text starts to appear. [NVIDIA's NIM benchmarking guide](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html) defines TTFT as the time from query submission to the first received token, including request queuing, [prefill](https://theaiengineering.com/glossary/prefill-and-decode), and network latency. Teams usually track TTFT as percentiles such as p50 and p95, because tail latency under load can differ sharply from the median.

## What affects time to first token

- **Prompt length.** The server runs prefill over the entire input and builds the [KV cache](https://theaiengineering.com/glossary/kv-cache) before it can produce the first token, so longer prompts raise TTFT.
- **Queuing and load.** Under heavy traffic, a request waits for capacity before its prefill starts.
- **Cache hits.** [Prompt caching](https://theaiengineering.com/glossary/prompt-caching) skips prefill for a previously seen prefix, which makes it one of the most direct ways to cut TTFT for applications with long system prompts.
- **Model size and hardware.** Larger models and slower accelerators take longer to complete prefill.
- **Reasoning.** A [reasoning model](https://theaiengineering.com/glossary/reasoning-model) can generate many reasoning [tokens](https://theaiengineering.com/glossary/tokens) first, so the first visible answer token arrives later than the first generated token.
- **Network hops.** Each proxy or gateway between client and model adds round-trip time.

## Time to first token vs latency

TTFT measures how quickly a response starts. [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency) (ITL), also called time per output token (TPOT), measures the average gap between subsequent tokens. End-to-end latency is the time to the complete response.

**Figure 1: End-to-end latency is roughly TTFT plus ITL times (N − 1). Timings are illustrative.**

Timeline of one streaming request: TTFT covers queueing and prefill up to the first token, then decode emits tokens 2 to N one ITL apart.

Spans:

- End-to-end latency: 0 to 1000 ms
  - TTFT: 0 to 360 ms
    - Queue: 0 to 90 ms
    - Prefill: 90 to 360 ms
  - Tokens 2 to N: 360 to 1000 ms
    - ITL: 360 to 440 ms (× (N − 1))

Markers:

- 360 ms: first token

Interactive chat and voice applications are most sensitive to TTFT. Batch jobs that only use the final output care more about end-to-end latency and throughput. Serving engine choice also affects both metrics; see [vLLM vs Ollama](https://theaiengineering.com/articles/vllm-vs-ollama).

In production, track TTFT per model, provider, and route with [LLM observability](https://theaiengineering.com/glossary/llm-observability) tooling. The OpenTelemetry GenAI semantic conventions define a `gen_ai.server.time_to_first_token` histogram metric for this purpose, and the [LLM inference optimization guide](https://theaiengineering.com/articles/llm-inference-optimization) discusses ways to reduce it.

## Related terms

- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [Reasoning model](https://theaiengineering.com/glossary/reasoning-model): A reasoning model is a large language model trained, usually with reinforcement learning, to generate intermediate reasoning tokens before its final answer, spending more inference compute to solve multi-step problems.
- [LLM observability](https://theaiengineering.com/glossary/llm-observability): LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.
- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
- [LLM streaming](https://theaiengineering.com/glossary/llm-streaming): LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.

## Guides that cover time to first token

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is time to first token (TTFT) in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/time-to-first-token

- Title: What is time to first token (TTFT) in LLMs?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/time-to-first-token
- Markdown: https://theaiengineering.com/glossary/time-to-first-token.md
