What is time to first token (TTFT) in LLMs?

Also called TTFT, first token latency

Definition

Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.

Time to first token is the latency metric closest to what a user perceives in a streaming interface: how long before text starts to appear. NVIDIA's NIM benchmarking guide defines TTFT as the time from query submission to the first received token, including request queuing, prefill, and network latency. Teams usually track TTFT as percentiles such as p50 and p95, because tail latency under load can differ sharply from the median.

What affects time to first token#

  • Prompt length. The server runs prefill over the entire input and builds the KV cache before it can produce the first token, so longer prompts raise TTFT.
  • Queuing and load. Under heavy traffic, a request waits for capacity before its prefill starts.
  • Cache hits. Prompt caching skips prefill for a previously seen prefix, which makes it one of the most direct ways to cut TTFT for applications with long system prompts.
  • Model size and hardware. Larger models and slower accelerators take longer to complete prefill.
  • Reasoning. A reasoning model can generate many reasoning tokens first, so the first visible answer token arrives later than the first generated token.
  • Network hops. Each proxy or gateway between client and model adds round-trip time.

Time to first token vs latency#

TTFT measures how quickly a response starts. Inter-token latency (ITL), also called time per output token (TPOT), measures the average gap between subsequent tokens. End-to-end latency is the time to the complete response.

Timeline of one streaming request: TTFT covers queueing and prefill up to the first token, then decode emits tokens 2 to N one ITL apart. 0 ms250 ms500 ms750 ms1000 msEnd-to-end latencyTTFTQueuePrefillTokens 2 to NITL× (N − 1)First token
Figure 1: End-to-end latency is roughly TTFT plus ITL times (N − 1). Timings are illustrative.

Interactive chat and voice applications are most sensitive to TTFT. Batch jobs that only use the final output care more about end-to-end latency and throughput. Serving engine choice also affects both metrics; see vLLM vs Ollama.

In production, track TTFT per model, provider, and route with LLM observability tooling. The OpenTelemetry GenAI semantic conventions define a gen_ai.server.time_to_first_token histogram metric for this purpose, and the LLM inference optimization guide discusses ways to reduce it.

Cite this definition

The AI Engineering Editors. "What is time to first token (TTFT) in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/time-to-first-token

Title
What is time to first token (TTFT) in LLMs?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM inference and serving hub · Updated