---
title: "What is LLM streaming? Token streaming and SSE explained"
description: "LLM streaming sends a model's output token by token as it is generated, usually over server-sent events. How it works and what changes for errors and usage."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/llm-streaming"
markdown_url: "https://theaiengineering.com/glossary/llm-streaming.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["LLM streaming","token streaming","streaming responses","server-sent events"]
citation: "The AI Engineering Editors. \"What is LLM streaming? Token streaming and SSE explained.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-streaming"
---

# What is LLM streaming? Token streaming and SSE explained

Also called token streaming, streaming responses, server-sent events.

> **Definition of LLM streaming:** LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.

LLM streaming is a response mode in which a model API sends output incrementally while the model generates it, instead of returning one complete response at the end. Setting `stream: true` on OpenAI's or Anthropic's API switches the response to server-sent events (SSE), a series of small events that each carry a fragment of text or a change of state. Users see text after the [time to first token](https://theaiengineering.com/glossary/time-to-first-token) instead of after the last [token](https://theaiengineering.com/glossary/tokens), and the gap between the two grows with the length of the answer.

## How LLM streaming works

LLM streaming works over one long-lived HTTP response with the `text/event-stream` content type. The server writes an event each time the model produces new output, and the client parses each event as it arrives and appends the text. Event names and payloads differ by provider, so clients and gateways need a parser for each format.

The [WHATWG HTML standard](https://html.spec.whatwg.org/multipage/server-sent-events.html) defines the SSE format: an event is a block of `data`, `event`, `id`, and `retry` fields, a blank line dispatches it, and a line starting with a colon is a comment. Providers define their own event types on top:

- OpenAI's [streaming guide](https://developers.openai.com/api/docs/guides/streaming-responses) describes typed events for the Responses API, such as `response.created`, `response.output_text.delta`, `response.completed`, and `error`.
- Anthropic's [streaming documentation](https://platform.claude.com/docs/en/build-with-claude/streaming) defines a fixed flow: `message_start`, then a start, one or more deltas, and a stop event for each content block, then `message_delta` and `message_stop`, with `ping` events in between.

**Figure 1: One Anthropic Messages stream: a single request, then server-sent events until message_stop.**

A client sends one streaming request; the API answers with a 200 event stream carrying message_start, repeated content deltas, a message_delta with usage, and message_stop.

Steps:

1. Client → Model API: POST, stream: true
2. Model API → Client: 200, text/event-stream
3. Model API → Client: message_start
4. Model API → Client: content_block_delta, repeated
5. Model API → Client: message_delta: stop reason, usage
6. Model API → Client: message_stop

Agent protocols use the same transport. MCP's Streamable HTTP [transport](https://theaiengineering.com/glossary/mcp-transport) lets a server answer any request with a single JSON object or an SSE stream, and the [A2A protocol](https://theaiengineering.com/glossary/a2a-protocol) lists SSE among the existing standards it builds on.

## What changes when you stream

Streaming changes how an application handles errors, usage, tool calls, and moderation, because the response is no longer one object. A stream can fail after the HTTP 200 status has been sent, final token counts arrive near the end, and tool arguments arrive as fragments that are not valid JSON until their block closes.

- Errors can arrive inside the stream. Anthropic's streams can carry an `error` event such as `overloaded_error`, which corresponds to HTTP 529 in a non-streaming call. Anthropic documents recovering an interrupted stream by saving the partial text and sending a continuation request; tool use and thinking blocks cannot be partially recovered.
- Usage arrives last. The token counts in Anthropic's `message_delta` events are cumulative, so the final cost is known only when the stream ends.
- [Tool calling](https://theaiengineering.com/glossary/tool-calling) arguments stream as partial JSON strings in Anthropic's `input_json_delta` events, and the client parses them once the content block stops.
- Moderation gets harder. OpenAI's guide notes that streaming makes completions harder to moderate, because partial output is harder to evaluate, so output [guardrails](https://theaiengineering.com/glossary/guardrails) must buffer the stream or check it incrementally.

Streaming leaves total generation time unchanged and moves the first visible output earlier. Two metrics describe a stream: time to first token and [inter-token latency](https://theaiengineering.com/glossary/inter-token-latency), the gap between later tokens. Every proxy on the path must forward events as they arrive, because an [AI gateway](https://theaiengineering.com/glossary/ai-gateway) or reverse proxy that buffers the response holds back the first token until it flushes. The [AI gateway explainer](https://theaiengineering.com/articles/what-is-an-ai-gateway) covers how gateways handle streamed responses.

## Related terms

- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Tool calling](https://theaiengineering.com/glossary/tool-calling): Tool calling is an LLM capability in which the model, given a list of tool definitions, outputs a structured request naming a tool and its arguments, which the application executes and returns.
- [Guardrails](https://theaiengineering.com/glossary/guardrails): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits.

## Guides that cover llm streaming

- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [AI engineer roadmap: skills, projects, and resources by stage](https://theaiengineering.com/articles/ai-engineer-roadmap): A practical AI engineer roadmap in seven stages, each with the skills to learn, a project to ship, and resources worth the time, from Python to production.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [What is AI engineering? The discipline, the stack, and the work](https://theaiengineering.com/articles/what-is-ai-engineering): What is AI engineering? A plain definition, how it differs from ML and software engineering, the layers of the stack, and what AI engineers do day to day.
- [A2A protocol vs MCP: differences and when to use each](https://theaiengineering.com/articles/a2a-vs-mcp): A2A protocol vs MCP: A2A connects AI agents to other agents, and MCP connects an agent to tools and data. Compare agent cards, tasks, transports, and security.
- [Semantic caching for LLMs: how it works and when it pays off](https://theaiengineering.com/articles/semantic-caching): Semantic caching returns a stored LLM response for prompts with similar meaning. Learn how a semantic cache works, how to tune it, and what never to cache.
- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is LLM streaming? Token streaming and SSE explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-streaming

- Title: What is LLM streaming? Token streaming and SSE explained
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/llm-streaming
- Markdown: https://theaiengineering.com/glossary/llm-streaming.md
