What is LLM streaming? Token streaming and SSE explained

Also called token streaming, streaming responses, server-sent events

Definition

LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.

LLM streaming is a response mode in which a model API sends output incrementally while the model generates it, instead of returning one complete response at the end. Setting stream: true on OpenAI's or Anthropic's API switches the response to server-sent events (SSE), a series of small events that each carry a fragment of text or a change of state. Users see text after the time to first token instead of after the last token, and the gap between the two grows with the length of the answer.

How LLM streaming works#

LLM streaming works over one long-lived HTTP response with the text/event-stream content type. The server writes an event each time the model produces new output, and the client parses each event as it arrives and appends the text. Event names and payloads differ by provider, so clients and gateways need a parser for each format.

The WHATWG HTML standard defines the SSE format: an event is a block of data, event, id, and retry fields, a blank line dispatches it, and a line starting with a colon is a comment. Providers define their own event types on top:

  • OpenAI's streaming guide describes typed events for the Responses API, such as response.created, response.output_text.delta, response.completed, and error.
  • Anthropic's streaming documentation defines a fixed flow: message_start, then a start, one or more deltas, and a stop event for each content block, then message_delta and message_stop, with ping events in between.
A client sends one streaming request; the API answers with a 200 event stream carrying message_start, repeated content deltas, a message_delta with usage, and message_stop. ClientParses events as they arriveModel APIWrites events as tokens are generated1POST, stream: true2200, text/event-stream3message_start4content_block_delta, repeated5message_delta: stop reason, usage6message_stop
Figure 1: One Anthropic Messages stream: a single request, then server-sent events until message_stop.

Agent protocols use the same transport. MCP's Streamable HTTP transport lets a server answer any request with a single JSON object or an SSE stream, and the A2A protocol lists SSE among the existing standards it builds on.

What changes when you stream#

Streaming changes how an application handles errors, usage, tool calls, and moderation, because the response is no longer one object. A stream can fail after the HTTP 200 status has been sent, final token counts arrive near the end, and tool arguments arrive as fragments that are not valid JSON until their block closes.

  • Errors can arrive inside the stream. Anthropic's streams can carry an error event such as overloaded_error, which corresponds to HTTP 529 in a non-streaming call. Anthropic documents recovering an interrupted stream by saving the partial text and sending a continuation request; tool use and thinking blocks cannot be partially recovered.
  • Usage arrives last. The token counts in Anthropic's message_delta events are cumulative, so the final cost is known only when the stream ends.
  • Tool calling arguments stream as partial JSON strings in Anthropic's input_json_delta events, and the client parses them once the content block stops.
  • Moderation gets harder. OpenAI's guide notes that streaming makes completions harder to moderate, because partial output is harder to evaluate, so output guardrails must buffer the stream or check it incrementally.

Streaming leaves total generation time unchanged and moves the first visible output earlier. Two metrics describe a stream: time to first token and inter-token latency, the gap between later tokens. Every proxy on the path must forward events as they arrive, because an AI gateway or reverse proxy that buffers the response holds back the first token until it flushes. The AI gateway explainer covers how gateways handle streamed responses.

Cite this definition

The AI Engineering Editors. "What is LLM streaming? Token streaming and SSE explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-streaming

Title
What is LLM streaming? Token streaming and SSE explained
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM inference and serving hub · Updated