What is LLM streaming? Token streaming and SSE explained
Also called token streaming, streaming responses, server-sent events
Definition
LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.
LLM streaming is a response mode in which a model API sends output incrementally while the model generates it, instead of returning one complete response at the end. Setting stream: true on OpenAI's or Anthropic's API switches the response to server-sent events (SSE), a series of small events that each carry a fragment of text or a change of state. Users see text after the time to first token instead of after the last token, and the gap between the two grows with the length of the answer.
How LLM streaming works#
LLM streaming works over one long-lived HTTP response with the text/event-stream content type. The server writes an event each time the model produces new output, and the client parses each event as it arrives and appends the text. Event names and payloads differ by provider, so clients and gateways need a parser for each format.
The WHATWG HTML standard defines the SSE format: an event is a block of data, event, id, and retry fields, a blank line dispatches it, and a line starting with a colon is a comment. Providers define their own event types on top:
- OpenAI's streaming guide describes typed events for the Responses API, such as
response.created,response.output_text.delta,response.completed, anderror. - Anthropic's streaming documentation defines a fixed flow:
message_start, then a start, one or more deltas, and a stop event for each content block, thenmessage_deltaandmessage_stop, withpingevents in between.
Agent protocols use the same transport. MCP's Streamable HTTP transport lets a server answer any request with a single JSON object or an SSE stream, and the A2A protocol lists SSE among the existing standards it builds on.
What changes when you stream#
Streaming changes how an application handles errors, usage, tool calls, and moderation, because the response is no longer one object. A stream can fail after the HTTP 200 status has been sent, final token counts arrive near the end, and tool arguments arrive as fragments that are not valid JSON until their block closes.
- Errors can arrive inside the stream. Anthropic's streams can carry an
errorevent such asoverloaded_error, which corresponds to HTTP 529 in a non-streaming call. Anthropic documents recovering an interrupted stream by saving the partial text and sending a continuation request; tool use and thinking blocks cannot be partially recovered. - Usage arrives last. The token counts in Anthropic's
message_deltaevents are cumulative, so the final cost is known only when the stream ends. - Tool calling arguments stream as partial JSON strings in Anthropic's
input_json_deltaevents, and the client parses them once the content block stops. - Moderation gets harder. OpenAI's guide notes that streaming makes completions harder to moderate, because partial output is harder to evaluate, so output guardrails must buffer the stream or check it incrementally.
Streaming leaves total generation time unchanged and moves the first visible output earlier. Two metrics describe a stream: time to first token and inter-token latency, the gap between later tokens. Every proxy on the path must forward events as they arrive, because an AI gateway or reverse proxy that buffers the response holds back the first token until it flushes. The AI gateway explainer covers how gateways handle streamed responses.
Related terms
Guides that cover llm streaming
- What is an AI gateway? How it works and when you need one
- AI engineer roadmap: skills, projects, and resources by stage
- LLM observability: what to capture, which metrics, and tools
- What is AI engineering? The discipline, the stack, and the work
- A2A protocol vs MCP: differences and when to use each
- Semantic caching for LLMs: how it works and when it pays off
- Reasoning models: how they work and when to use them
Cite this definition
The AI Engineering Editors. "What is LLM streaming? Token streaming and SSE explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-streaming
- Title
- What is LLM streaming? Token streaming and SSE explained
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated