What are structured outputs in LLMs?
Also called structured output, JSON schema mode, constrained decoding
Definition
Structured outputs is an LLM API feature that constrains generation so the model's response always conforms to a developer-supplied JSON Schema, producing output that code can parse without retries.
Structured outputs is a feature of model APIs and inference engines that guarantees a response matches a schema, usually JSON Schema, instead of relying on the model to follow a formatting instruction. OpenAI launched Structured Outputs on August 6, 2024, and reported that on its own complex-schema eval, gpt-4o-2024-08-06 with the feature scored 100%, while gpt-4-0613 scored under 40%. Anthropic and open-source servers such as vLLM offer equivalent features.
How structured outputs work#
The underlying technique is constrained decoding. The provider compiles the schema into a grammar. At each generation step the model assigns a probability to every token in its vocabulary, and the decoder masks out any token that would make the output violate the grammar. If the schema says the next value is a boolean, only tokens that can begin true or false remain eligible.
Anthropic's documentation says its structured outputs "guarantee schema-compliant responses through constrained decoding" and lists two exceptions: a refusal may not match the schema, and a response cut off at the token limit may be incomplete. Providers also support only a subset of JSON Schema. Anthropic, for example, rejects recursive schemas and numeric constraints such as minimum and maximum. vLLM supports JSON Schema, regular expressions, and context-free grammars through its xgrammar and guidance backends.
Structured outputs vs JSON mode#
| Approach | Valid JSON guaranteed | Schema match guaranteed | Typical use |
|---|---|---|---|
| Prompting for JSON | No | No | Prototypes |
| JSON mode | Yes | No | Free-form JSON objects |
| Structured outputs | Yes | Yes | Extraction, classification, API responses |
| Strict tool calling | Yes, for arguments | Yes, for arguments | Actions the application executes |
OpenAI's Structured Outputs guide states the difference directly: both features produce valid JSON, but only Structured Outputs ensures schema adherence. The same guide recommends tool calling when the model needs to connect to functions or data, and a response schema when the reply itself must have a fixed shape.
Why structured outputs matter#
Without schema enforcement, a pipeline needs parsing code, validation, and retry loops for malformed responses. Structured outputs remove that class of failure, which matters most in multi-step agents, where one malformed response can break every later step. Schema-valid output can still be wrong: a correctly typed field can contain a hallucination. Value-level checks therefore still belong in guardrails and evals, and schema design is part of prompt engineering, since clear field names and descriptions guide what the model writes into each field.
Related terms
Guides that cover structured outputs
- What is tool calling? How LLM function calling works
- What is context engineering? A practical guide for AI agents
- Agentic workflows: five patterns and when to use each
- How to build an MCP server: a Python and TypeScript tutorial
- vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use
- AI agent frameworks compared: LangGraph, CrewAI, ADK, and more
- LLM routing: strategies, fallbacks, and learned routers
- Reasoning models: how they work and when to use them
- LLM evaluation: a practical guide to evals, metrics, and judges
- What is the Model Context Protocol (MCP)? How it works
- AI engineer roadmap: skills, projects, and resources by stage
Cite this definition
The AI Engineering Editors. "What are structured outputs in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/structured-outputs
- Title
- What are structured outputs in LLMs?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated