What is tool calling? How LLM function calling works
Tool calling is the mechanism that lets a large language model request an action outside itself, such as querying a database, calling an API, or running code, by returning a structured call that the application executes. Tool calling, also called function calling, is the foundation of AI agents: every agent step that reads from or changes the outside world is a tool call.
This explainer walks through the loop end to end with a provider-neutral JSON example, then covers parallel calls, structured outputs, tool design, error handling, security, and how the Model Context Protocol fits on top.
What is tool calling?#
Tool calling is an API feature in which a model, given tool definitions, returns a structured request naming a tool and its arguments instead of plain text. The model never executes the tool. The application runs the call, sends the result back, and the model continues with that result in its context.
A large language model only generates tokens, and Google's Gemini documentation states that the application, not the model, executes the function. Tool calling adds a contract: the model emits a call that matches a schema, and your code determines whether and how to act on it.
Providers also distinguish tools by where they run. Anthropic separates client tools, which run in your application, from server tools such as web search and code execution, which run on Anthropic's infrastructure. An AI agent is, at its core, a loop of tool calls that continues until a task is done.
Function calling vs tool calling#
Function calling and tool calling name the same mechanism. OpenAI introduced it as function calling in June 2023, and most providers now use "tool" as the umbrella term because a tool can also be a provider-run capability. A function is the most common kind of tool: one your code implements and describes with a JSON Schema.
Field names differ across APIs even though the shape is the same:
| Part | OpenAI Responses API | Anthropic Messages API | MCP |
|---|---|---|---|
| Tool definition | name, description, parameters |
name, description, input_schema |
name, description, inputSchema |
| Model's call | function_call item with call_id and JSON-string arguments |
tool_use block with id and object input |
Client sends tools/call with name and arguments |
| Result | function_call_output with call_id and output |
tool_result with tool_use_id, content, optional is_error |
content, optional structuredContent, isError |
| Force or block calls | tool_choice: auto, required, none, or a named function |
tool_choice: auto, any, tool, or none |
Not applicable |
| Disable parallel calls | parallel_tool_calls: false |
disable_parallel_tool_use: true |
Not applicable |
Agent frameworks hide these differences behind one interface; the agent frameworks comparison covers the options.
How does tool calling work?#
Tool calling works as a loop between your application and the model. The application sends the conversation plus tool definitions; the model returns either a final answer or one or more tool calls; the application runs each call and appends the results; and the loop repeats until the model returns an answer with no further calls.
Step 1: define the tool#
A tool definition has a name, a description, and a JSON Schema for its parameters. The provider-neutral example below maps onto each API's field names in the table above.
{
"name": "get_order_status",
"description": "Look up the shipping status of a customer order. Use when the user asks where an order is or when it will arrive.",
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"description": "Order ID in the form ORD-12345"
},
"include_items": {
"type": "boolean",
"description": "Whether to include line items in the result"
}
},
"required": ["order_id", "include_items"],
"additionalProperties": false
}
}Step 2: the model returns a tool call#
When a user asks, "Where is order ORD-48213?" the model's response contains a call instead of, or in addition to, text. The call carries an ID that the result must reference. OpenAI returns arguments as a JSON-encoded string that your code parses; Anthropic returns input as a parsed object.
{
"type": "tool_call",
"id": "call_01",
"name": "get_order_status",
"arguments": { "order_id": "ORD-48213", "include_items": false }
}Steps 3 and 4: execute and return the result#
The application validates the arguments, checks that the current user may see this order, runs the lookup, and sends the result back with the same ID. The model then generates the user-facing answer, or another tool call if the task needs one.
{
"type": "tool_result",
"call_id": "call_01",
"content": "{\"status\": \"shipped\", \"carrier\": \"UPS\", \"eta\": \"2026-11-09\"}"
}The loop itself is short code, and both OpenAI and Anthropic ship SDK helpers that run it. The OpenAI function calling guide and Anthropic's tool use documentation both show the full round trip in several languages.
Parallel tool calling#
Parallel tool calling means the model returns several independent tool calls in one response, such as weather lookups for three cities, and the application runs them concurrently before the next model call. Parallel calls cut round trips and latency, but every call needs its own result matched by ID, and calls with side effects need extra care.
OpenAI's guide advises assuming a response can contain zero, one, or several calls, and setting parallel_tool_calls to false when exactly zero or one call is wanted. Anthropic requires all results from a parallel batch to come back in a single user message, with the tool results placed before any text. Gemini supports both parallel calls and compositional calls, where one call's output feeds the next.
Practical rules for parallel calls:
- Run independent calls concurrently, and return every result with its matching ID, including a per-call error when one call fails.
- Disable parallel calls for ordered side effects, such as "create the account, then send the welcome email."
- Cap the batch size in application code, because a single response can request more calls than your rate limits allow.
Structured outputs vs tool calling#
Structured outputs constrain a model's response to a supplied JSON Schema, and the same mechanism powers strict tool calling. Use tool calling when the model should trigger an action or fetch data from your system. Use a structured response format when you only need the final answer in a fixed shape, such as an extraction result that your code parses.
With strict: true, OpenAI's API makes function calls reliably adhere to the schema instead of best effort. Strict mode requires additionalProperties: false on every object and every property listed as required; optional fields are expressed as a union with null. Anthropic's strict tool use enforces JSON Schema compliance on tool inputs through grammar-constrained sampling. Gemini offers a validated mode for function schema adherence.
| Approach | What it guarantees | Good fit |
|---|---|---|
| Asking for JSON in the prompt | Nothing; parse and retry on failure | Prototypes and models without schema support |
| JSON mode | Syntactically valid JSON, not schema adherence | Cases where any JSON object is acceptable |
| Structured outputs as response format | Final response matches the schema | Extraction, classification, UI rendering |
| Strict tool calling | Tool arguments match the tool's schema | Actions and data lookups |
OpenAI describes structured outputs as the evolution of JSON mode and recommends them where supported. Two limits remain: each provider supports only a subset of JSON Schema, and schema-valid is not the same as correct. A strict call can still name the wrong order ID or a value the model invented, which is a hallucination in a well-formed wrapper.
How to improve tool calling: tool design guidelines#
Tool calling accuracy depends heavily on tool design. Clear names, specific descriptions, tight schemas with enums, a small set of distinct tools, and compact results produce fewer wrong calls. OpenAI's guidance proposes an "intern test": a person given only the tool definitions should be able to use each tool correctly.
Guidelines drawn from OpenAI's function calling guide and Anthropic's engineering post on writing tools for agents:
- Describe purpose, parameters, and output. State what the tool does, when to use it and when not to, the format of each parameter, and what the result means.
- Use enums and structure to rule out invalid states. OpenAI's example is that
toggle_light(on: bool, off: bool)permits a contradictory call; a single enum does not. - Do not ask the model for values your code already has. If the session knows the user ID or order ID, inject it in code instead of exposing it as a parameter.
- Consolidate steps that always run together. Anthropic recommends a
search_contactstool over alist_contactstool that returns everything for the model to filter. - Name parameters without ambiguity, such as
user_idinstead ofuser, and namespace related tools with a shared prefix so tools from different services stay distinct. - Keep the initial tool set small. OpenAI gives fewer than 20 functions at the start of a turn as a soft target, and both OpenAI and Anthropic offer tool search so rarely used tools load only when needed.
- Return high-signal results. Resolve opaque IDs to readable names, paginate long lists, truncate logs, and consider a parameter that switches between concise and detailed output.
Tool definitions and results both occupy the context window, so tool design is also a context engineering decision. Test any change on realistic multi-step tasks before shipping it.
Handling tool errors and retries#
Tool errors fall into two classes: errors the model can fix, such as a missing argument or an unknown ID, and errors it cannot, such as an outage. Return fixable errors to the model as tool results with a specific message. Handle the rest in code with timeouts, bounded retries, and idempotency for side-effecting actions.
The MCP specification draws the same line. Protocol errors, such as an unknown tool or a request that fails the schema, return as JSON-RPC errors. Tool execution errors, such as API failures, input validation errors, and business logic errors, return inside the result with isError: true and carry feedback the model can use to correct its next call. Anthropic's API uses an is_error flag on tool results for the same purpose, and its documentation recommends instructive messages, such as naming the rate limit and when to retry, over a bare "failed."
| Error | Example | Handled by | Response |
|---|---|---|---|
| Invalid arguments | Order ID in the wrong format | Model | Error result explaining the expected format |
| Not found | Order does not exist | Model | Error result; the model can ask the user |
| Transient failure | Timeout or HTTP 503 from the service | Application | Retry with backoff, then report the failure |
| Permission denied | User may not view this order | Application | Error result that does not leak data |
| Loop or repetition | Same failing call repeated | Application | Stop after a set number of attempts |
Three rules belong in application code. Validate arguments against the schema even when strict mode is on, because the schema cannot express every business rule. Attach an idempotency key to any write so that a retried call after a timeout does not create a second order or payment. And cap tool calls per task and consecutive failures, logging every call in your LLM observability traces.
Tool calling security#
Tool calling turns model output into actions, so security depends on what happens when that output is wrong or manipulated. The main risks are prompt injection through tool results, tools with more permission than the task needs, and data exfiltration. Controls belong in application code: least-privilege tools, authorization outside the model, input validation, and human approval for high-impact actions.
Any tool result that contains outside content, such as a web page, an email, or a support ticket, can carry instructions written by an attacker. This indirect prompt injection works because the model processes tool results and instructions in the same context. Treat tool output as untrusted data, and never rely on the system prompt alone to stop a manipulated call.
The OWASP entry on Excessive Agency traces damaging actions to three root causes: excessive functionality, excessive permissions, and excessive autonomy. Its mitigations translate directly into tool design:
- Expose only the tools a workflow needs, and prefer narrow tools over open-ended ones such as unrestricted shell access or arbitrary URL fetching.
- Run tools with the requesting user's permissions, for example through OAuth with narrow scopes, not a shared administrator credential.
- Enforce authorization in the downstream system rather than asking the model whether an action is allowed.
- Require human approval before irreversible or high-impact actions, such as payments, deletions, or outbound messages.
The MCP specification adds that a person should be able to deny tool invocations, and that clients must treat tool annotations, such as a hint that a tool is read-only, as untrusted unless the server is trusted. Guardrails can screen arguments and results, but they complement these controls rather than replace them.
Tool calling vs MCP#
Tool calling is the model-level mechanism for requesting an action; the Model Context Protocol is a standard for how applications discover and invoke tools hosted on external servers. MCP sits on top of tool calling: an MCP client lists a server's tools, passes them to the model in the provider's tool format, and forwards the model's calls to the server.
In the current MCP tools specification, each tool has a name, a description, an inputSchema, and optionally an outputSchema and behavior annotations. Results can include structuredContent that conforms to the output schema, and servers can notify clients when their tool list changes. Some providers can also connect to MCP servers from inside their API; OpenAI lists MCP access among its built-in tools.
The choice between them depends on how widely a tool will be reused. Direct function calling suits a few tools that belong to one application. An MCP server suits integrations that several agents, editors, or teams reuse without rewriting tool code for each provider; the Model Context Protocol explainer covers the architecture.
Where to go next#
Tool calling is the mechanism every agent depends on, so the next steps build on it: context engineering explains how tool definitions and results consume the context window, and what are AI agents shows how tool calls combine into multi-step loops. For related guides on prompts, context, and tool use, see the prompt engineering topic hub.
Frequently asked questions
Is function calling the same as tool calling?
- Yes, in practice. OpenAI introduced the feature as function calling in June 2023, and providers now use tool calling as the umbrella term. A function is a tool that your own code implements and describes with a JSON Schema. Tools also include provider-run capabilities such as web search or code execution, which the provider executes on its own infrastructure.
Is MCP just tool calling?
- No. Tool calling is how a model requests an action within one API call. The Model Context Protocol is a client-server standard that lets an application discover tools, resources, and prompts on external servers and invoke them over JSON-RPC. An MCP client still hands the discovered tools to the model through ordinary tool calling, so MCP builds on tool calling rather than replacing it.
What is the difference between tool calling and RAG?
- Retrieval-augmented generation fetches relevant documents and inserts them into the prompt before the model generates an answer. Tool calling lets the model request a specific action or lookup during generation, such as querying a database or calling an API. Many agents implement retrieval as a tool, so the model can issue a search call when a step needs external knowledge.
Can an LLM call an API directly?
- Not by itself. A language model only generates tokens. With tool calling, the model generates a structured request naming the API operation and its arguments, and the application or the provider's platform makes the actual HTTP call. That separation is what lets the application validate arguments, enforce permissions, and log every action before anything reaches the API.
Is there a benchmark for tool calling?
- The Berkeley Function Calling Leaderboard (BFCL) from UC Berkeley is a widely used public benchmark for tool calling. Its current version, V4, adds agentic evaluation to the single-turn and multi-turn tests of earlier versions. Public scores help shortlist models, but tool accuracy depends on your own tool definitions, so test candidates against your real tools and tasks.
Terms in this guide
Cite this guide
The AI Engineering Editors. "What is tool calling? How LLM function calling works." The AI Engineering, updated Aug 08, 2026. https://theaiengineering.com/articles/tool-calling
- Title
- What is tool calling? How LLM function calling works
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Published
- Updated