---
title: "What is tool calling? How LLM function calling works"
description: "Tool calling lets an LLM request actions from your code through structured calls. Learn the request loop, parallel calls, structured outputs, security, and MCP."
summary: "Tool calling is the mechanism by which a language model returns a structured request to run a function that your application executes. This explainer walks through the loop with a provider-neutral JSON example and covers parallel calls, strict schemas, tool design, errors, security, and MCP."
type: "explainer"
canonical_url: "https://theaiengineering.com/articles/tool-calling"
markdown_url: "https://theaiengineering.com/articles/tool-calling.md"
topic: "Prompt and context engineering"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-08-08"
updated: "2026-08-08"
keywords: ["tool calling","function calling","function calling vs tool calling","tool calling vs mcp","structured outputs","parallel tool calling"]
citation: "The AI Engineering Editors. \"What is tool calling? How LLM function calling works.\" The AI Engineering, updated Aug 08, 2026. https://theaiengineering.com/articles/tool-calling"
---

# What is tool calling? How LLM function calling works

## TL;DR

- Tool calling is a loop: the application sends tool definitions with JSON Schema parameters, the model returns a structured call, the application runs it, and the result goes back to the model.
- Function calling and tool calling name the same mechanism; tool calling is the broader term because providers also offer built-in tools such as web search and code execution.
- Strict modes constrain tool arguments to the tool's JSON Schema, which prevents malformed arguments but not wrong ones, so applications still validate inputs and authorize every action.
- Fixable tool errors should go back to the model as results with a specific message, while timeouts, bounded retries, and idempotency belong in application code.
- MCP does not replace tool calling: an MCP client discovers tools on servers and passes them to the model, which still invokes them through the provider's tool calling interface.

Tool calling is the mechanism that lets a large language model request an action outside itself, such as querying a database, calling an API, or running code, by returning a structured call that the application executes. Tool calling, also called function calling, is the foundation of AI agents: every agent step that reads from or changes the outside world is a tool call.

This explainer walks through the loop end to end with a provider-neutral JSON example, then covers parallel calls, structured outputs, tool design, error handling, security, and how the Model Context Protocol fits on top.

## What is tool calling?

Tool calling is an API feature in which a model, given tool definitions, returns a structured request naming a tool and its arguments instead of plain text. The model never executes the tool. The application runs the call, sends the result back, and the model continues with that result in its context.

A [large language model](https://theaiengineering.com/glossary/large-language-model) only generates tokens, and Google's Gemini documentation states that the application, not the model, executes the function. Tool calling adds a contract: the model emits a call that matches a schema, and your code determines whether and how to act on it.

Providers also distinguish tools by where they run. Anthropic separates client tools, which run in your application, from server tools such as web search and code execution, which run on Anthropic's infrastructure. An [AI agent](https://theaiengineering.com/glossary/ai-agent) is, at its core, a loop of tool calls that continues until a task is done.

## Function calling vs tool calling

Function calling and tool calling name the same mechanism. OpenAI introduced it as function calling in June 2023, and most providers now use "tool" as the umbrella term because a tool can also be a provider-run capability. A function is the most common kind of tool: one your code implements and describes with a [JSON Schema](https://theaiengineering.com/glossary/json-schema).

Field names differ across APIs even though the shape is the same:

| Part | OpenAI Responses API | Anthropic Messages API | MCP |
|---|---|---|---|
| Tool definition | `name`, `description`, `parameters` | `name`, `description`, `input_schema` | `name`, `description`, `inputSchema` |
| Model's call | `function_call` item with `call_id` and JSON-string `arguments` | `tool_use` block with `id` and object `input` | Client sends `tools/call` with `name` and `arguments` |
| Result | `function_call_output` with `call_id` and `output` | `tool_result` with `tool_use_id`, `content`, optional `is_error` | `content`, optional `structuredContent`, `isError` |
| Force or block calls | `tool_choice`: auto, required, none, or a named function | `tool_choice`: auto, any, tool, or none | Not applicable |
| Disable parallel calls | `parallel_tool_calls: false` | `disable_parallel_tool_use: true` | Not applicable |

Agent frameworks hide these differences behind one interface; the [agent frameworks comparison](https://theaiengineering.com/articles/ai-agent-frameworks) covers the options.

## How does tool calling work?

Tool calling works as a loop between your application and the model. The application sends the conversation plus tool definitions; the model returns either a final answer or one or more tool calls; the application runs each call and appends the results; and the loop repeats until the model returns an answer with no further calls.

**Figure 1: The tool calling loop. Steps 2 through 4 repeat until the model returns no tool calls.**

Sequence between an application and a model API: send messages and tools, receive a tool call, run it, send the result with the same id, receive the final answer.

Steps:

1. Application → Model API: messages + tool definitions
2. Model API → Application: tool call: id, name, args
3. Application → Application: validate, authorize, execute
4. Application → Model API: result with the same id
5. Model API → Application: final answer

### Step 1: define the tool

A tool definition has a name, a description, and a JSON Schema for its parameters. The provider-neutral example below maps onto each API's field names in the table above.

```json
{
  "name": "get_order_status",
  "description": "Look up the shipping status of a customer order. Use when the user asks where an order is or when it will arrive.",
  "parameters": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "Order ID in the form ORD-12345"
      },
      "include_items": {
        "type": "boolean",
        "description": "Whether to include line items in the result"
      }
    },
    "required": ["order_id", "include_items"],
    "additionalProperties": false
  }
}
```

### Step 2: the model returns a tool call

When a user asks, "Where is order ORD-48213?" the model's response contains a call instead of, or in addition to, text. The call carries an ID that the result must reference. OpenAI returns `arguments` as a JSON-encoded string that your code parses; Anthropic returns `input` as a parsed object.

```json
{
  "type": "tool_call",
  "id": "call_01",
  "name": "get_order_status",
  "arguments": { "order_id": "ORD-48213", "include_items": false }
}
```

### Steps 3 and 4: execute and return the result

The application validates the arguments, checks that the current user may see this order, runs the lookup, and sends the result back with the same ID. The model then generates the user-facing answer, or another tool call if the task needs one.

```json
{
  "type": "tool_result",
  "call_id": "call_01",
  "content": "{\"status\": \"shipped\", \"carrier\": \"UPS\", \"eta\": \"2026-11-09\"}"
}
```

The loop itself is short code, and both OpenAI and Anthropic ship SDK helpers that run it. The [OpenAI function calling guide](https://developers.openai.com/api/docs/guides/function-calling) and [Anthropic's tool use documentation](https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview) both show the full round trip in several languages.

## Parallel tool calling

Parallel tool calling means the model returns several independent tool calls in one response, such as weather lookups for three cities, and the application runs them concurrently before the next model call. Parallel calls cut round trips and latency, but every call needs its own result matched by ID, and calls with side effects need extra care.

OpenAI's guide advises assuming a response can contain zero, one, or several calls, and setting `parallel_tool_calls` to `false` when exactly zero or one call is wanted. Anthropic requires all results from a parallel batch to come back in a single user message, with the tool results placed before any text. Gemini supports both parallel calls and compositional calls, where one call's output feeds the next.

Practical rules for parallel calls:

- **Run independent calls concurrently**, and return every result with its matching ID, including a per-call error when one call fails.
- **Disable parallel calls for ordered side effects**, such as "create the account, then send the welcome email."
- **Cap the batch size** in application code, because a single response can request more calls than your [rate limits](https://theaiengineering.com/glossary/rate-limiting) allow.

## Structured outputs vs tool calling

Structured outputs constrain a model's response to a supplied JSON Schema, and the same mechanism powers strict tool calling. Use tool calling when the model should trigger an action or fetch data from your system. Use a structured response format when you only need the final answer in a fixed shape, such as an extraction result that your code parses.

With `strict: true`, OpenAI's API makes function calls reliably adhere to the schema instead of best effort. Strict mode requires `additionalProperties: false` on every object and every property listed as required; optional fields are expressed as a union with `null`. Anthropic's strict tool use enforces JSON Schema compliance on tool inputs through grammar-constrained sampling. Gemini offers a validated mode for function schema adherence.

| Approach | What it guarantees | Good fit |
|---|---|---|
| Asking for JSON in the prompt | Nothing; parse and retry on failure | Prototypes and models without schema support |
| JSON mode | Syntactically valid JSON, not schema adherence | Cases where any JSON object is acceptable |
| [Structured outputs](https://theaiengineering.com/glossary/structured-outputs) as response format | Final response matches the schema | Extraction, classification, UI rendering |
| Strict tool calling | Tool arguments match the tool's schema | Actions and data lookups |

OpenAI describes structured outputs as the evolution of JSON mode and recommends them where supported. Two limits remain: each provider supports only a subset of JSON Schema, and schema-valid is not the same as correct. A strict call can still name the wrong order ID or a value the model invented, which is a [hallucination](https://theaiengineering.com/glossary/hallucination) in a well-formed wrapper.

## How to improve tool calling: tool design guidelines

Tool calling accuracy depends heavily on tool design. Clear names, specific descriptions, tight schemas with enums, a small set of distinct tools, and compact results produce fewer wrong calls. OpenAI's guidance proposes an "intern test": a person given only the tool definitions should be able to use each tool correctly.

Guidelines drawn from OpenAI's function calling guide and Anthropic's engineering post on writing tools for agents:

- **Describe purpose, parameters, and output.** State what the tool does, when to use it and when not to, the format of each parameter, and what the result means.
- **Use enums and structure to rule out invalid states.** OpenAI's example is that `toggle_light(on: bool, off: bool)` permits a contradictory call; a single enum does not.
- **Do not ask the model for values your code already has.** If the session knows the user ID or order ID, inject it in code instead of exposing it as a parameter.
- **Consolidate steps that always run together.** Anthropic recommends a `search_contacts` tool over a `list_contacts` tool that returns everything for the model to filter.
- **Name parameters without ambiguity**, such as `user_id` instead of `user`, and namespace related tools with a shared prefix so tools from different services stay distinct.
- **Keep the initial tool set small.** OpenAI gives fewer than 20 functions at the start of a turn as a soft target, and both OpenAI and Anthropic offer tool search so rarely used tools load only when needed.
- **Return high-signal results.** Resolve opaque IDs to readable names, paginate long lists, truncate logs, and consider a parameter that switches between concise and detailed output.

Tool definitions and results both occupy the [context window](https://theaiengineering.com/glossary/context-window), so tool design is also a [context engineering](https://theaiengineering.com/articles/context-engineering) decision. Test any change on realistic multi-step tasks before shipping it.

## Handling tool errors and retries

Tool errors fall into two classes: errors the model can fix, such as a missing argument or an unknown ID, and errors it cannot, such as an outage. Return fixable errors to the model as tool results with a specific message. Handle the rest in code with timeouts, bounded retries, and idempotency for side-effecting actions.

The MCP specification draws the same line. Protocol errors, such as an unknown tool or a request that fails the schema, return as [JSON-RPC](https://theaiengineering.com/glossary/json-rpc) errors. Tool execution errors, such as API failures, input validation errors, and business logic errors, return inside the result with `isError: true` and carry feedback the model can use to correct its next call. Anthropic's API uses an `is_error` flag on tool results for the same purpose, and its documentation recommends instructive messages, such as naming the rate limit and when to retry, over a bare "failed."

| Error | Example | Handled by | Response |
|---|---|---|---|
| Invalid arguments | Order ID in the wrong format | Model | Error result explaining the expected format |
| Not found | Order does not exist | Model | Error result; the model can ask the user |
| Transient failure | Timeout or HTTP 503 from the service | Application | Retry with backoff, then report the failure |
| Permission denied | User may not view this order | Application | Error result that does not leak data |
| Loop or repetition | Same failing call repeated | Application | Stop after a set number of attempts |

Three rules belong in application code. Validate arguments against the schema even when strict mode is on, because the schema cannot express every business rule. Attach an idempotency key to any write so that a retried call after a timeout does not create a second order or payment. And cap tool calls per task and consecutive failures, logging every call in your [LLM observability](https://theaiengineering.com/articles/llm-observability) traces.

## Tool calling security

Tool calling turns model output into actions, so security depends on what happens when that output is wrong or manipulated. The main risks are prompt injection through tool results, tools with more permission than the task needs, and data exfiltration. Controls belong in application code: least-privilege tools, authorization outside the model, input validation, and [human approval](https://theaiengineering.com/glossary/human-in-the-loop) for high-impact actions.

Any tool result that contains outside content, such as a web page, an email, or a support ticket, can carry instructions written by an attacker. This indirect [prompt injection](https://theaiengineering.com/glossary/prompt-injection) works because the model processes tool results and instructions in the same context. Treat tool output as untrusted data, and never rely on the [system prompt](https://theaiengineering.com/glossary/system-prompt) alone to stop a manipulated call.

The [OWASP entry on Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) traces damaging actions to three root causes: excessive functionality, excessive permissions, and excessive autonomy. Its mitigations translate directly into tool design:

- **Expose only the tools a workflow needs**, and prefer narrow tools over open-ended ones such as unrestricted shell access or arbitrary URL fetching.
- **Run tools with the requesting user's permissions**, for example through OAuth with narrow scopes, not a shared administrator credential.
- **Enforce authorization in the downstream system** rather than asking the model whether an action is allowed.
- **Require human approval** before irreversible or high-impact actions, such as payments, deletions, or outbound messages.

The MCP specification adds that a person should be able to deny tool invocations, and that clients must treat tool annotations, such as a hint that a tool is read-only, as untrusted unless the server is trusted. [Guardrails](https://theaiengineering.com/glossary/guardrails) can screen arguments and results, but they complement these controls rather than replace them.

## Tool calling vs MCP

Tool calling is the model-level mechanism for requesting an action; the Model Context Protocol is a standard for how applications discover and invoke tools hosted on external servers. MCP sits on top of tool calling: an MCP client lists a server's tools, passes them to the model in the provider's tool format, and forwards the model's calls to the server.

**Figure 2: MCP standardizes the connection between the application and tool servers; tool calling handles the connection to the model.**

The host application with an MCP client sits between the model API, reached through tool calling, and four MCP servers, reached through MCP.

Components:

- **Model API**: Tool definitions in, tool calls out in the provider's tool format
- **Host application**: MCP client converts MCP tools and routes each tool call over MCP: tools/list, tools/call
- **Issue tracker**
- **Database**
- **Docs**
- **Internal APIs**

Connections:

- Host application → Model API: tool calling
- Host application → Issue tracker: MCP
- Host application → Database: MCP
- Host application → Docs: MCP
- Host application → Internal APIs: MCP

In the [current MCP tools specification](https://modelcontextprotocol.io/specification/2026-07-28/server/tools), each tool has a name, a description, an `inputSchema`, and optionally an `outputSchema` and behavior annotations. Results can include `structuredContent` that conforms to the output schema, and servers can notify clients when their tool list changes. Some providers can also connect to MCP servers from inside their API; OpenAI lists MCP access among its built-in tools.

The choice between them depends on how widely a tool will be reused. Direct function calling suits a few tools that belong to one application. An [MCP server](https://theaiengineering.com/glossary/mcp-server) suits integrations that several agents, editors, or teams reuse without rewriting tool code for each provider; the [Model Context Protocol explainer](https://theaiengineering.com/articles/what-is-model-context-protocol) covers the architecture.

## Where to go next

Tool calling is the mechanism every agent depends on, so the next steps build on it: [context engineering](https://theaiengineering.com/articles/context-engineering) explains how tool definitions and results consume the context window, and [what are AI agents](https://theaiengineering.com/articles/what-are-ai-agents) shows how tool calls combine into multi-step loops. For related guides on prompts, context, and tool use, see the [prompt engineering topic hub](https://theaiengineering.com/topics/prompt-engineering).

## Frequently asked questions

### Is function calling the same as tool calling?

Yes, in practice. OpenAI introduced the feature as function calling in June 2023, and providers now use tool calling as the umbrella term. A function is a tool that your own code implements and describes with a JSON Schema. Tools also include provider-run capabilities such as web search or code execution, which the provider executes on its own infrastructure.

### Is MCP just tool calling?

No. Tool calling is how a model requests an action within one API call. The Model Context Protocol is a client-server standard that lets an application discover tools, resources, and prompts on external servers and invoke them over JSON-RPC. An MCP client still hands the discovered tools to the model through ordinary tool calling, so MCP builds on tool calling rather than replacing it.

### What is the difference between tool calling and RAG?

Retrieval-augmented generation fetches relevant documents and inserts them into the prompt before the model generates an answer. Tool calling lets the model request a specific action or lookup during generation, such as querying a database or calling an API. Many agents implement retrieval as a tool, so the model can issue a search call when a step needs external knowledge.

### Can an LLM call an API directly?

Not by itself. A language model only generates tokens. With tool calling, the model generates a structured request naming the API operation and its arguments, and the application or the provider's platform makes the actual HTTP call. That separation is what lets the application validate arguments, enforce permissions, and log every action before anything reaches the API.

### Is there a benchmark for tool calling?

The Berkeley Function Calling Leaderboard (BFCL) from UC Berkeley is a widely used public benchmark for tool calling. Its current version, V4, adds agentic evaluation to the single-turn and multi-turn tests of earlier versions. Public scores help shortlist models, but tool accuracy depends on your own tool definitions, so test candidates against your real tools and tasks.

## Terms in this guide

- [Tool calling](https://theaiengineering.com/glossary/tool-calling): Tool calling is an LLM capability in which the model, given a list of tool definitions, outputs a structured request naming a tool and its arguments, which the application executes and returns.
- [Structured outputs](https://theaiengineering.com/glossary/structured-outputs): Structured outputs is an LLM API feature that constrains generation so the model's response always conforms to a developer-supplied JSON Schema, producing output that code can parse without retries.
- [Model Context Protocol (MCP)](https://theaiengineering.com/glossary/model-context-protocol): Model Context Protocol (MCP) is an open protocol, introduced by Anthropic in November 2024, that standardizes how AI applications connect to external tools, data sources, and prompts through MCP servers.
- [MCP server](https://theaiengineering.com/glossary/mcp-server): An MCP server is a program that exposes tools, resources, and prompts to AI applications through the Model Context Protocol, running locally over stdio or remotely over Streamable HTTP.
- [Prompt injection](https://theaiengineering.com/glossary/prompt-injection): Prompt injection is an attack in which text supplied to a large language model, typed by a user or hidden in content the model reads, alters the application's intended behavior or output.
- [AI agent](https://theaiengineering.com/glossary/ai-agent): An AI agent is a software system in which a large language model runs in a loop, calling tools, reading their results, and repeating until the task is complete or a stop condition is reached.
- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Guardrails](https://theaiengineering.com/glossary/guardrails): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits.
- [JSON Schema](https://theaiengineering.com/glossary/json-schema): JSON Schema is a declarative vocabulary, itself written in JSON, for describing the structure of JSON data; LLM APIs and MCP use it to define tool arguments and constrain model output.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [JSON-RPC](https://theaiengineering.com/glossary/json-rpc): JSON-RPC is a lightweight, transport-agnostic remote procedure call protocol that encodes method calls, results, and errors as JSON objects; version 2.0 is the message format of MCP and A2A.
- [Human-in-the-loop](https://theaiengineering.com/glossary/human-in-the-loop): Human-in-the-loop (HITL) is an agent design in which the run pauses at defined checkpoints, such as before a sensitive tool call, so a person can approve, edit, or reject the action.
- [System prompt](https://theaiengineering.com/glossary/system-prompt): A system prompt is the set of developer-written instructions sent to a language model ahead of the conversation, defining its role, rules, output format, and tool use for every turn that follows.

## Related guides

- [What is the Model Context Protocol (MCP)? How it works](https://theaiengineering.com/articles/what-is-model-context-protocol): The Model Context Protocol (MCP) is an open standard for connecting AI apps to tools and data. Learn its architecture, transports, tool calls, and security.
- [What is context engineering? A practical guide for AI agents](https://theaiengineering.com/articles/context-engineering): Context engineering is the practice of choosing what fills an LLM context window. Learn its components, why context rot happens, and how to measure results.
- [What are AI agents? How agents work and when to use one](https://theaiengineering.com/articles/what-are-ai-agents): AI agents are LLM systems that call tools in a loop until a task is done. Learn how the loop works, agents vs agentic AI, examples, failure modes, and limits.
- [AI agent frameworks compared: LangGraph, CrewAI, ADK, and more](https://theaiengineering.com/articles/ai-agent-frameworks): An AI agent frameworks comparison of LangGraph, LangChain, CrewAI, the OpenAI, Claude, and Google SDKs, Pydantic AI, Mastra, and more, by language and license.

Part of the [Prompt and context engineering](https://theaiengineering.com/topics/prompt-engineering) topic hub.

## Cite this guide

The AI Engineering Editors. "What is tool calling? How LLM function calling works." The AI Engineering, updated Aug 08, 2026. https://theaiengineering.com/articles/tool-calling

- Title: What is tool calling? How LLM function calling works
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-08-08
- Updated: 2026-08-08
- URL: https://theaiengineering.com/articles/tool-calling
- Markdown: https://theaiengineering.com/articles/tool-calling.md

To cite one section, link to its anchor:

- [What is tool calling?](https://theaiengineering.com/articles/tool-calling#what-is-tool-calling)
- [Function calling vs tool calling](https://theaiengineering.com/articles/tool-calling#function-calling-vs-tool-calling)
- [How does tool calling work?](https://theaiengineering.com/articles/tool-calling#how-does-tool-calling-work)
- [Parallel tool calling](https://theaiengineering.com/articles/tool-calling#parallel-tool-calling)
- [Structured outputs vs tool calling](https://theaiengineering.com/articles/tool-calling#structured-outputs-vs-tool-calling)
- [How to improve tool calling: tool design guidelines](https://theaiengineering.com/articles/tool-calling#how-to-improve-tool-calling-tool-design-guidelines)
- [Handling tool errors and retries](https://theaiengineering.com/articles/tool-calling#handling-tool-errors-and-retries)
- [Tool calling security](https://theaiengineering.com/articles/tool-calling#tool-calling-security)
- [Tool calling vs MCP](https://theaiengineering.com/articles/tool-calling#tool-calling-vs-mcp)
- [Where to go next](https://theaiengineering.com/articles/tool-calling#where-to-go-next)
- [Frequently asked questions](https://theaiengineering.com/articles/tool-calling#faq)
