Function Calling (Tool Calling)

Definition

The model never runs your function. Function calling is the mechanism by which a language model asks your program to do something. You send the model a list of tools — each one a name, a sentence of description, and a JSON Schema for its arguments — and instead of replying in prose, the model can reply with a structured request: get_weather({"city": "Lisbon"}). That request is text. The model has no network socket, no file handle, no database connection and no shell. Your code receives the request, decides whether to honour it, runs the actual function, and sends the return value back as another message — and only then does the model write an answer.

Almost every confusing thing about the feature dissolves once that is clear. The model cannot "accidentally delete the database", because it cannot delete anything; a program you wrote can, after you read a JSON object and chose to pass it to your delete function. There is no sandbox to configure, because the sandbox is your dispatch code, and its strength is whatever you put in it. Equally, no amount of writing "only call this for internal users" in a tool description enforces anything — descriptions are prompt text, and prompt text is a suggestion. The check has to live in the function.

The naming is worth getting right because half the documentation you will read predates the other half. OpenAI shipped the feature on 13 June 2023 for gpt-4-0613 and gpt-3.5-turbo-0613, as a functions parameter with a matching function_call. At DevDay on 6 November 2023 those were superseded by tools and tool_choice, partly so that a single response could carry more than one call. "Function calling" and "tool calling" now name the same mechanism, providers use both, and a code sample using functions= is simply from 2023.

How It Works

Four messages, and only one of them does anything

A tool-calling turn is a conversation with a specific shape. Nothing about it is magic; it is four messages and one if statement.

First request. You send the usual system and user messages plus an array of tool definitions. A minimal one is a name, a description, and a JSON Schema: {"name": "get_weather", "description": "Current conditions for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}. The description matters more than the schema does — it is the only thing telling the model when this tool is the right one, and it is read as prompt text, not as documentation.

The model's reply. If the model decides a tool is needed, the response comes back with a tool_calls array instead of (or alongside) content, each entry carrying an id, a name, and an arguments string that your code must parse. The response's finish reason says tool_calls. This message is the entire output of the model for this turn. It has changed nothing in the world.

Your executor. Your code parses the arguments, looks the name up in a dispatch table, checks that this caller is allowed to invoke that function with those arguments, and runs it. This step is ordinary software. It is also the only place any policy exists.

The result goes back as a message. You append the model's own tool-call message to the transcript, then a message with role tool, carrying the same tool_call_id and the return value serialised as a string, and you send the whole transcript again. The API is stateless: the second request repeats everything the first one contained, plus two more messages. The model reads the result and writes the final answer — or requests another tool, and the loop goes round again.

Termination is a model decision. The loop ends when a reply arrives with no tool calls in it. That is the only signal you get, which is why every production loop also carries a hard iteration cap: a model that keeps re-requesting a tool that keeps failing will otherwise spin until it fills the context window. What to do when the tool itself fails — a timeout, a 500, or arguments that are perfectly valid JSON and still name a customer who does not exist — is the subject of error handling in AI systems, and the short version is that the failure goes back to the model as a tool message like any other result.

The arithmetic of a round trip

One tool call costs a minimum of two model invocations: one to emit the call, one to interpret the result. A chain of n dependent calls costs n + 1. That is the single most useful number on this page, because it converts directly into latency and cost.

Put illustrative timings on it. If a model turn takes about 2 seconds and a tool call about 200 ms, a plain completion answers in ~2 s, a single tool call in ~4.2 s, and a three-step chain in ~8.6 s — more than four times the latency of the answer the user thinks they asked for. Those timings are an illustration rather than a measurement, but the multiplier is structural: you cannot have fewer than two model turns, and every additional dependent step buys another full one.

You re-send every tool definition on every request

Because the API is stateless, the whole tool array is part of each request in the loop. That makes the schema block a fixed tax paid repeatedly, and it is larger than most teams estimate. A measurement filed against the Model Context Protocol specification in 2026 counted the tokens in eleven production tools: 6,807 in total, from 103 for a trivial stats call to 1,024 for a batch executor — a mean of about 620 tokens per tool. The same report puts a typical session of 20–30 registered tools at 15–30 KB of context before the user has typed anything.

Run that through the round-trip table. A three-step chained task issues four requests, so those eleven tools cost 4 × 6,807 ≈ 27,000 tokens of schema for one task — none of it the user's problem, the model's reasoning, or the answer. This is a different cost from the one on the AI agent page, and the two add up: there, the growing transcript makes token use rise super-linearly with step count; here, a constant block is re-billed once per step.

The model is choosing from a menu it has to re-read

The context cost is the visible half. The invisible half is that a longer menu makes the choice worse — and tool-use benchmarks exist to catch exactly this, because the failure hides. What they score is not whether your code ran but whether the model picked the right tool, filled its arguments with usable values, and invented nothing: the three errors that recur are the wrong tool, malformed arguments, and hallucinated ones — argument values the model made up that are syntactically fine and factually baseless. A JSON Schema on your side rejects the malformed case and is silent on the other two.

Anthropic's November 2025 report on deferred tool loading is the cleanest evidence that menu length is itself one of those error sources, because it holds the model and the available tools fixed and varies only whether all the schemas sit in context. Loading definitions on demand cut a library of fifty-plus tools from ~77K tokens to ~8.7K — about an 85% reduction — and lifted the model's measured tool-selection accuracy: on the weaker of the two configurations Anthropic tested, by roughly half in relative terms, bought by showing the model fewer options at once rather than by changing the model. The absolute scores are model-specific and move as models do; the direction and the size of the effect are the durable part.

The mechanism is attention dilution, and it is why the reflex to "just expose everything" backfires. A model asked to pick one of fifty tools is reading fifty descriptions written by fifty different people, several of which overlap. It is also why the Berkeley Function-Calling Leaderboard scores irrelevance detection as its own category — the ability to answer without calling anything when no tool applies. Calling the wrong tool and calling a tool at all are two separate failures, and the second one is the one people forget to measure.

Types

There is a real taxonomy here, and it is defined by round trips rather than by what the functions do. The Berkeley leaderboard's categories use these names, and so do most provider APIs.

Single calling is one tool per assistant turn: two model invocations, one result. This is the case that is now largely solved — near-saturated on the single-call benchmark categories, which is why BFCL v4 (released April 2026) reweighted most of its score toward multi-turn agentic tasks. On that harder mix nobody is close to solved: across the thirteen models a third-party tracker listed in July 2026, the leader scored 0.750 and the mean was 0.611. Treat the figures as a snapshot — leaderboards move, and the point is the gap between single-call and multi-turn, not the ranking.

Parallel calling is several tool calls in one assistant message, dispatched concurrently by your executor. It is only correct when the calls are independent — three city lookups, not "get the user id, then fetch their orders" — and when it applies, it collapses n round trips into one and makes wall-clock tool time the maximum rather than the sum. The LLMCompiler paper (Kim et al., December 2023) measured up to 3.7× lower latency and 6.7× lower cost than a sequential ReAct loop on the same tasks, with about 9% better accuracy — the accuracy gain being the surprise, and attributable to the model planning the whole set at once instead of drifting across turns. This is the capability the functions → tools rename existed to enable.

Sequential or chained calling is the dependent case, where each call needs the previous result. It cannot be parallelised, it costs n + 1 model invocations, and it is where an agentic workflow starts and this page hands off.

Cutting across all three is who decides. tool_choice is typically auto (the model may call zero, one or many), required (it must call something), a named tool (it must call that one), or none (it may not). Forcing a specific tool turns the model into a parameter extractor and removes the decision that makes function calling interesting — which is exactly what you want when you already know the operation and only need the arguments filled in.

Real-World Applications

Coding agents are function calling with a very small tool list. When Claude Code or Cursor reads a file, greps a repository, edits a line or runs a test, each of those is a tool call: the model emits edit_file({...}), the harness validates and applies it, and the diff or the test output comes back as a tool message. Nothing in the model touched the disk. This also explains why file permissions and command allow-lists live in the harness rather than in the prompt — the harness is the only component that executes anything.

MCP is a distribution mechanism for tool schemas, not a replacement for this. A Model Context Protocol server's job is to hand a client a list of tools and to execute the ones it is asked to run. What reaches the model is still a schema in the context window and what comes back is still a tool_calls array. MCP changes where the definitions come from — one protocol instead of bespoke glue per integration — and it is why the token arithmetic above suddenly mattered to everyone: connecting five servers can add dozens of tools you never chose individually.

Provider-hosted tools are the one real exception to "the model does not execute". When you enable a server-side code interpreter or web search, the provider runs the loop inside its own infrastructure and returns the result to you already resolved. The model still only emits a request; it is just that somebody else's executor honours it. Knowing which side of that line a given tool sits on determines who is liable for what it does.

Retrieval becomes a decision instead of a step. Classic RAG retrieves before generating, every time. Exposed as a tool, search becomes something the model may invoke zero times for "hello", once for a factual question, or three times with refined queries when the first result was thin — which is a better system when the retrieval is expensive and a worse one when the model declines to search and answers from memory instead.

Structured extraction with a forced tool. A large amount of production "function calling" never calls a function at all: you define a tool whose schema is the record you want, set tool_choice to that tool, and use the argument object as the output. It is the most reliable way to get schema-conformant JSON out of a model, and the function on the other end is often just a database insert.

Key Concepts

• A tool definition is prompt text with a validator attached. The JSON Schema constrains the shape of the arguments; the description string is what actually decides whether the tool gets picked. Rewording a description changes behaviour, and renaming a parameter from q to search_query measurably changes how often it is filled in correctly. This is prompt engineering wearing an API's clothes.
• The tool_call_id is what makes parallel calls tractable. Each result message quotes the id of the request it answers, so three concurrent calls can return out of order and the model still knows which weather belongs to which city. Dropping or reusing an id is a common source of confusing behaviour with no error attached.
• Your executor is a policy enforcement point, whether or not you designed it as one. The model may have read an untrusted web page one message earlier, so a tool call is an untrusted request. Authority has to come from the session — who is logged in, what they may touch — not from the arguments the model supplied.
• Tool definitions are the one part of the context you control completely. Prompts drift and transcripts grow, but the tool array is a list you wrote, and pruning it is the cheapest available intervention on both cost and accuracy.

Challenges

The context tax is charged before anyone says anything. Twenty tools at ~620 tokens each is about 12,000 tokens of overhead on every single request in the loop — and unlike a long conversation, you cannot summarise it away. Prompt caching reduces what you pay for those tokens without reducing the context window they occupy or the attention they dilute, which is why pagination and on-demand loading appeared as soon as tool libraries got past a couple of dozen entries.

The failure that leaves no trace is a tool that was never called. A model that answers a question about live inventory from its training data returns a fluent, well-formed paragraph, logs nothing, and raises nothing. It is why irrelevance and relevance detection are scored separately on BFCL, and why a tool-call rate is worth putting on a dashboard: a sudden drop is often the first sign that a description was reworded or a schema grew a required field.

Arguments can be perfectly valid and completely wrong. The schema will confirm that order_id is a string and quantity is an integer, and it has no opinion about whether that order exists. Catching that class needs something to check against — a lookup, a constraint, a second call — and it is covered in full on the error handling page rather than duplicated here.

A tool call is a confused deputy waiting to happen, and prompt injection is how that deputy gets its orders. The model is a text generator that may have just read an attacker's document, and the tool call it emits is byte-identical whether it came from the user's intent or from an instruction hidden in that document. Since the model has no notion of who is asking, the blast radius equals whatever your executor is willing to do — which is the argument for read-only tools by default, per-tool authorisation, and never letting the model choose the credentials a call runs under.

Schemas rot in a way prompts do not. Change a function's signature and the schema, the dispatch code and the model's learned habits fall out of step at different rates. Because the description is also the selection signal, tool definitions need versioning and review like an external API contract, not like an internal helper.

Future Trends

The clearest direction is not putting the tools in the context at all. Deferred loading — the model is given a search tool over the tool catalogue and pulls in definitions when it needs them — went from a workaround to a shipped provider feature in November 2025, and the large tool-selection accuracy jump above explains why it spread quickly. It also inverts the design advice of the previous two years: a large, well-described tool library is now an asset again, provided the model sees only the slice it needs.

A more radical version is replacing the calls with code. Anthropic's November 2025 write-up on code execution with MCP presents tools as importable functions in a sandboxed runtime and has the model write a short program instead of emitting one call per step. Loops, filtering and intermediate results then stay inside the sandbox instead of crossing the context window twice, and their reported example fell from 150,000 tokens to 2,000 — a 98.7% reduction. The trade is that you now need a real code sandbox, which is a much larger security surface than a dispatch table.

The third direction is standardisation above the call. MCP standardised where tool definitions come from; the Agent2Agent Protocol is doing the same for delegation between agents, having absorbed IBM's Agent Communication Protocol in August 2025. Both sit on top of this mechanism rather than replacing it — one system asking another to do something is still a structured request that somebody else's code decides whether to honour.

Code Example

The whole loop, with no provider SDK, so that the shape is visible. call_model stands in for whichever chat endpoint you use; every provider's version of this is the same five steps.

import json

TOOLS = [
    {
        "name": "get_weather",
        "description": "Current conditions for a city. Use for questions about "
                       "weather right now, not forecasts.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "City name"}},
            "required": ["city"],
        },
    }
]

def get_weather(city: str) -> dict:
    """The real function. Note that the model never reaches this line itself."""
    return {"city": city, "temp_c": 19, "condition": "cloudy"}

# The dispatch table is the security boundary. A name the model invents
# that is not a key here simply does not run.
HANDLERS = {"get_weather": get_weather}

def run(user_message: str, max_steps: int = 5) -> str:
    messages = [{"role": "user", "content": user_message}]

    for step in range(max_steps):
        reply = call_model(messages=messages, tools=TOOLS, tool_choice="auto")
        messages.append(reply)

        # Termination: an ordinary message with no tool calls means we are done.
        if not reply.get("tool_calls"):
            return reply["content"]

        # One message can carry several calls; independent ones can run concurrently.
        for call in reply["tool_calls"]:
            name = call["function"]["name"]
            args = json.loads(call["function"]["arguments"])

            handler = HANDLERS.get(name)
            if handler is None:
                result = {"error": f"unknown tool {name!r}"}
            else:
                try:
                    result = handler(**args)          # <- the only execution anywhere
                except Exception as exc:
                    # A failure is just another result. The model reads it and retries
                    # or gives up; it does not see an exception.
                    result = {"error": str(exc)}

            messages.append({
                "role": "tool",
                "tool_call_id": call["id"],           # binds this result to that request
                "content": json.dumps(result),
            })

    return "stopped: hit the step limit without a final answer"

Three things in that listing are the whole subject. handler(**args) is the only line that does anything to the world, and it is in your file, not the model's. HANDLERS.get(name) means an invented tool name is a dead end rather than an incident. And the for loop's bound is there because nothing else guarantees the model will ever stop asking.