Making Tool Calling Reliable on Local Models

Tool calling is the seam where language models stop being sophisticated autocomplete and start being autonomous agents. A model that can reliably invoke search_web, run_sql, or send_email with well-formed arguments is qualitatively more useful than one that cannot, and the entire agentic software stack — from LangChain orchestrators down to bare MCP servers — treats tool calling as a first-class primitive. The problem is that outside of the frontier cloud APIs, tool calling is fragile by default. Locally-hosted open models, especially those in the 3–14 billion parameter range, fail in ways that surprise developers who are used to GPT-class reliability: they hallucinate function names, produce JSON that is a token or two away from being valid, call tools that should not be called, and skip calls that obviously should happen. Getting this working reliably requires layering several techniques, each of which addresses a different root cause. This post is a practical map of those layers.

Why Local Models Struggle

The gap between a frontier API and a locally-hosted 8B model on tool calling is not primarily about raw capability — it is about post-training. Models like GPT-4o and Claude Sonnet have been fine-tuned on enormous datasets of function-call traces, covering diverse schemas, edge cases, and failure corrections. That fine-tuning produces reliable tool-call behavior even under unusual schema shapes. Most open weights models have received substantially less of this treatment, and smaller models have even less headroom in their attention and parameter budget to internalize the protocol reliably.

Three specific weaknesses amplify the problem. First, instruction-following fidelity degrades sharply at small scales. A 7B model told to respond only with a JSON function call, nothing else, will often prefix the call with a sentence of reasoning, which breaks every downstream parser. Second, local models are acutely sensitive to the exact tokenization of the system prompt and tool-description block. Each model family — Qwen, Llama, Mistral, Hermes — uses its own chat template, with different delimiters, different placement of tool schemas, and different conventions for where the model should write its function-call token. Using the wrong template is the single most common cause of catastrophic tool-call failure, worse even than model capability. Third, most open models were not evaluated against schema-drift scenarios: tool definitions that have changed since training, schemas that use $ref indirection, or enum fields expressed as const. The model’s implicit mental model of “how tool schemas look” can silently diverge from the actual schema you have provided.

The Taxonomy of Failures

Before reaching for solutions, it is worth naming the failure modes precisely, because different techniques fix different problems.

Malformed JSON arguments. The model produces output that is syntactically invalid JSON: trailing commas, unquoted keys, truncated strings, mismatched brackets. These are the easiest to detect (a JSON parser tells you immediately) and the easiest to fix with constrained decoding. They are common in smaller models even on simple schemas.

Wrong or hallucinated tool name. The model calls search_documents when the registered tool is search_files, or invents get_weather_forecast when no such tool exists. This failure is harder to catch without explicit name validation; it passes JSON parse but fails schema validation.

Wrong argument types or values. The tool is called correctly but an integer is passed as a string, a required field is missing, an enum value is fabricated (“high-priority” instead of the schema’s “high”), or a nested object is flattened into a string. Constrained decoding handles some of these; others require semantic validation.

Spurious tool calls (false positives). The model calls a tool when the user’s request does not require one. Common in models that have been overfitted to “always use tools if they are available.” This inflates latency and API costs and can have real side effects when the tool does something write.

Missing tool calls (false negatives). The model answers in natural language when it should have delegated to a tool. Often caused by a long, complex system prompt that buries the tool-use instruction, or by the model deciding it “knows” the answer.

Schema drift. Your tool definitions evolve — you rename a field, add a required argument, or change a type — but the model’s behavior does not update because it has developed a prior from earlier turns or from its pretraining. This is especially insidious in long sessions.

Semantically wrong but structurally valid calls. The model produces perfectly valid JSON that passes every schema check, but the argument values are nonsensical given the user’s intent. Constrained decoding cannot help here; only LLM-based or test-execution-based validation can catch it.

Constrained Decoding: The Single Biggest Lever

Constrained decoding (also called structured generation or grammar-based sampling) modifies the token selection process so that the model physically cannot emit a token sequence that violates a given formal constraint. At each step the logits of all out-of-grammar tokens are set to negative infinity, and sampling proceeds normally among the remaining tokens. The result is that the model’s output is guaranteed to be syntactically valid with respect to the grammar — malformed JSON, wrong field names (when enumerated), and wrong types become structurally impossible.

GBNF Grammars in llama.cpp

llama.cpp implements a BNF variant called GBNF (GGML BNF) that can be passed to the server via the grammar parameter. The project ships a Python converter, examples/json_schema_to_grammar.py, that translates a JSON Schema object into a GBNF grammar string. A fragment for a simple two-field tool-call schema looks like this:

1
2
3
4
5
6
7
8
9

root   ::= object
object ::= "{" ws "\"name\"" ws ":" ws string ws "," ws "\"arguments\"" ws ":" ws args-object ws "}"
args-object ::= "{" ws args-pairs ws "}"
args-pairs  ::= args-pair ("," ws args-pair)*
args-pair   ::= "\"query\"" ws ":" ws string
             | "\"max_results\"" ws ":" ws number
string ::= "\"" ([^"\\] | "\\" .)* "\""
number ::= [0-9]+ ("." [0-9]+)?
ws     ::= [ \t\n]*

Pass this to llama-server as --grammar-file mycall.gbnf or via the /completion endpoint’s grammar field. The server applies the grammar mask at every token step and the model cannot deviate. Note a known limitation as of mid-2026: the converter has bugs with PCRE shorthands (\d, \w, \s) in pattern constraints — rewrite those patterns as explicit character classes [0-9], [a-zA-Z0-9_] and so on if the converter fails.

Outlines, lm-format-enforcer, and XGrammar

For Python-based inference stacks, three libraries dominate the constrained generation space, each with a different architecture:

Outlines builds a finite-state machine (FSM) from the target schema, precomputes the valid-token mask for every FSM state, and performs a state lookup at each decoding step. It is fast (the mask is precomputed, not recomputed on the fly) and supports JSON Schema, regex, and Python Enum constraints. In RAG benchmarks with one-shot prompting it achieved roughly 93% success on structured-output tasks, rising to ~97% with two-shot context.

lm-format-enforcer performs dynamic on-the-fly constraint checking rather than full precomputation, which makes it more flexible for formats that partially depend on prior model output and better suited to multi-turn RAG flows. It trades some throughput for flexibility.

XGrammar uses a pushdown automaton that maintains a persistent execution stack, making it better suited to deeply nested, recursively defined formats like complex JSON. As of March 2026 it became the default structured generation backend for vLLM, SGLang, and TensorRT-LLM, with benchmarks showing under 40 microseconds per-token overhead — effectively zero at normal generation speeds. The follow-up paper, XGrammar-2, extends it to handle agentic multi-step output patterns.

A minimal Outlines example that enforces a tool-call schema:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

import outlines
import json
from pydantic import BaseModel
from typing import Literal

class SearchArgs(BaseModel):
    query: str
    max_results: int

class ToolCall(BaseModel):
    name: Literal["search_files", "read_file", "list_dir"]
    arguments: SearchArgs

model = outlines.models.transformers("Qwen/Qwen3-8B-Instruct", device="cuda")
generator = outlines.generate.json(model, ToolCall)

prompt = "<|im_start|>user\nFind all Python files modified today<|im_end|>\n<|im_start|>assistant\n"
result: ToolCall = generator(prompt)
print(result.name, result.arguments)

The result object is a validated Pydantic instance — it cannot be a malformed JSON string.

Structured Output via vLLM and Ollama

Both major local inference servers expose constrained generation without requiring you to manage the grammar yourself.

vLLM accepts a guided_json field in its chat completion request body (which takes a JSON Schema object) or guided_grammar for raw EBNF. Since v0.8.x, XGrammar is the default backend:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-14B-Instruct",
    "messages": [{"role": "user", "content": "List the top 3 log files"}],
    "tools": [...],
    "tool_choice": "required",
    "guided_json": {
      "type": "object",
      "properties": {
        "name": {"type": "string", "enum": ["list_files", "read_file"]},
        "arguments": {"type": "object"}
      },
      "required": ["name", "arguments"]
    }
  }'

Ollama exposes constrained generation via the format parameter. Passing "format": "json" guarantees valid JSON; passing a full JSON Schema object (supported since Ollama 0.4.x) applies schema-level constraints at the token level using llama.cpp’s GBNF backend. Ollama’s native tool-calling endpoint (tools parameter in /api/chat) also applies light structural constraints on the tool-call output for supported model templates.

For a deeper look at running these inference servers, see the posts on local LLMs with Ollama and self-hosted AI inference.

The Chat Template Problem

Constrained decoding fixes structural validity, but it does nothing for the upstream problem of the model not knowing where or how to emit a tool call. Every model family has a distinct chat template — a jinja2 template baked into the tokenizer_config.json — that controls exactly how system messages, human turns, assistant turns, and tool-call tokens are serialized into the token stream before the model sees them.

The differences are not cosmetic. They involve:

• Different special tokens. Qwen3 uses <|im_start|> / <|im_end|> with role names as text. Llama 4 uses <|start_header_id|> / <|end_header_id|>. Mistral uses [INST] / [/INST] with no explicit system-turn delimiters.
• Different tool-definition placement. Hermes-format models expect a <tools> XML block inside the system message. Llama 4 encodes tools as a JSON array in a dedicated tools turn role. Qwen3 natively supports both its own format and Hermes-style via a template switch.
• Different tool-call tokens. Qwen3-Instruct wraps tool calls in <tool_call> / </tool_call> tags. Llama 4 uses a Python-callable syntax introduced in Llama 3.2 and retained in Llama 4 (the “pythonic” parser in vLLM). Mistral tool calls are delimited by [TOOL_CALLS] and require 9-digit call IDs.

Using the wrong template — say, feeding a Llama 4 model through a Mistral template because you copy-pasted a config — produces one of two outcomes: the model generates tool calls in its native format which your parser does not understand, or the model stops generating tool calls at all because the instruction signal is garbled. Both look like “the model doesn’t support tool calling” when the real problem is plumbing.

The rule is: always load the template from the model’s own tokenizer_config.json. Inference frameworks that do this correctly (vLLM, llama.cpp via --chat-template, Ollama via model-bundled Modelfiles) will handle it automatically for well-known architectures. The danger zone is rolling your own prompt construction code without testing it against the model’s tokenizer.

A note on known template bugs: Qwen3 had a chat-template crash on parallel tool calls and a “thinking bleed” issue (thinking-mode tokens leaking into the tool call body) that were patched in August 2025. If you are running an older Qwen3 GGUF, verify the template version in the model card.

The Practical Reliability Stack

Constrained decoding and correct templates together get you most of the way, but a production-grade pipeline needs several more layers. The diagram below shows a full constrained-generation / validate-retry pipeline:

User turn
    |
    v
+----------------------------+
|  Build prompt with         |
|  correct chat template     |
|  + tool schema injection   |
+----------------------------+
    |
    v
+----------------------------+
|  Inference engine          |
|  (vLLM / llama.cpp /       |
|   Ollama)                  |
|  with constrained decoding |
|  (XGrammar / GBNF)         |
+----------------------------+
    |
    v
+----------------------------+
|  Structural validation     |
|  - JSON parse              |
|  - Schema validation       |
|    (jsonschema / Pydantic) |
|  - Name in tool registry?  |
+----------------------------+
    |              |
    | valid        | invalid (up to N retries)
    |              +----> append error + raw output
    v                     to context, re-run
+----------------------------+
|  Semantic validation       |
|  (optional)                |
|  - required args present?  |
|  - arg values plausible?   |
|  - LLM-as-judge check      |
+----------------------------+
    |
    v
+----------------------------+
|  Execute tool call         |
|  Catch runtime exceptions  |
|  Return result to context  |
+----------------------------+
    |
    v
  Next turn / final answer

Step 1: Pick a model trained for tool use. The baseline matters. Not all open models have equivalent tool-call post-training even at the same parameter count. See the model comparison table below.

Step 2: Use the correct template and parser. As discussed above. Do not manually construct prompts for tool-calling unless you have verified the template character for character.

Step 3: Constrain output to the JSON schema. Pass the exact schema of the expected tool call to your inference engine’s guided-generation endpoint. Enumerate allowed tool names in the schema.

Step 4: Validate and retry on failure. Even with constrained decoding, semantic failures still occur. A simple retry loop — append the failed output and a structured error message to the conversation context, then regenerate — recovers from many errors. Two to three retries cover the majority of recoverable failures; beyond that the model is usually confused at a deeper level.

Step 5: Keep the tool surface small. Reliability degrades roughly logarithmically with the number of registered tools, because the model must attend to a larger schema. If your tool list exceeds 8–10 entries, consider dynamic tool selection: a lightweight first pass retrieves the 3–5 most relevant tools, and only those are presented to the model in the second pass.

Step 6: Write tool descriptions precisely. Vague tool names produce ambiguous calls. A tool called process will be called incorrectly far more than one called convert_csv_to_json. Include a one-sentence description, concrete examples of when to use it versus similar tools, and explicit notes about what it does not do.

Step 7: Use few-shot examples. Including one or two examples of correct tool calls for your specific schemas in the system prompt dramatically improves small model reliability. The examples act as an in-context demonstration of the expected format and help override any prior the model has built from pretraining.

For agents that depend heavily on tool calling, the post on building AI agents that work covers the broader orchestration patterns, and the Hermes desktop app deep dive is a detailed case study of one such agent in production. For output validation and safety guardrails that sit alongside the tool-call pipeline, see the guardrails for production LLM applications post.

Model Comparison: Who Does Tool Calling Well in 2026

The open-model landscape moves fast enough that any precise ranking is stale within a few months. With that caveat, the following table reflects the state as of mid-2026 and focuses specifically on tool-calling capability rather than general benchmark scores.

If you are running quantized GGUF models via Ollama or llama.cpp — covered in detail in the quantization deep dive and the Jan local LLM runner post — note that tool-call reliability generally holds through Q4_K_M quantization but can degrade noticeably at Q2 or Q3 levels, where the model loses precision on the instruction-following signal that governs whether to call a tool at all.

For recent self-hosting setup guides on Llama 4 and the Qwen3 model family, see the Llama 4 and Gemma 4 self-hosting post.

MCP as the Wiring Layer

The Model Context Protocol, introduced by Anthropic in late 2024 and donated to the Agentic AI Foundation (a Linux Foundation directed fund) in December 2025, has established itself as the standard integration layer above function calling. MCP and function calling are not competitors: function calling is the model API mechanism; MCP is the protocol that standardizes how tool definitions, authentication, and results flow between a host application and tool servers. A compliant MCP server exposes a tools/list endpoint that returns schema-conformant tool definitions, and a tools/call endpoint that executes them. The host translates MCP tool definitions into whatever native format the underlying model expects.

The practical benefit for local deployments is discoverability and reuse. A community ecosystem of over 200 MCP servers now covers GitHub, PostgreSQL, Slack, Docker, Kubernetes, filesystem operations, and more. Any local LLM host that implements the MCP client protocol can connect to these servers without per-integration glue code. Transport options include stdio (for local in-process servers) and HTTP+Streamable HTTP (the current recommended transport in 2026) for remote servers.

For local LLM stacks, MCP matters because it externalizes the tool schema management problem: instead of manually constructing and maintaining tool JSON schemas in your application code, the MCP server owns the schema and the host fetches it dynamically. This reduces schema drift errors considerably, because the model always receives the current schema rather than a stale copy.

The deep-dive on MCP architecture, server implementations, and client patterns is at the MCP deep dive.

Measuring Reliability

You cannot improve what you cannot measure, and ad-hoc manual testing of tool calling is almost useless — models are inconsistent enough that ten successful calls may precede a systematic failure on edge cases. Building a minimal eval harness is worth the investment.

A useful tool-call eval harness has four measurement layers, patterned after frameworks like the Berkeley Function Calling Leaderboard (BFCL):

Layer 1 — Valid-call rate. What fraction of responses contain a structurally valid tool call (parseable JSON, registered tool name, required fields present)? This is the baseline floor; constrained decoding should push this close to 100%.

Layer 2 — Correct-tool rate. Among valid calls, what fraction selected the right tool given the user’s request? Measure this with labeled test cases where the expected tool is known.

Layer 3 — Argument accuracy. Among correct-tool calls, do the argument values match expected ground truth? This requires either exact-match scoring (for deterministic fields like enum values or numeric IDs) or semantic scoring (embedding similarity or LLM-as-judge) for free-text arguments.

Layer 4 — False-positive and false-negative rates. How often does the model call a tool when it should not (false positive), and how often does it not call one when it should (false negative)? Include “no tool needed” cases in your test set.

A minimal harness structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

import json
from typing import Any

def score_tool_call(
    model_output: str,
    expected_tool: str | None,          # None = no call expected
    expected_args: dict[str, Any] | None,
    tool_registry: dict,
    arg_match_fn=None,
) -> dict:
    result = {"valid": False, "correct_tool": False, "arg_score": 0.0, "spurious": False}
    try:
        call = json.loads(model_output)
    except json.JSONDecodeError:
        return result
    result["valid"] = True

    name = call.get("name")
    if expected_tool is None:
        result["spurious"] = name is not None
        return result

    if name not in tool_registry:
        return result
    result["correct_tool"] = (name == expected_tool)

    if result["correct_tool"] and expected_args is not None and arg_match_fn:
        result["arg_score"] = arg_match_fn(call.get("arguments", {}), expected_args)

    return result

Run this across a labeled test set of at least 40–60 cases covering single-tool calls, multi-tool selections, and no-call scenarios. Separate a held-out split from your development split — models (and prompts) can be inadvertently overfit to a small dev set.

The BFCL also evaluates an “irrelevance detection” bucket specifically measuring false-positive suppression, which is worth incorporating. A 2025 NESTFUL study found that even GPT-4o achieves only ~28% accuracy on nested tool-call sequences, highlighting how much harder multi-step chaining is than single-call accuracy.

For a broader treatment of LLM evaluation methodology, the post on LLM evals and testing AI applications covers harness design, dataset curation, and metrics selection.

Honest Trade-Offs

Constrained decoding is not a free lunch, and a complete picture requires acknowledging what it costs.

Answer quality. Forcing the model into a rigid JSON structure removes its ability to use chain-of-thought reasoning tokens during the constrained portion of the generation. Several benchmarks have found that requiring JSON output reduced accuracy on reasoning tasks by 15–27 percentage points compared to natural language generation. One mitigation is a “think then call” pattern: allow unconstrained generation for a reasoning scratchpad, then switch to constrained mode for the final tool-call token sequence. Qwen3’s thinking mode supports this natively.

Latency. Grammar-based sampling adds overhead, though XGrammar has reduced this to near-zero for well-structured JSON grammars (under 40 microseconds per token). GBNF grammars in llama.cpp add more overhead, particularly for complex recursive grammars. The retry loop adds further latency when it triggers; budgeting for one retry on ~5–10% of calls is realistic.

Grammar brittleness. GBNF grammars and JSON Schema constraints are sensitive to schema changes. If you add a new required field to a tool definition and forget to update the grammar, constrained decoding will silently prevent the model from ever outputting that field, producing calls that fail validation. MCP’s dynamic schema fetching helps here, but only if the constrained decoding backend re-derives its grammar on each request.

Semantic correctness is not guaranteed. This bears repeating: constrained decoding guarantees syntactic validity, not correctness. A model can produce {"name": "delete_file", "arguments": {"path": "/etc/passwd"}} — perfectly valid JSON, correctly named tool, correct schema — and still be catastrophically wrong. Structural validation is necessary but not sufficient; semantic and safety checks must sit outside the grammar layer. The guardrails patterns discussed in the production LLM guardrails post apply directly here.

Small models have a hard floor. Constrained decoding can guarantee structure, but it cannot give a 3B model the situational judgment to know when to call a tool versus when to answer from memory, or which of five available tools is most appropriate for an ambiguous request. Below roughly 7B parameters the false-negative and incorrect-tool rates remain high even with all the other techniques applied. For reliability-critical applications, parameter count is not negotiable.

Verdict

Tool calling reliability on local models is not one problem — it is a stack of problems that each require their own fix. Using the right chat template stops the most common catastrophic failures. Constrained decoding eliminates structural invalidity. A validate-retry loop catches residual errors. Thoughtful tool-surface design and few-shot examples improve semantic accuracy. Measurement via a real eval harness tells you where you actually stand. No single technique is sufficient alone, but applied together they make local tool calling competitive with cloud APIs for many practical use cases — particularly on Qwen3 and Llama 4 class models running on consumer or prosumer hardware. The hard floor is semantic correctness: grammars can enforce shape, but they cannot enforce meaning, and that gap is where careful prompt engineering, smaller tool surfaces, and LLM-as-judge validation earn their keep.

Sources

• XGrammar: Flexible and Efficient Structured Generation Engine for LLMs (arxiv.org)
• Structured Outputs — vLLM Documentation
• Tool Calling — vLLM Documentation
• llama.cpp Grammar and Structured Output (GitHub)
• Function Calling — Qwen Documentation
• Tool Calling — Ollama Documentation
• Model Context Protocol Specification 2025-11-25 (modelcontextprotocol.io)
• LLM Agent Evaluation Metrics in 2026 — Confident AI
• How Tool Chaining Fails in Production LLM Agents and How to Fix It (futureagi.substack.com)