Streaming tool calls, and the validation ladder

Lesson 5 of 5 in Function Calling Deep Dive: The Wire Under Tool Use.

Everything so far assumed you received the whole assistant message at once. Real interfaces stream, and streaming a tool call is stranger than streaming prose: you are watching a JSON object being typed one fragment at a time, and for most of that time it is not valid JSON.

Two audiences care. Users, because a tool-use turn can take seconds and a blank screen reads as broken. And your loop, because the temptation to act on a half-arrived call is exactly the bug that ships to production and executes something nobody requested.

What actually arrives, in what order

A streamed assistant turn is a sequence of deltas. Any prose the model writes first arrives as token fragments. Then the tool call opens: the name usually lands early and whole, followed by the arguments accumulating character by character, followed by an event saying that call is finished — and finally a stop reason for the whole turn.

Each fragment is tagged so you can attribute it: an index, a block id, or both. That tagging is what makes multiple concurrent calls reassemblable.

Why UIs care: the name is a progress bar

The moment search_orders arrives you can render “Searching orders…” — before a single argument has finished streaming. That is a two-to-three second improvement in perceived latency for free, and it is why every good agent UI shows tool activity live rather than a spinner.

There is a second, quieter reason: showing the tool name and arguments as they arrive is oversight. A user who can see “about to call delete_branch” has a reason to hit stop. MCP’s guidance points the same way — clients SHOULD show tool inputs before calling, and there SHOULD always be a human in the loop able to deny an invocation.

Never treat partial arguments as truth

{"customer_id":"CUS-4011 is not a customer id — it is a prefix. Accumulate fragments into a buffer and parse once, when the stream tells you the call is complete.

Some UIs do run a tolerant partial-JSON parser mid-stream to preview arguments as they form. That is a display trick, and it must stay in the display layer: the executor takes one input, the assembled and validated call. If your renderer and your executor share a code path, a partial parse becomes a real action.

Assemble by id, not by arrival order

With parallel calls, fragments from different calls interleave in the stream. Keep a buffer keyed by index or block id, append each fragment to its own buffer, and dispatch only on the completion event — never on “the JSON happens to parse now”, which can be true halfway through a nested object.

The same discipline pays off in your traces: record the assembled call, not the fragments. A trace of 400 deltas is unreadable; a trace of one call with its arguments and its result is the thing you will actually debug.

Cancellation, and the property it gives you for free

If you dispatch only on completion, then a user who stops the stream mid-call has caused no side effects at all — the tool never ran. That is a genuinely valuable safety property, and it exists only because of where you put the dispatch.

Do the opposite — start work optimistically on a partially formed call to shave latency — and cancellation stops the text while your infrastructure carries on doing something the model had not finished asking for.

So the call is assembled. Now run it down the ladder, in this order, because each rung answers a different question and skipping one is a specific class of outage:

Parse — is this a structure at all? Validate — does the structure satisfy the declared schema? Authorize — is this caller allowed to make this exact call, on this resource, at this amount? Then execute — and even then, distinguish “the tool failed” from “the call was wrong”.

The rungs are not interchangeable. A schema-valid call can be forbidden. An authorized call can still be nonsense in the world. And the response you owe the model differs at every rung: a validation error it should retry, an authorization refusal it should not, an execution failure it may retry later.

A tool call just arrived. Walk the ladder.

Interactive decision tree — outcomes:

  • Return a specific error result and let the model retry

    Name the field, quote the constraint, show a legal value: “limit must be an integer between 1 and 20; you sent 40.” Generic “invalid input” gives the model nothing to correct and it will re-send the same call. Feed the error back as a result keyed to the call id — that is the channel the loop already has — and cap retries at two or three.

  • Stop retrying — the tool is the problem, not the attempt

    Three identical failures is a design signal, not bad luck: an ambiguous description, an over-loaded parameter, a schema the model cannot satisfy from what it knows. Break the loop, surface it to a human or a fallback path, and fix the contract. Retries are not cheap either — each one resends the whole transcript.

  • Refuse in the runtime, and tell the model it was refused

    Authorization is not validation: this call was legal and still must not happen. Refuse in code the model cannot argue with, return a plain “not permitted” result so the loop can adapt, and log it. Repeated refusals on sensitive tools are a signal worth alerting on — that pattern is what injected instructions look like from the inside.

  • Execute, then report the miss as a readable result

    Run it and say what happened in words the model can use: “no order matches ORD-2026-114873.” MCP models this as isError: true inside the result rather than a protocol failure, precisely so the model can self-correct. An empty result with no explanation is how agents end up telling users they have no orders.

  • Execute — and design the result like a prompt

    All four rungs passed. Now remember the result goes straight into the context window: keep it small, structured, and honest about truncation. What comes back is the next thing the model reasons over, and unbounded tool output is both a cost problem and, when the content is attacker-controlled, a poisoning surface.

Interactive flashcard deck.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.