Prompt and tool unit tests: the evals that need no model

Lesson 4 of 5 in Regression Suites in CI: Evals That Run On You.

The most under-built layer of agent testing is the layer that involves no model at all.

An agent is mostly ordinary software wearing a probabilistic hat: tools are functions, schemas are contracts, parsers are parsers, retry logic is retry logic. All of it is deterministically testable. Teams skip it because the interesting failures feel model-shaped — and then spend token budget and minutes of CI discovering that a date parser mishandles month boundaries.

Split the surface in two. Tools you test as functions: call them directly, offline, with no model in the loop. Prompts you test with canned contexts: freeze the input the model would have seen, run one pinned model call, and assert on the shape of what comes back.

Tool tests (no model)

Test the tool the way you would test any function you own — happy path, edge cases, errors, and the shape of the error text.

test('lookup_order returns a not-found result, not an exception')
test('refund rejects amount > cap with a message the model can act on')
test('search_kb truncates to 8 KB and marks the result as truncated')
test('every tool in the manifest validates against the JSON Schema')
test('tool names are unique and stable — renames break saved prompts')

Two assertions here that people miss. First: error text is model input. A tool that raises a stack trace teaches the model nothing; one that returns “amount exceeds the 100 EUR cap for this account; escalate to a human” lets the model recover — so the wording is worth an assertion. Second: truncation must be labelled, because an agent that cannot tell “no match” from “I only saw the first 8 KB” will confidently assert the wrong thing.

Prompt contract tests

Freeze a context and assert on the structure of the response. One pinned model call, no tools executed, no side effects — cheap enough to keep on the commit path.

given  context = fixtures/ticket_0142.json   # frozen: prompt + history + tool defs
when   one model call, model=lm-large-2026-06, temperature=0
then   output parses as JSON
       and required fields present: status, action_taken, confidence
       and status in {resolved, routed, escalated}
       and the first tool call is search_kb        # not answer-from-memory
       and no call to refund                       # amount is over the cap

You are testing the contract, not the prose: does the model, given exactly this context, produce a well-formed decision of the right kind? These catch prompt edits that break the output contract or shift tool selection, and they localise the failure to a single turn instead of a whole run.

Replay tests (recorded)

Take recorded model outputs from real runs and re-run your own code over them. Zero model calls, zero cost, milliseconds.

replay 300 recorded assistant turns through:
  - the tool-call parser        (malformed arguments, unknown tool names)
  - the schema validator        (missing fields, wrong enum values)
  - the truncation handler      (finish_reason = length)
  - the redaction filter        (no PII leaves the boundary)

Replay is where you keep the ugly real-world outputs your parser once choked on. Every parsing incident should end with its transcript in this corpus — a permanent, free regression test. It cannot tell you whether the model behaves well; it tells you your code survives what the model actually does.

Four cheap layers — what each one can and cannot see
Layer	Model calls?	Catches	Blind to
Tool unit test	No	Logic bugs, bad error text, unlabelled truncation, boundary cases.	Whether the model ever chooses this tool, or reads its result correctly.
Tool schema contract test	No	Renames, type drift, required-field changes, duplicate names, oversized manifests.	Whether the description still guides selection — that needs a model.
Prompt contract test (canned context)	One, pinned	Broken output contracts, wrong first tool choice, forbidden calls at a single decision point.	Multi-turn drift, recovery after a tool error, anything that emerges over a long run.
Replay test (recorded outputs)	No	Parser and validator regressions against outputs the model really produced.	Any change in model behaviour — the recordings are frozen in the past.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.