Non-determinism compounds: why agent testing is statistical
Lesson 4 of 5 in LLM Mechanics for Agent Builders — No Math Required.
In a chatbot, non-determinism is a nuisance: the same question gets slightly different phrasings. In an agent, it is a structural force — because the agent loop feeds every output back in as the next input. A slightly different grep query on turn 2 returns different files; different files lead to a different fix on turn 5; by turn 12 two runs of the identical task are in different parts of the codebase doing different things. Runs do not vary around a fixed path. They diverge, the way weather forecasts diverge from a whisper of difference in the starting conditions.
So how do you test a system whose behaviour is a distribution? The way you test anything statistical: sample it. Run the same task 20, 50, 100 times. Define what “pass” means — the test suite goes green, the refund lands in the right account, an LLM judge scores the trace acceptable — and measure the pass rate. One successful run is a coin that came up heads once. “Works” becomes a number: passes 87 of 100 runs, up from 74 last release. Building those numbers — the task suites, the checks, the golden datasets, the regression tracking — is the entire evals discipline, and it gets its own domain in this academy.
| Model property | In a single call (chatbot) | Multiplied across an agent run |
|---|---|---|
Priced per token | One prompt, one reply — cost is easy to predict. | The growing transcript is resent every call: cost scales with turns × transcript length, roughly quadratic in run length. |
Bounded context window | Rarely binds — most questions fit with room to spare. | Tool results pile up until the window forces truncation or compaction — and the agent loses facts mid-task. |
Stateless between calls | Invisible — the client resends a short chat history. | The transcript is the agent’s only memory: anything not resent never happened, and “memory” must be engineered. |
Sampled output | Same question, slightly different wording. Mostly harmless. | Each output feeds the next input — small variations compound into divergent runs, so testing becomes pass rates, not assertions. |
Seconds of latency per call | One wait — feels like a fast website. | Twenty sequential calls plus tool execution: minutes per run, which decides where agents are even viable. |
Tool: Eval Suite Builder — Feel the statistics yourself: in the Eval Builder you’ll run one task many times and watch a pass rate emerge from individually plausible runs.
Key terms: non-determinism, pass rate, eval, LLM-as-judge, trace
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.