Why agents fail — an honest preview
Lesson 5 of 5 in What Is an Agent? The Loop That Changes Everything.
You now know what an agent is. Before the rest of this academy teaches you how to build one, you deserve the uncomfortable part: the same property that makes agents valuable makes them fail in ways no previous software did. Two root causes drive nearly everything.
Root cause one: non-determinism. Run the same agent on the same task twice and you can get different tool sequences, different intermediate results, different outcomes. The model samples from a probability distribution; temperature zero narrows but does not eliminate the variance, and any change in context — a timestamp, a reordered file listing — can tip a decision. Your instincts from deterministic software (“it worked in the demo, ship it”) are actively misleading here.
Root cause two: compounding errors. A chatbot makes one model call per answer; an agent makes a chain of decisions, and each one builds on the last. Chain twenty 95%-reliable steps and the whole run succeeds about 36% of the time (0.95²⁰ ≈ 0.36). Worse than the arithmetic: a wrong step doesn’t just fail, it feeds a wrong observation into every later decision. Step three misreads a config file; steps four through twenty diligently solve the wrong problem.
| Instinct from traditional software | Why it breaks on agents | What replaces it | Where you learn it |
|---|---|---|---|
“It passed the test, so it works” | One passing run is one sample from a distribution that also contains failures. | Run the task many times; ship on pass rates against a golden dataset, not pass/fail. | → the evals domain |
“Reproduce the bug, then fix it” | The failing path may not recur on demand — the model may simply not take it again. | Capture full traces of every run so the failure is recorded even when it isn’t reproducible. | → the agentops domain |
“Add a rule to prevent it” | Prompt rules are probabilistic; the model can be argued out of them by injected text it reads while working. | Enforce invariants in the runtime: least privilege tools, caps, gates. | → the security domain |
“More steps of automation = more value” | Each added step multiplies the failure rate and widens the blast radius of early mistakes. | Shorter loops, checkpoints, and verification steps that catch errors before they compound. | → the architectures domain |
Myth: “The next model generation will fix this”
Better models raise per-step reliability — say 95% to 99%. Chain twenty steps at 99% and you still fail one run in six (0.99²⁰ ≈ 0.82). Capability improvements shrink the problem; only engineering — verification, checkpoints, containment — changes its shape. Teams that wait for the model to save them ship demos, not products.
Myth: “We tested it for a week and saw no failures”
A 4%-failure agent shows you a clean week easily, then fails four times in the first hundred production runs — on other people’s data, with other people’s edge cases. Absence of observed failure in a handful of runs is weak evidence about a distribution. This is why serious teams measure pass rates over hundreds of runs before raising the autonomy dial.
Myth: “Non-determinism means agents are unusable for serious work”
The opposite lesson. Aviation runs on unreliable components arranged into reliable systems — redundancy, checklists, monitoring. Agents follow the same playbook: verification steps, bounded blast radius, human gates where it counts, and honest measurement. Non-determinism is a design constraint, not a verdict — the rest of this academy is the playbook.
Interactive flashcard deck.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.