Field Notes

Practitioner case studies (composite teaching cases, labeled as such) and working patterns.

• The support bot that read its own email — Three reasonable tickets over four months assembled the lethal trifecta inside a support agent. Nothing alerted. A customer noticed on day nine.
• The MCP server nobody reviewed — A convenient third-party MCP server auto-updated overnight, its tool descriptions started giving the agent orders, and every trace looked clean for six days. What went wrong, how it was found, and the review discipline that replaced “just install it”.
• Five Agents, One Deadlock — A team split a document pipeline into five agents shaped like their org chart. It demoed perfectly, then produced telephone-game handoffs, two agents each waiting on the other, contradictory records, and a 6x token bill — here is how they found it and what they rebuilt.
• The weekend that cost $40,000 — A research agent shipped on a Friday with no per-run budget, an unbounded search tool, and a retry counter that lived in worker memory. By Monday it had burned about 12.8 billion input tokens — and every dashboard was green.
• A governed multi-agent system, end to end — One realistic system — a supervisor and two workers that research and draft a sourced brief — built with every control wired in: per-agent identity, scoped tools, an egress allowlist, one approval gate, traces, evals, budgets, a rollout ladder, and a kill switch you have actually pulled.
• Your First Eval Suite, in a Week — You have an agent in production and no evals. Here is the five-day recipe: harvest twenty real cases from traces, write the checks a machine can settle, add outcome grading, add one trajectory check and exactly one judge, then wire it into CI with thresholds you can defend on a Friday afternoon.
• The Tool Hardening Pass — A repeatable two-day audit you run over an existing agent’s toolset before it goes anywhere near production: inventory what every tool can reach, classify by reversibility and blast radius, tighten the schemas, rewrite the descriptions to say when not to use the tool, make errors readable by a model, then decide gate or notify or free — one tool at a time.
• Choosing a platform without regretting it — Four weeks to name a platform — AWS AgentCore, Microsoft Foundry Agent Service, Google’s Agent Platform (formerly Vertex AI), or self-hosting your framework. The recipe: five questions that actually decide it, a two-week bake-off on your own golden set, a ledger of what ports and what anchors you, and the exit plan you write on day one.