Evaluation & Observability

Traces, evals, LLM judges, and knowing whether your agent actually works

• Trace Anatomy: Reading an Agent Run Like a Professional — The trace is the only record you get of a run that will never repeat. Learn what a span carries, what to record on model calls, tool calls and decisions, where the OpenTelemetry GenAI conventions actually stand — then diagnose three traces that look fine and are not. (5 lessons, 40 min)
• Eval Fundamentals: You Cannot Improve What You Cannot Measure — The centrepiece discipline of agent engineering: turning “it worked when I tried it” into a pass rate on a golden dataset. Build the dataset, separate outcome evals from trajectory evals, climb the grading ladder from deterministic checks to judges, and turn every incident into a permanent test case. (6 lessons, 45 min)
• LLM-as-Judge: An Instrument You Calibrate, Not an Oracle You Trust — A model grading a model is a measuring instrument with known biases and a drift problem. Learn when a judge is the right tool and when it is lazy engineering, how to write rubrics a judge can actually apply, the five biases and their mitigations, and the calibration protocol that turns a judge score into evidence. (5 lessons, 45 min)
• Regression Suites in CI: Evals That Run On You — The discipline that turns a pile of eval cases into a gate nobody can walk around: a deterministic-first pyramid, per-commit and nightly tiers, pinned model versions, honest thresholds over N runs, and the offline-to-online bridge that catches what your golden dataset never will. (5 lessons, 40 min)
• Observability in Production: Watching Agents at Scale — You cannot read ten thousand traces a day. Learn the three signals a production agent must emit, how to alert on distributions instead of single runs, what the observability platforms actually do as of September 2026, how online evaluation catches drift you never deployed — and the five charts an agent team really looks at. (5 lessons, 45 min)
• Cost and Latency Budgets You Can Defend — Where an agent run’s tokens actually go (the whole transcript resends every turn, so cost grows quadratically-ish with turn count), which levers really cut cost and latency, how to enforce a budget in the runtime instead of wishing for one in the prompt, and how to bake off cost, latency, and quality on the same golden set. (5 lessons, 40 min)