Reliability Plumbing: Timeouts, Retries, Idempotency, Breakers

The unglamorous machinery that keeps agents alive: nested timeouts at every layer, retries with judgment instead of reflex, idempotency keys so the model can ask twice and your runtime executes once, circuit breakers and backpressure for flaky tools and exhausted quotas, and deterministic fallbacks that degrade to less autonomy instead of improvising.

Content current as of 2026-09.

Lessons

  1. Timeouts at every layer
  2. Retries with judgment
  3. Idempotency keys: ask twice, execute once
  4. Circuit breakers, rate limits, and backpressure
  5. Deterministic fallbacks: degrade, do not improvise