Circuit breakers, rate limits, and backpressure
Lesson 4 of 5 in Reliability Plumbing: Timeouts, Retries, Idempotency, Breakers.
A flaky tool does not fail one run. It poisons every run that touches it — because each affected run spends its full timeout waiting, then its retry budget re-waiting, then arrives at the model with nothing useful to say. Ten seconds of tool flakiness becomes ten minutes of agent latency and a queue that stops draining.
The fix is the circuit breaker, borrowed unchanged from ordinary service engineering (Michael Nygard’s Release It! is where most engineers first meet it): watch the failure rate for one dependency, and when it crosses a threshold, stop calling it and fail immediately. Failing fast in 5 milliseconds is strictly better than failing slow in 30 seconds, because the fast failure leaves the run enough budget to do something else.
Circuit breaker state machine, per dependency
- Closed — calls pass through, outcomes counted
Normal operation. The breaker is a counter over a sliding window: failures, timeouts, and slow calls all count as failures.
- Failure rate over the window exceeds the threshold?
Rate over a window, not a raw count — otherwise a busy tool trips constantly and a rarely-used one never does. Require a minimum call volume before the rate means anything.
- Open — no calls leave the process
The dependency gets total relief, which is often what it needs to recover. Your runs get instant, cheap failures.
- Return a typed tool-unavailable observation
This is the agent-specific half of the pattern. The model must learn the tool is down in a form it can act on — see the callout below.
- Cool-down elapsed?
Long enough for a restart or a failover to complete; short enough that recovery is noticed promptly.
- Half-open — allow one probe call
Exactly one, not a flood. The whole point is to test the water without re-creating the load that broke it.
- Probe succeeded?
One success closes the breaker; one failure re-opens it and restarts the cool-down, usually with a longer interval.
- Closed again — counters reset
Breakers must be per dependency, and agents have two kinds. A per-tool breaker isolates one bad integration so the other nineteen tools keep working. A per-model breaker covers the inference provider itself — and because every run needs the model, tripping it means shedding load at the front door rather than degrading gracefully mid-run.
Scope them carefully. One breaker for "all tools" trips on a single bad integration and takes the whole agent down; one breaker per endpoint of a multi-tenant API may never trip because failures are spread thin. The unit is usually one tool against one backing service, sometimes split per tenant when tenants can independently break.
Per-tool breaker
Signal: failure rate, timeout rate, and p99 latency for one tool against one backing service.
Trips on: a rate threshold over a sliding window, with a minimum call volume so a single failure out of two calls does not trip it.
Effect on the run: the tool disappears from the agent’s effective capability set. Ideally you also drop it from the tool list in the next model call — an unavailable tool the model cannot see is one it cannot waste turns on.
Watch for: partial degradation. A tool that returns wrong answers quickly never trips a failure-rate breaker. That is an eval problem, not a breaker problem.
Per-model breaker
Signal: inference errors, timeouts, and streaming stalls for one provider or deployment.
Trips on: the same rate logic — but the consequence is bigger, because no run can proceed without a model.
Effect on the run: this is where a secondary model earns its keep, if your evals cover it. Failing over to an unevaluated model is not reliability, it is an untested deployment during an incident.
Watch for: capacity, not correctness. A provider under strain often keeps answering while getting slower — so include a latency-based trip, or your breaker stays closed while every run misses its deadline.
Rate limiting and quota
Signal: 429s, throttle responses, and your own token/request accounting.
Response: reduce concurrency, not just delay. A throttled request retried at the same parallelism produces the same throttle plus extra load.
Design for exhaustion. Quota running out is a state, not an exception: a tenant whose monthly budget is gone at 09:00 on the 28th should hit a designed path — a clear message, the scripted fallback, or a queue for the next window — not a stack trace. Per-tenant budgets stop one heavy user from consuming everyone else’s capacity.
Watch for: multiple limits with different units (requests per minute versus tokens per minute). You can be far under one and pinned against the other.
Backpressure and queue depth
Signal: queue depth, oldest-message age, and the ratio of arrival rate to completion rate. Oldest-message age is the honest one: depth alone hides whether anything is moving.
Response: admission control — refuse or defer new work at the edge while in-flight runs finish. Agents make this urgent because a run holds resources for minutes, so a small arrival surge becomes an enormous backlog before any metric looks alarming.
Effect on the caller: shed load visibly. "We are at capacity, try in ten minutes" is a better product than a request that silently ages out 40 minutes later. Streaming clients need the same honesty about accepted-but-queued work — see the crosswalk to protocols/sessions-and-streaming.
Watch for: unbounded queues. An unbounded queue does not remove backpressure, it converts it into latency you cannot see until the deadline.
Tool: Agent Incident Tabletop — A flaky tool, a throttled provider, and a queue that stops draining — usually on the same afternoon. Run the tabletop and practise which lever you reach for first.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.