Retries with judgment
Lesson 2 of 5 in Reliability Plumbing: Timeouts, Retries, Idempotency, Breakers.
Retry is the most dangerous one-line fix in distributed systems, and agents make it worse. A normal service retries a request. An agent retries a request that the model asked for, inside an agent loop that will happily ask again if the first answer looks wrong — so a single flaky dependency can produce three layers of duplication: the HTTP client retries, the runtime retries, and the model re-requests.
The discipline is simple to state and constantly violated: classify the error first, then choose a policy. "Retry on failure" is not a policy. It is a bet that every failure is transient, which is false often enough to cost money.
Transient infrastructure errors — retry, with backoff and jitter
Connection resets, 502/503/504, TLS handshake failures, DNS blips, a provider’s capacity wobble. These are the errors retries were invented for: the same call a second later has a genuinely different chance of working.
Use exponential backoff with full jitter — random delay in [0, base · 2^attempt], capped. Jitter matters more than the exponent: without it, every worker that failed at the same moment retries at the same moment, and your recovery attempt becomes a second outage (the thundering herd).
Rate limits and throttling — back off on the server’s terms, not yours
A 429 or a provider throttle is not a failure, it is the system telling you its capacity. Honour the retry hint the provider sends if there is one; otherwise back off and, crucially, reduce concurrency rather than just delaying the same load.
Retrying a rate limit at the same parallelism is how a soft limit becomes a hard one. This is a backpressure signal, and lesson four turns it into an admission-control decision.
Deterministic client errors — never retry
400 (malformed), 403 (missing scope), 404 (no such resource), 422 (business rule rejected the request). The same call will fail identically forever, so a retry loop here is pure latency and pure cost.
Do something better: turn the error into an observation the model can use. 400: field order_id is required fed back into the context window lets the model fix its own call. A silent retry teaches it nothing and burns the run budget.
Ambiguous timeouts — the class that needs a key, not a policy
Timeouts, connection resets after the request was sent, and worker kills mid-call. Transient-looking, but the write may already have landed. Retrying is safe only if the call carries an idempotency key the downstream system honours.
If it does not, this class is not retryable at all — it is escalatable. That asymmetry is the whole argument of the next lesson.
Bad model output — retry at the model layer, and cap it
Malformed JSON arguments, a hallucinated tool name, a schema violation. The tool never ran, so there is nothing to make idempotent — the right move is to re-prompt with the validation error attached, which is a different retry budget from the tool retry budget.
Cap it hard (two or three attempts) and count it. A model that cannot produce valid arguments for a tool after three tries has a schema problem or a prompt problem, and looping is just paying to rediscover that. Constrained decoding removes most of this class at the source.
One more control that teams skip: a retry budget for the whole run, not per call site. Per-call retry limits multiply — three attempts each across twelve tool calls is thirty-six extra requests in the worst case, all inside a deadline sized for twelve.
Give the run a total retry allowance (say, five) and spend it globally. When it is gone, stop retrying and start degrading. The budget also gives you a metric worth alerting on: a rising retries per successful run is the earliest signal that a dependency is going bad, usually hours before error rates move.
Interactive sorting exercise: Ten failures from one week of agent runs. Sort each into the retry policy it deserves. Ask the two questions in order: will repeating plausibly succeed, and is repeating safe?
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.