Timeouts at every layer
Lesson 1 of 5 in Reliability Plumbing: Timeouts, Retries, Idempotency, Breakers.
Every hung agent you will ever debug is a missing timeout. Not a bad prompt, not a clever failure mode — a call that was allowed to wait forever, in a system where something upstream had already given up.
An agent run is a stack of nested clocks. The model call has one. Each tool call has one. The whole run has one. And the client — a browser, a queue worker, a load balancer — has its own, which you do not control and which is usually shorter than you think.
The rule that makes the stack coherent: inner budgets must fit inside outer budgets, and the outer budget must fit inside the caller’s patience. A 120-second run deadline that permits six 60-second tool calls is not a deadline, it is a decoration.
Key terms: timeout, deadline propagation, stopping condition, agent loop, idempotency, graceful degradation
| Clock | What it guards against | What expiry should do | Symptom when it is missing |
|---|---|---|---|
Per model call — one inference request | A provider hanging mid-generation, or a long chain-of-thought response that will never fit the run budget anyway. | Retry is usually safe — inference has no side effects. Retry with backoff, against the remaining run budget, and count the attempt. | Runs that sit at 100% "thinking" until the client disconnects. Cost accrues for tokens nobody will read. |
Per tool call — one function execution | A slow database, a third-party API with no SLA, a shell command waiting on stdin that will never come. | It depends on side effects. Read-only: retry or skip. Writes: you are now in unknown state — see the warning below. | One flaky dependency freezes the whole loop. The trace shows a span that simply never closes. |
Per run — the whole loop, wall-clock | The loop that will not terminate: re-verifying, re-planning, re-reading the same file forty times. | Abort — but abort deliberately: flush a checkpoint, emit a partial result, hand off to the fallback path from lesson five. | Runaway spend and a stopping condition that is really "whatever kills the process first". |
Per step / turn budget — a count, not a clock | Fast loops that burn 200 cheap iterations well inside the wall-clock deadline. | Abort or degrade. A step budget is the cheapest guard against non-deterministic thrash. | The run finishes inside its deadline and still costs forty times the estimate. |
Timeouts are only useful if the number travels. Deadline propagation means the run’s absolute deadline — a timestamp, not a duration — is passed down into every model call and every tool call, and each layer takes the minimum of its own limit and the time actually remaining.
Without propagation you get the classic absurdity: a run with four seconds left starts a tool call configured for thirty. The tool call cannot possibly help, it cannot be cancelled cleanly, and its result will arrive to a run that no longer exists. Compute the remaining budget at every call site and refuse to start work you cannot finish.
A clock just fired. Retry, skip, or abort?
Interactive decision tree — outcomes:
- Retry with backoff
Inference and reads are safe to repeat. Back off with jitter, decrement the retry budget, and recompute the remaining deadline before each attempt — a retry that outlives the run is worse than no retry.
- Skip the step and tell the model
Return a typed observation —
tool_unavailable: search timed out, no result— so the model can route around the gap instead of guessing. Silence gets filled with invention; an explicit failure gets handled. - Reconcile, then decide
Re-present the idempotency key (or query the resource) to learn whether the action happened. Then either replay the recorded result or make exactly one fresh attempt. This is the only safe retry for a write, and it is why lesson three comes before lesson two in importance.
- Stop. Do not retry.
An unkeyed, unqueryable write in unknown state is not a retry decision, it is an incident. Fail the step, record what was attempted, and escalate to the human queue. Then go add an idempotency key to that tool so the next timeout is boring.
- Abort cleanly
Write a checkpoint, emit whatever partial output is genuinely useful, and return an honest error. A deliberate abort at the deadline is a feature; a process killed by a supervisor is a mystery.
- Abort and hand off
Partial side effects mean somebody has to finish or reverse them. Checkpoint the state, summarise what was done and what was not, and hand the run to the human queue — the deterministic fallback from lesson five.
Tool: Trace Debugger — Timeouts are easiest to diagnose from the span tree — a child span that never closes, or a parent that closed while its child was still running. Practise reading those shapes in the Trace Debugger.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.