The weekend that cost $40,000
A research agent shipped on a Friday with no per-run budget, an unbounded search tool, and a retry counter that lived in worker memory. By Monday it had burned about 12.8 billion input tokens — and every dashboard was green.
A composite teaching case: realistic fiction assembled from well-documented public patterns — not a real engagement.
Tessellate sells competitor briefs to sales teams. Their agent — internally, Beacon — takes a company name, searches the open web, reads what it finds, and writes a two-page brief with cited sources. One job per lead. About sixty briefs a day, quietly, for four months.
On a Friday afternoon in March, three changes shipped together. None of them looked like a cost change.
One: the turn cap came off. Beacon had been capped at twelve turns, and the team had a stack of complaints about truncated briefs on obscure companies. So the cap was replaced with a softer stopping condition — the model decides when it has enough sources and says so. Two: search_web was upgraded. It used to return ten snippets; now it returned the full text of the top four pages, because full text produced better briefs in the eval set. Three: an outer retry was added, so that a run which failed to produce a schema-valid brief got requeued instead of silently dropping the lead. The attempt counter lived in a dictionary on the worker process.
Dana, the engineer who shipped it, watched twenty runs go green in staging and went home at 17:20. The change was good. Every individual change was, in fact, good.
At 02:00 on Saturday a scheduled CRM sync backfilled 1,900 dormant leads from an acquired account. Each one enqueued a brief job. The invoice for the next sixty-two hours came to $39,700 against a normal weekend spend of about sixty dollars.
Key terms: token budget, retry budget, loop depth, kill switch, prompt caching, cost per successful run
One job, as it actually ran that weekend
- Saturday 02:00 — 1,900 leads backfilled
A routine CRM sync from an acquired account. Nobody thought of it as an agent-triggering event, because nobody had drawn the line from “rows arrive” to “model calls happen”.
- One brief job enqueued per lead
No per-hour admission control. The queue accepted 1,900 jobs in under a minute, and the worker pool autoscaled on queue depth from 4 to 96 — a sensible reflex for throughput and precisely the wrong one for spend.
- Run starts — 6k-token context, no turn cap
System prompt, tool schemas, and the task. The system prompt also carried the current timestamp to the second, which quietly defeated any cross-run prompt-cache reuse.
- search_web returns 4 full pages (~15k tokens)
Full page text, untruncated. For a well-covered company this is signal. For a dormant lead with a dead website it is 15k tokens of navigation menus, cookie banners, and unrelated press releases.
- Model: enough sources to write the brief?
The only stopping condition left. Fed mostly noise, the model reasonably concluded it needed to keep looking — and every extra look made the next turn more expensive.
- Context passes the model window — provider error
Around turn 28 the request stopped fitting. The run died with a hard provider error after having already been billed for every earlier turn.
- Validator: publishedAt on every source?
Friday also tightened the brief schema: every citation now required a publication date. Most scraped pages do not expose one. This rejection was deterministic — retrying it could never succeed.
- Job requeued — attempt counter in worker memory
The counter capped attempts at three. It lived in a process-local dictionary, so any worker restart reset it to zero for every job that worker was tracking.
- Attempts under the cap of 3?
Workers restarted twice under memory pressure on Saturday. 2,040 jobs produced 3,380 billed runs — the cap never bound on the jobs that mattered.
- Job abandoned — no alert
A job that exhausted its attempts vanished. There was no dead-letter queue and no counter, so “we are failing every niche lead” was invisible.
- Brief delivered
About 38% of jobs — the well-covered companies — finished in six or seven turns and looked exactly like a healthy weekend.
Three multipliers stacked, and each one is worth understanding on its own.
The resend tax. A model call is stateless. Every turn of the loop resends the entire transcript so far — system prompt, tool schemas, every previous tool result — as fresh input tokens. So a run that adds a constant chunk of context per turn does not cost linearly in turns; it costs roughly quadratically. Add 15k tokens per turn and by turn 22 you have billed about 3.6 million input tokens for a single brief. This is the arithmetic behind the resend tax in Cost and Latency Budgets You Can Defend, and it is the single most important number an agent builder can hold in their head. Prompt caching blunts it when the prefix is stable — but Beacon put a live timestamp in its system prompt, so no two runs shared a cacheable prefix, and long tool waits aged out the within-run cache anyway. Caching mechanics and TTLs differ by provider and change often; check the current docs rather than assuming yesterday’s behaviour.
The retry multiplier. Retries are cheap when failures are transient and expensive when they are deterministic. The publishedAt rejection was deterministic: the same page will never grow a publication date on attempt two. So every retry paid full price — a fresh 6k-token context, another twenty-plus turns of resent transcript — to reach the identical rejection. Failure classification is exactly what Reliability Plumbing: Timeouts, Retries, Idempotency, Breakers is for, and this run had none of it: one code path, retried blindly.
No ceiling anywhere. Removing the turn cap moved the stopping condition from the runtime into the model’s judgment. That is a category error, and it is the same one you meet in the containment ladder in Tool Scoping and Least Privilege: Making the Dangerous Thing Impossible — a limit the model can talk itself out of is not a limit. There was no per-run token budget, no cumulative token ledger, no cost ceiling on the job, no concurrency cap on the worker pool, and no per-hour admission control on the queue. The only real ceiling in the system was the model’s context window, and the system found it about 900 times.
| Run length | Context on the last turn | Cumulative billed input | Illustrative cost |
|---|---|---|---|
Before Friday — snippet search, 12-turn cap | ~50k tokens | ~126k tokens | ~$0.45 per brief |
6 turns — a well-covered company | 81k tokens | 261k tokens | ~$0.85 |
12 turns — the old cap, new search tool | 171k tokens | 1.06M tokens | ~$3.30 |
22 turns — the weekend median | 321k tokens | 3.60M tokens | ~$11.00 |
28+ turns — dies on the context window, then retries | over the window | 6.7M tokens | ~$20.50, for nothing |
The weekend — 2,040 jobs, 3,380 billed runs | — | ~12.8 billion tokens | $39,700 on the invoice |
Alarm 1 — the monthly spend alert: right idea, wrong denominator
What we had. An alert at 80% of the platform-wide monthly commitment, which was large enough to absorb a lot of surprise. Why it stayed green. Beacon’s $2,500 monthly budget was a line in a spreadsheet, not a threshold anywhere; the weekend consumed most of the platform commitment before the percentage tripped, and it tripped at 23:00 on Sunday.
What replaced it. A per-agent spend rate — dollars per hour — with per-agent monthly ceilings enforced at the account level as a backstop. Budgets that live only in spreadsheets are not budgets.
Alarm 2 — queue depth: healthy, because the work was getting done
What we had. A depth alarm on the job queue at 500 pending jobs for ten minutes. Why it stayed green. Twenty-four workers kept up. Depth spiked at 02:00, drained by 02:40, then oscillated cheerfully all weekend as requeues arrived at roughly the rate the pool cleared them. The queue was working perfectly. That was the problem.
What replaced it. Alarms on throughput composition, not depth: runs started per hour, and the ratio of first attempts to retries. A queue that is 60% retries is a queue in trouble no matter how shallow it is.
Alarm 3 — error rate: flat, because the failures were business outcomes
What we had. A 5xx and exception-rate alarm on the worker service. Why it stayed green. A schema-invalid brief was not an exception — it was an expected outcome, logged at info level, handled by the shiny new requeue path. The code did exactly what it was told, so nothing threw.
What replaced it. Outcome accounting as a first-class metric: every run ends in exactly one of delivered, budget-exhausted, validation-rejected, or error, and the mix is alarmed against its trailing baseline. The most expensive failures rarely raise exceptions.
Alarm 4 — p95 latency: it fired, into a muted channel
What we had. A latency alarm on run duration, which genuinely tripled. It posted to a Slack channel. Why nobody saw it. That channel had been muted by half the team in January after a flaky-test bot spent three weeks spamming it. Classic alert fatigue: the signal existed, the delivery path was dead, and no one had audited the path in eight months.
What replaced it. Anything that means “stop and look now” pages the on-call rotation through the pager, and every alarm route is tested in the monthly drill. If an alert cannot wake someone, it is a log line with delusions of grandeur.
Alarm 5 — the provider console: the number that arrives too late
What we had. The provider’s billing dashboard, which the team checked on Mondays. Why it did not help. Vendor cost reporting lags — often by hours, and the granularity and freshness vary by provider and change over time, so verify yours rather than trusting a number you remember. Even a perfect billing feed is the wrong instrument: it tells you what you already spent.
What replaced it. Self-computed spend from the trace pipeline. Every span already carries input and output token counts; multiply by a price table you own and you have a dollar figure within seconds of the call, which is what Monitoring and Incident Response: When the Agent Is the Incident means by making cost an operational metric instead of a finance one.
The 02:40 page nobody got — walk it as if you had
Interactive decision tree — outcomes:
- You lose thirty minutes to a stale number
Vendor billing views lag and aggregate. At 41× and climbing, thirty minutes is roughly $2,000 of illustrative spend. Your own traces already hold token counts per call — treat them as the authoritative real-time cost signal and the invoice as reconciliation.
- Worst available action
Restarting is exactly what reset the in-memory attempt counters in the first place. You would have re-armed every job for another three attempts and destroyed the in-flight evidence. Never restart a runaway before you have measured it.
- Right diagnosis, wrong hour
Shipping a schema change at 03:00 to a system you do not yet understand is how a cost incident becomes a correctness incident. Stop the bleeding first; the fix is a daylight decision with a review. Containment and repair are separate steps.
- Six times slower, still unbounded
Throttling divides the rate; it does not cap the total. At 8 workers you would still have spent thousands by Monday, and you would have made the incident quieter and longer. Concurrency ceilings are a good steady-state control and a bad stop button.
- The queue is the other half of the switch
A kill switch that only stops the agent leaves a queue that redelivers the moment you re-enable — or worse, that a different consumer picks up. Rehearse the stop as one action across the whole path: admission, queue, workers, and the loop itself.
- This is the whole playbook
Confirm from your own telemetry, stop the path end to end, contain before you repair, then re-enter through a narrow gate with the new limits enforced and the cost rate on screen. Same staged-exposure discipline as Rollout and Kill Switches: Controlled Exposure — a runaway is a rollout in reverse.
- You fixed one of three multipliers
The turn cap bounds one run. It does nothing about 1,400 queued jobs, deterministic retries, or a backfill that can arrive again next Saturday. Ask what the worst plausible weekend now costs, in dollars, and enforce that number before you re-enable.
Nobody got that page. The reconstruction later showed that roughly nine tenths of the bill was spent between 02:00 and noon on Saturday, which is the part that stung: a page at 02:40 and a stop by 03:00 would have cost about $3,000 instead of $39,700. The failure was not that the agent misbehaved — given its inputs, Beacon behaved sensibly at every single turn. The failure was that a system with no ceiling was left running unwatched for sixty-two hours.
Dana found it at 08:50 on Monday, from a Slack message asking why the finance dashboard looked strange. She read three traces end to end — the discipline in Trace Anatomy: Reading an Agent Run Like a Professional — and had the mechanism within twenty minutes: turn counts in the thirties, the same job ids repeating, and publishedAt missing on nearly every source of every rejected brief.
The postmortem produced one sentence the team now applies to every agent they ship: any quantity that can grow without bound during a run must have a number attached to it, and that number must be enforced by code the model cannot reach. Turns, tokens, tool calls, retries, concurrent runs, dollars. Six numbers. Write them down before launch, because the version of this incident you get without them is not a bug you can fix — it is the system working as designed, at speed, while everyone sleeps.
Here is what actually shipped in the two weeks after, and what each control cost. The right-hand column matters as much as the others: controls that are free are usually controls that do not bind.
| Control | Where it is enforced | What it cost us |
|---|---|---|
Per-run token ledger. Cumulative input+output tokens tracked per run; at 70% of budget the agent enters finish-up mode and must write the brief from what it has; at 100% the run aborts with a structured | Inside the loop, checked before every model call. Not in the system prompt. | Two engineer-days. About 3% of briefs now come back explicitly marked incomplete instead of silently perfect — Sales prefers this, once told. |
Turn cap of 18, restored, plus a loop-depth alarm when p99 turns per run exceeds 15 for fifteen minutes. | Loop counter in the runtime; alarm in the trace pipeline. | Half a day. Reporting had to learn a third outcome besides success and failure. |
Durable retry budget. Attempt count lives on the job record, not in the worker; validation failures are classified non-retryable; two attempts then dead-letter, with a daily count on the dashboard. | Queue message plus job table — survives restarts, redeploys, and autoscaling. | One engineer-day, and a standing morning chore: somebody reads the dead-letter queue. |
Bounded tool output. | Tool wrapper, enforced on the response before it ever reaches the context window. | Brief quality dropped four points on the internal rubric for two weeks until extraction was tuned. It came back up. |
Cost-rate alert. Dollars per hour per agent, computed from trace token counts against a price table the team owns; pages on-call at 3× the trailing 7-day same-hour median. | Trace pipeline to pager — not to a Slack channel. | Four false pages in the first month; the evaluation window was widened to fifteen minutes and it settled. |
Ceilings outside the agent. Max 8 concurrent runs per agent, admission control at 200 new jobs per hour, and a hard account-level spend cap for the weekend as the last backstop. | Worker pool config, queue consumer, provider account budget. | Backfills now take hours instead of minutes. Nobody has missed the minutes. |
A kill switch that stops the whole path — one flag, read by admission control, the queue consumer, and every worker between turns — rehearsed monthly with a stopwatch. | Runtime plus the drill calendar. Untested switches do not count. | First drill: 14 minutes to fully stop, because workers only re-read the flag between jobs. After the fix: 90 seconds. |
Cost per successful brief replaced cost per run on every dashboard. | A dashboard definition — the cheapest change on this list. | It made one team look three times worse than they believed they were. That was the point. |