Three shapes of a deployed agent

Lesson 1 of 5 in Deploying and Versioning Agents: Ship It Like Software.

An agent in a notebook is a demo. An agent in production is a service with a topology — a shape that determines its latency profile, what happens when it crashes, where its state lives, and which platform primitives you can even use.

Almost every deployed agent lands in one of three shapes:

Request–response (chat-shaped). A caller opens a request, the loop runs, the answer comes back on the same connection. Seconds to a couple of minutes. The client is waiting.
Queue–worker (task-shaped). A job is enqueued, a worker picks it up, the loop runs detached, the result is written somewhere the caller polls or gets notified about. Minutes to hours. Nobody is holding the line.
Durable long-running. The run outlives any single process: it checkpoints its state, can be evicted or crash, and resumes from the last checkpoint — possibly hours or days later, possibly after waiting on a human. This is durable execution applied to the agent loop.

The trap is picking a topology by accident. Teams prototype in the chat shape because that is what the SDK quickstart does, then push a 40-minute research task through the same HTTP endpoint and discover that their load balancer kills idle connections at 60 seconds, their retry-on-timeout fires the whole run a second time, and a deploy in the middle of the afternoon silently murders every run in flight.

Pick the shape from the workload, not from the quickstart. Latency expectation, failure semantics, and state needs decide it — and each shape has a different answer to the only question that matters at 3 a.m.: what happens to a run that was halfway done?

Key terms: agent loop, durable execution, checkpoint, idempotency, blast radius

The three topologies, on the four axes that decide between them
Topology	Latency profile	Failure semantics	State needs	Platform primitives that fit
Request–response — chat, copilot, inline assistant	Sub-second to ~2 minutes; the caller blocks. Streaming tokens hide latency but not timeouts.	Crash = the caller sees a 5xx and retries. Retries re-run side effects unless your tools carry idempotency keys. Anything mid-flight at deploy time is lost.	Session-scoped only. Conversation history in a store, nothing else worth keeping.	A managed agent endpoint (a published platform agent, a container behind a load balancer, a serverless function). Cheap and boring — which is the point.
Queue–worker — batch triage, nightly enrichment, ticket processing	Seconds to hours; nobody is waiting on a socket. You control concurrency by worker count.	Crash = the message returns to the queue and is redelivered. At-least-once delivery makes idempotency mandatory, and a dead-letter queue is where poisoned tasks go to be looked at.	Per-task state in the job payload plus a result store. Fully external — workers stay disposable.	A queue plus a worker pool: any container runtime, a serverless consumer, or a platform agent invoked over a non-chat protocol (webhook-shaped invocation rather than a chat completion).
Durable long-running — deep research, migrations, multi-day approvals	Minutes to days, including waiting — for a human, for a slow external system, for tomorrow.	Crash = resume from the last checkpoint, not restart. Failure becomes a pause, and the run is expected to survive both eviction and deploys.	Durable, versioned run state: step history, tool results, cursor, pending approvals. The whole point.	A durable-execution engine or a runtime that persists session state and resumes it — plus a framework checkpointer under the loop. The most expensive shape to operate, so earn it.

Platform primitives — current as of September 2026

Managed agent runtimes now ship long-session primitives directly, which is what makes the durable shape practical without building your own orchestrator. Two documented examples, both worth re-checking against current docs and service quotas before you design around them:

Amazon Bedrock AgentCore Runtime runs each session in a dedicated microVM keyed by a runtimeSessionId, with sessions documented as terminating after 15 minutes of inactivity or an 8-hour maximum lifetime — and session state described as ephemeral, with durable context belonging in AgentCore Memory. A second substrate, Runtime Instances (GA August 2026), runs sessions up to 14 days on isolated EC2 instances with persistent volumes that re-attach on resume under the same session id.
Microsoft Foundry hosted agents run per session in a VM-isolated sandbox with a persistent filesystem ($HOME and /files), a configurable idle timeout of 2–60 minutes (default 15), and scale-to-zero with stateful resume: the platform deprovisions compute on idle and restores state when the session is used again.

Numbers like these move. Treat them as shape evidence — long sessions are a first-class primitive now — and cite the live docs for the values.

Request–response

Deploy checklist

Set a hard turn and wall-clock budget below the shortest timeout in the path (client, gateway, load balancer). The agent should give up before the socket does.
Stream partial output so a 90-second run does not look like a hang.
Make retries safe: an automatic client retry on timeout means the loop may already have sent the email. Idempotency keys on every write tool.
Drain on deploy: stop accepting new requests, let in-flight runs finish, then swap. Otherwise every rollout is a small outage.

Where it fails: anything that legitimately takes longer than a socket wants to live. Do not solve that by raising the timeout to 15 minutes.

Queue–worker

Deploy checklist

Assume at-least-once delivery. Design every tool call as idempotent or guarded by a dedupe key derived from the task id.
Cap redelivery. Three attempts, then dead-letter — an agent that fails on a poisoned input will fail identically forever, burning tokens each time.
Put the agent version in the message metadata and the result record, so you can attribute a bad batch to a version later.
Decide what a mid-deploy worker restart means: with a visibility timeout, the task simply comes back. That is the topology working as designed.

Where it fails: interactive work. Users will not tolerate "your answer will appear in the results table."

Durable long-running

Deploy checklist

Define the checkpoint boundary before you write the loop: after every tool result is the safe default, and it is the boundary state-machine thinking gives you for free.
Persist run state outside the compute: a durable store, not the container filesystem you were lucky enough to get back.
Record the full version tuple in the checkpoint (next lesson). Resuming a run on newer code is a mixed-version execution, whether or not you meant it.
Budget for waiting. A session that is idle-but-alive can still cost money — AgentCore Runtime documents billing peak memory for the life of a session even when no CPU is consumed, so an agent parked for three days is a line item.

Where it fails: short, cheap, chatty work. You will have built a distributed system to answer a question.

Five workloads, five deployment decisions

Interactive decision tree — outcomes:

Request–response
Right call. The caller is blocked, the work fits inside a socket, and streaming keeps it feeling alive. Cap turns and wall-clock below the tightest timeout in the path, and drain in-flight requests on deploy.
Hybrid: synchronous front door, queued long tail
The most common mature shape. Serve the p95 inline, detach the tail into a job with its own result channel, and show the user that it went long instead of timing out. Two topologies means two sets of failure semantics — document both.
Queue–worker
Right call. Independent bounded tasks with at-least-once delivery is exactly what queues are for. Idempotency keys make redelivery safe; the dead-letter queue turns a poisoned record into a ticket instead of an infinite retry bill.
Durable long-running execution
Right call. When restarting is unacceptable — hours of work, side effects already committed, or a human parked mid-run — the run needs checkpoints and a resume path. Pay the extra operational cost only for workloads like these.
Not an agent — a three-step job
Correct, and the most valuable answer in the tree. A fixed, enumerable path is a workflow: cheaper, deterministic, testable with assertions. Deploying an LLM loop here adds cost, latency, non-determinism, and an injection surface for zero capability.
Over-built for the workload
It will work, and you will maintain a queue, a result store, and a polling UI to deliver something a synchronous call did in 8 seconds. Topology cost is paid every day by the on-call engineer; buy it only when the workload demands it.
Holding a connection (or a session) hostage
This is the classic production failure. Sockets die at the gateway, retries silently re-run committed side effects, and any deploy takes every in-flight run with it. Waiting on a human is even worse: you are paying for parked compute to hold state that belongs in a checkpoint.
Retry-from-the-top on an expensive run
Queues restart tasks; they do not resume them. For a six-hour run that means burning six hours of tokens to re-reach the failure — and re-executing every side effect along the way. Once restart cost exceeds checkpoint cost, you have crossed into durable territory.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.