Two streams: tokens and events

Token stream

What it carries: deltas of assistant text (and, on most providers, deltas of tool-call arguments as they are generated).

What the UI does with it: appends. The rendering rule is buffer += delta.

Properties you must respect:

• Frames are order-dependent and not idempotent. Replay a text delta and the word appears twice.
• A delta is not valid JSON on its own. Tool-call arguments arrive as {"cus, tomer_i, d": "A-9 — you cannot parse until the call is complete, so never act on a half-built argument.
• The stream ending is not the same as the work succeeding. Always confirm terminal status from the event layer.

Event stream

What it carries: run status transitions, step created/completed, tool call started, tool output received, approval required, usage totals, terminal status, errors.

What the UI does with it: renders activity — “Searching orders…”, “Waiting for your approval”, a step timeline, a spinner that means something.

Properties you must respect:

• Events should be individually meaningful and ideally idempotent — “run 7f3 is now in_progress” can be applied twice with no harm. Design them that way deliberately.
• Events are what your traces and evals consume later. The same event log that drives the UI is your debugging record.
• Best-effort delivery is normal. Assume some events are missed and reconcile against authoritative state (lesson 3).

What a real agent UI subscribes to

Both, multiplexed over one connection, with the UI keeping two separate state machines.

A production agent surface renders roughly four things at once: the transcript (from committed messages), the in-flight answer (from token deltas), the activity line (from run/step events), and the interrupt (from an approval-required event that has to block the UI until answered).

The failure mode of implementing only tokens: 30 seconds of dead air while a tool runs, users hammering the retry button, and no way to show an approval prompt.

The failure mode of implementing only events: technically correct, no perceived-latency win, and answers that materialise in one lump — which users read as slower even when it is not.

Rule 1 · Do nothing expensive per frame

Append to a buffer and flush on a timer — every 50–100 ms is invisible to humans and cuts your work by an order of magnitude. Persist on events, not on tokens: one write when a step completes beats 400 writes while it runs.

Rule 2 · Bound every buffer, and decide what to drop

An unbounded queue is a memory leak with a delay fuse. Bound it, and make the shedding policy explicit: coalesce text deltas (three fragments concatenated are still correct), but never drop events — dropping “requires_action” means an approval prompt that never appears. Different frames, different policies.

Rule 3 · Fan-out is a multiplier

One agent run shown in a browser, a mobile app, and a Slack thread is three consumers at three speeds. Do not let the slowest one throttle the run: terminate the stream into your own durable event log, then let each client read from that at its own pace. This is the same move that makes reconnection possible.

Rule 4 · The network in between is not on your side

Reverse proxies buffer, CDNs coalesce, corporate middleboxes close idle connections, and mobile radios drop sockets on a bus ride. Send keepalives, expect disconnects as routine rather than exceptional, and never treat “the stream ended” as “the run finished”.