Computer use and browser agents: injection at the width of the web

Lesson 4 of 5 in Sandboxing and Computer Use: Isolation for What You Cannot Pre-Approve.

Computer use is the pattern where the agent stops calling APIs and starts driving a screen: screenshots and the page tree go in, clicks and keystrokes come out. It exists for an excellent reason — the long tail of systems that will never have an API, the internal portal from 2011, the supplier site with no integration and no intention of building one.

It is also the widest attack surface an agent can have, for three independent reasons. The agent reads whatever the page says, so every page is untrusted input and every rendered byte is a candidate instruction — indirect prompt injection at the width of the web. The permission model collapses to a session cookie: there is no per-action schema to scope, so the agent can do whatever the logged-in human could do, with no parameter caps to enforce. And the audit trail degrades from typed tool calls to a stream of coordinates and screenshots, which is far harder to review, alert on, or replay.

A scoped API tool versus driving a browser
PropertyScoped API toolComputer use / browser agent

What the agent perceives

A typed response you defined and can filter before it reaches the model.

Whole rendered pages: body text, alt text, hidden elements, ads, comments, other users’ content.

Permission granularity

Per operation and per parameter — read-only scopes, amount caps, recipient allowlists.

Whatever the authenticated session can do. The unit of authority is “logged in.”

Injection surface

The tool’s response payload — bounded, and you can strip markup and links.

Every site visited, plus anything those sites embed. Text invisible to a human still parses.

Auditability

A structured trace: name, arguments, result, timestamp. Alertable and diffable.

Screenshots, DOM snapshots and click coordinates. Reviewable only with real tooling — session recording, live view, video.

Failure mode

A rejected call, a validation error, a clear exception.

A confidently wrong click: the wrong row, the wrong button, a submitted form, a second submission after a retry.

Right tool when

An API exists, or could be written. Almost always prefer this.

No API exists and the workflow is genuinely worth the containment budget.

Rule 1 · A fresh, isolated browser profile every session

No developer profile. No extensions. No saved passwords, no autofill, no history, no bookmark bar full of internal tools. The profile is created for the session and destroyed with it, inside an ephemeral environment.

Why it is rule one: a browser profile is a bundle of ambient credentials — cookies for every SaaS product the owner uses, refresh tokens, SSO sessions. Handing an agent an existing profile is the browser version of injecting production keys into the sandbox, except worse, because nobody can enumerate what is in there.

Persisted authenticated profiles are a real feature with real uses (long-running workflows on one site that will not accept programmatic login). If you use one, pin it to a single allowlisted domain, keep untrusted browsing out of that session entirely, and treat it as a stored secret with rotation and an owner.

Rule 2 · Never a real human’s logged-in session

The tempting shortcut is to let the agent drive a browser already logged in as the employee. That grants the agent the union of everything that employee can reach — mail, HR, admin consoles — and destroys attribution: the audit logs of every downstream system now say the human did it.

Instead: a dedicated automation identity, provisioned per workflow, with the narrowest possible entitlements on the target system. Credentials come from a vault at the moment of use, never from the profile store. And if the target site cannot express a low-privilege role, that is a finding — write it down and gate the workflow behind a human rather than pretending the risk away.

Rule 3 · Domain allowlist, and egress control around the browser

The agent visits the sites the task requires and nothing else. Enforce it at the network layer as well as in the prompt — a browser is a general-purpose HTTP client with a rendering engine, and a page can initiate requests the agent never chose.

An allowlist does double duty: it cuts the injection surface and it cuts the exfiltration channel, because a rendered image or a background fetch can only reach approved hosts. ForcedLeak is the cautionary tale on the exfil side — Noma Labs reported on 2025-09-25 that Salesforce Agentforce could be made to leak CRM data through an expired domain still sitting on the CSP allowlist. Allowlists need owners and expiry checks, or they rot into open doors.

Rule 4 · A human can watch, take over, and stop it

Live streaming, session recording, and a kill switch that terminates the session mid-action. For consequential steps — the submit, the send, the purchase, the delete — an approval gate that shows the human the actual screen state, not a summary the model wrote.

This is where computer use differs from a scoped tool: you cannot describe the pending action in a schema, so the review has to be visual. Design for the reviewer to be able to say no in a few seconds, and remember the human-in-the-loop failure mode from foundations — a gate that always says yes has already become full autonomy.

Rule 5 · Fix the plan’s targets before the browsing starts

Decide outside the agent what the irreversible parameters are — which recipient, which amount, which record — and enforce them in the runtime. Then no amount of page text can retarget the action, because the target was never the model’s to choose after reading the web.

This is the structural answer to “the page told the agent to do something else.” You cannot filter your way out of injection, so remove the decision from the injectable path. The same instinct as Meta’s Agents Rule of Two (2025-10-31): if a session processes untrusted input, touches sensitive data, and can change state or communicate externally, it should not run autonomously — it needs human supervision or another dependable validation mechanism.

One honest note on reliability, because it changes the security calculus. Multi-step GUI automation is, as of September 2026, the least reliable surface agents have: layouts shift, modals appear, timing races, anti-bot checks fire, and a click that landed correctly a hundred times lands on a different row when a table re-sorts. Public demos are curated; your flows are not. Measure per-task pass rates on your workflows before believing anything, and re-measure after every site change.

Unreliability and risk multiply rather than trade off. A confused agent retries — and retries on a browser mean double submissions, duplicate orders, and repeated messages. So the containment rules above are not only anti-attacker measures; they are also what keeps an ordinary bad day from becoming an external, irreversible, customer-visible event.

Tool: Prompt Injection Range — Fire indirect-injection payloads at a browsing agent and watch which containment rule stops which chain, in the Injection Range.

Key terms: computer use, prompt injection, lethal trifecta, egress control, kill switch, approval gate

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.