Isolation boundaries, layer by layer
Lesson 2 of 5 in Multi-Tenant Isolation: One Platform, Many Customers, Zero Bleed.
“Are your tenants isolated?” is not answerable as one question. Isolation is a property of each layer separately, and a platform can be immaculate at one layer and wide open at the next — a per-run microVM that destroys its filesystem on exit gives you nothing if the agent inside it reads from a global memory namespace with a global API key.
So do the boring thing: enumerate the layers, name the boundary at each, and write down how you would prove it holds. Five layers cover almost every agent platform — session and compute, memory and retrieval, credentials, egress, and observability.
| Layer | The boundary that must exist | What a violation looks like | How you prove it holds |
|---|---|---|---|
Session / compute — the run itself | One tenant per run: a fresh process, container, microVM, or sandbox, with no filesystem, environment, or in-process cache carried over. Never multiplex two tenants inside one agent loop. | A long-lived worker keeps a warm client, a temp directory, or an in-memory conversation store keyed only by user id; tenant B inherits tenant A’s scratch state. | Write a canary file and an in-memory canary in a run for tenant A, then assert a tenant-B run cannot see either. Run it in CI, not once by hand. |
Memory / retrieval — the durable stores | A namespace per tenant in every store: memory records, vector collections, conversation history, uploaded files. Tenant id is a path segment or collection name, plus a provenance tag on every record. | One index for everyone with a filter applied in application code — and one code path that forgets, or a broad read key that matches another tenant’s records. | Query each store as tenant A with tenant B’s ids and assert an empty result. Then assert every item in an assembled context carries tenant A’s provenance tag before the model call. |
Credentials — the authority behind each tool | Per-tenant tool auth, brokered at call time from the pinned tenant id: downscoped tokens, per-tenant connections, or on-behalf-of user tokens. Never a global service account shared across tenants. | One vendor API key and a | Enumerate every tool credential and answer: whose data can this specific secret reach? If the answer is more than one tenant, that is a finding — see security/agent-identity-auth-secrets. |
Egress — where the agent may talk to | A per-tenant destination allowlist enforced at the network layer, so a run for tenant A can only reach A’s endpoints, A’s webhooks, and A’s approved third parties. Egress control is what turns a data-read into a non-exfiltration. | One platform-wide allowlist that is the union of every tenant’s destinations, so an injected instruction can POST tenant A’s data to tenant B’s (or an attacker’s) approved endpoint. | From a tenant-A run, attempt a connection to a tenant-B destination and assert it is refused at the network layer, not merely unused. |
Observability — traces, logs, and evals | Every trace, span, metric, and eval record carries a tenant attribute; access to them is scoped by tenant, and cross-tenant queries are a privileged, audited operation for a named group. | One log group and one dashboard where any support engineer can grep the whole corpus, with prompts and tool payloads recorded in full. Debugging becomes a lawful-looking cross-tenant read. | Check that the tenant attribute is required at ingest (drop or quarantine spans without it) and that a normal engineer’s role cannot query across tenants without an approval trail. |
How the tenant id has to travel
- Authenticated request arrives
The only trustworthy source of tenancy is the caller’s verified credential — a JWT claim, a SigV4 identity, a mTLS certificate. Not a header, not a body field, not the prompt.
- Resolve tenant from the verified claim
Map the claim to an internal tenant id and load that tenant’s configuration: tier, region, allowlist, quota, key material.
- Tenant resolvable and active?
- Reject the request
Never default, never infer, never fall back to a “shared” tenant. An unresolvable tenant is a 403, not a best guess.
- Harness pins tenantId into the run context
Pinned means immutable for the life of the run and invisible to the model: it lives in harness state, not in the context window.
- Session or sandbox created for this tenant only
One tenant per run. No warm worker reuse across tenants, no shared scratch volume, no shared browser profile.
- Stores addressed by tenant namespace
Memory, vector index, files, conversation history — all keyed by a path or collection containing the tenant id.
- Credential brokered for this tenant
The broker takes the pinned id, not a model-supplied argument, and returns a token whose own scope is one tenant.
- Egress allowlist loaded for this tenant
Network policy for this run permits only this tenant’s destinations, enforced outside the agent process.
- Every context item carries this tenant’s provenance?
A cheap pre-flight assertion over the assembled context — the single highest-value guard in the whole chain.
- Halt run, quarantine, alert
Treat a provenance mismatch as a security event, not a retryable error: capture the context, page the on-call, and do not let the model see the item.
- Model call proceeds; spans tagged with tenantId
Tagging at emit time is what makes per-tenant traces, per-tenant cost accounting, and per-tenant deletion possible later.
Two details in that chain do the heavy lifting. First, pinned and invisible: the tenant id lives in harness state, so no amount of clever text in a document, a tool response, or a user message can change it. Second, the provenance assertion before the model call: it is a few lines of code that turns every future mistake at every store into a caught error instead of a leak.
Then make the boundary testable. Keep two permanent canary tenants in staging, each holding a distinctive secret string, and run a suite that tries — as tenant A — to retrieve A’s canary (must pass), to retrieve B’s canary through every store and cache (must fail), and to reach B’s egress destination (must be refused at the network layer). A tenancy claim you cannot re-verify on every deploy is a claim that quietly expires.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.