Multi-Tenant MCP Architecture: Secrets Isolation & Environment Sandboxing

# Multi-Tenant MCP Architecture: Secrets Isolation & Environment Sandboxing


*By the security engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: Scaling MCP Across Hundreds of Client Environments

The Model Context Protocol (MCP) gave every LLM host a standard way to discover and invoke tools, resources, and prompts. But the protocol deliberately says nothing about *tenancy*. There is no tenant identifier in a tool call, no boundary in the wire format, and no notion of “who pays for this inference.” That is the platform engineer’s job — and it becomes existential the moment you run one MCP fleet for more than one customer.

The failure mode is not exotic. It is the platform that starts as a demo — one desktop config pointing at one server with one database — and then wins six enterprise pilots. Six becomes sixty, then six hundred agents per day, each with its own Postgres, its own S3 bucket, its own API keys into the customer’s Salesforce, Snowflake, and Stripe. The naive move is one server per tenant: correct, but unreachable at fleet scale. A hundred tenants mean a hundred deployments, a hundred patching windows, a hundred drift-prone IAM roles. The platform move is **shared control planes with isolated data and execution planes** — isolation that is structural, not procedural.

Everything below answers one question first: **what breaks if this layer is the only thing standing between tenant A and tenant B’s data?** If the answer is “everything,” the layer is load-bearing and must be enforced by the platform, never requested of the model. We cover secrets isolation, network sandboxing, protocol-layer session boundaries, tool sandboxing, and governance. The recurring theme: **every boundary must be a mechanism, not a convention.**

## 2. Secrets Isolation: AWS Secrets Manager vs. HashiCorp Vault per Tenant

Start with the most dangerous object in the system: the credential. A multi-tenant MCP server holds hundreds — the customer’s warehouse password, their model-provider key, their OAuth refresh tokens into a SaaS estate. The cardinal rule is that **no tenant secret may ever be decrypted by code reachable from another tenant’s session.** That eliminates the two most common architectures at once: a shared environment-variable bag, and a shared secrets table with a `tenant_id` column. A table with a tenant column is a *filter*, not a boundary — every rotation job, debug endpoint, or SQL error becomes a potential cross-tenant read. Secrets management in a multi-tenant platform is not about storage; it is about **decryption proximity**: not where the ciphertext lives, but which principal can decrypt it, and which session can invoke that principal.

### AWS Secrets Manager: the VPC-adjacent default

The clean pattern is a **per-tenant KMS key, a per-tenant secret, and a tenant-scoped IAM role** the MCP server assumes only while serving that tenant’s traffic:

“`json
{
“Statement”: [{
“Effect”: “Allow”,
“Action”: “secretsmanager:GetSecretValue”,
“Resource”: “arn:aws:secretsmanager:us-east-1:123456789012:secret:tenant-acme/*”
}, {
“Effect”: “Allow”,
“Action”: “kms:Decrypt”,
“Resource”: “arn:aws:kms:us-east-1:123456789012:key/tenant-acme”
}]
}
“`

The KMS key is the load-bearing wall: even a misconfigured IAM policy that lets a role *read* the secret blob fails at `kms:Decrypt` without the tenant key, and every fetch lands in CloudTrail with the role ARN for per-tenant forensics. Watch the cost model at scale, though — Secrets Manager bills per secret and per 10,000 calls, so cache values in memory with a short TTL, force-refresh on rotation events, and budget hard against uncached fetches.

### HashiCorp Vault: when the boundary must be dynamic

Vault earns its place when tenancy is more dynamic than “one static key per customer.” With **dynamic database credentials**, the server never holds a standing password at all: when tenant A’s session needs warehouse access, the server calls Vault with a policy-bound token, and Vault mints a **unique, time-boxed Postgres credential for that session**, revoking it when the session ends. A leaked secret’s blast radius drops from “the customer’s password, forever” to “a synthetic credential valid for fifteen minutes, usable only from the tenant’s network path.” For long-running agent sessions, that is the difference between a reportable breach and a non-event:

“`hcl
path “database/creds/acme-warehouse” {
capabilities = [“read”]
}
“`

The trade-offs are real. You now operate Vault — an HA stateful service with unseal ceremony and upgrade discipline, and the most-targeted service in your estate. Our heuristic: choose **Secrets Manager when tenancy is stable and cloud-native fits**; choose **Vault when you need ephemeral, session-scoped credentials** — mandatory for regulated customer data where standing passwords are unacceptable. Either way, the platform rule is identical: **the tenant boundary is the decryption key, never the application code.**

## 3. Network Sandboxing: VPC Endpoints and Security Groups

Secrets isolation answers “who can decrypt.” Network isolation answers “who can even *reach* the thing.” We build it with two primitives: **security groups as tenant firewalls** and **VPC endpoints as private, metadata-free ingress**.

The architecture is a **hub-and-spoke topology**. The MCP control plane (shared servers, session registry, rate limiters) lives in a hub VPC; each tenant gets a **spoke** — or a dedicated account for regulated workloads — containing only their data plane: Postgres, S3, warehouse connections. Spokes have no Internet Gateway, no NAT, no public subnets. The only path in is a peering or Transit Gateway attachment to the hub, filtered by a security group that references the hub’s MCP group ID — a *group*, not a CIDR:

“`
┌──────────── HUB VPC ────────────┐
│ ALB ─► MCP ASG (sg-mcp) │
│ │ │ │
│ Session Registry Secrets │
│ (sg-control) (sg-sec) │
└───────┬──────────┬──────────────┘
peer │ │ peer
┌───────▼──────┐ ┌─▼─────────────┐
│ Spoke A │ │ Spoke B │
│ sg-tenant-a │ │ sg-tenant-b │
│ Postgres/S3 │ │ Postgres/S3 │
└──────────────┘ └───────────────┘
“`

Tenant A’s database group allows inbound **only from `sg-mcp`**, only on the DB port, only over the peering link. Tenant B’s group is identical in shape, entirely disjoint in effect. Because rules reference groups rather than IPs, autoscaling the fleet never breaks tenant connectivity — and because there is no spoke-to-spoke route, a compromised spoke cannot pivot laterally.

The second primitive is **VPC endpoints for every service the tenant path touches** — S3, Secrets Manager, STS, DynamoDB. If the MCP server reached S3 over the public endpoint, every object fetch would transit the internet, outside the scope of your security groups and flow logs. Endpoints keep tenant data-plane traffic on the AWS backbone, inside your routing table, where policy applies uniformly. Two rules complete the posture: **never put a NAT gateway in a spoke** (spoke egress is how exfiltration happens), and **enable VPC Flow Logs on every attachment**, aggregated where agents cannot reach them — when an incident asks “did tenant A’s session touch tenant B’s bucket?”, flow logs are the only answer that survives broken application logging.

## 4. Protocol Layer Isolation: Enforcing `Mcp-Session-Id` Boundaries

Secrets and networks isolate at the infrastructure layer. But the MCP server is shared code — one process namespace, one connection pool, one log buffer — so the protocol layer must enforce a boundary no infrastructure policy can: **every tool call executes in exactly one tenant session, and the server proves it on every request.**

The boundary token is the session. In the MCP Streamable HTTP transport, the server issues a session identifier that the client returns on every request — commonly `Mcp-Session-Id`. The platform rule: **`Mcp-Session-Id` is a capability, not a claim.** The server must never trust the header alone; it resolves the session server-side, verifies its authenticated principal, and binds *everything* downstream — the secret-decryption role, the database connection, the tool allowlist, the rate-limit bucket — to the resolved session, never to header-derived strings:

“`
1. Resolve sess_9f3c… in the session registry
→ tenant: acme, principal: ops-bot, scopes: [salesforce:read]
2. Unknown / expired / revoked → 401, no handler runs
3. Bind tenantID = acme (server-side, not from input)
4. Open acme’s DB connection pool; assume acme’s decrypt role
5. Execute tool against acme’s allowlist; write audit event
“`

Three enforcement details separate a real boundary from theater. **First, no cross-session state in the handler:** connection pools are per-tenant, caches key by the *resolved* tenant ID, and logging redacts anything outside it — a bug that lets one session read another’s cache entry is a leak no network policy catches. **Second, sessions are short-lived and revocable:** an agent session can run for hours, so the registry (Redis or DynamoDB) carries a TTL refreshed on activity, and revocation events — rotated credentials, terminated employees, abuse thresholds — kill every live session for a tenant in seconds. **Third, IDs are opaque, high-entropy, and tombstoned:** a 128-bit random value encoding no tenant identity means the header is useless without registry access, and closed sessions fail closed on replay instead of resurrecting dead context. Write the server as if the header is attacker-controlled — on any shared host, it is.

## 5. Tool Sandboxing: gVisor or Firecracker for Untrusted Code

The layers so far isolate tenants from *each other*. Tool sandboxing isolates the platform from the **tools themselves** — and from the model invoking them. This matters most once tenants register custom tools, or the server executes code on behalf of a prompt-injectable LLM. The rule: **no tenant code ever executes in the MCP server process, or even on the same kernel.** Two technologies dominate, at different points on the security-to-overhead curve.

**gVisor** is a user-space kernel that intercepts the application’s syscalls and implements them in Go behind a layer that distrusts the host kernel’s syscall surface. It runs as a normal process via `runsc`, boots in milliseconds, and drops into containerd as a runtime class — operationally light. The pattern is one sandbox per tool invocation or short session: read-only root filesystem, no network by default, tmpfs for scratch. Its sweet spot is **high-frequency, low-privilege calls** — data transforms, retrieval steps, schema introspection — at thousands per minute where virtualization would be uneconomical. Its trade-off: it shares the host kernel’s posture, a strong boundary against application exploits but not against a hostile host.

**Firecracker** is a microVM — a minimal VMM booting a stripped Linux guest in ~125 ms with under 5 MiB of overhead. Each sandbox is a *real VM* with its own kernel and device model and no shared syscall surface: the right answer when tools run **tenant-supplied or model-generated code with network access to customer data**, where a sandbox escape would mean cross-tenant compromise. Its cost is operational — guest images, kernel updates, and snapshotting to keep a warm pool.

The platform pattern is a **two-tier execution model**: Tier 1 (gVisor, per call) for built-in trusted tools at tens-of-milliseconds latency; Tier 2 (Firecracker, per session) for tenant-registered or code-generating tools, each microVM wired into its own tenant’s network namespace — plugging straight into the VPC endpoints and security groups above, so even a fully compromised sandbox can only reach its own spoke. Whichever tier, three rules hold: **no outbound network by default** (egress only via an audited allowlist), **no host filesystem or Docker socket** (all I/O through the session-bound MCP tool interface), and **hard wall-clock and memory caps enforced by the sandbox** — a tool that can run forever is a tool that can mine cryptocurrency on your neighbor’s capacity, and “the model said stop” is not enforcement.

## 6. Governance: Multi-Tenant Cost Attribution and Rate-Limit Enforcement

Isolation without governance is unmanageable; governance without isolation is unfair. The final layer answers two business questions — **who pays, and who is hogging the platform** — from the same session-bound telemetry the security layers already produce.

**Cost attribution** starts with the session as the cost unit. Every invocation emits an event carrying the resolved tenant ID, tool, sandbox tier, wall-clock time, CPU, memory, tokens, and downstream calls (secrets lookups, database queries, model inference). Aggregated per tenant per day, that stream becomes a cost report — but only if tenant ID is **unforgeable in telemetry**, taken from the same resolved-session context as the security layers, never from client-supplied fields. Direct attribution (tenant → tool → resource) covers the common case; **shared-pool attribution** covers the control plane no single tenant caused — a platform fee plus usage-weighted allocation, recomputable from the audit log so finance trusts it and tenants can dispute it in the open.

**Rate limiting** is where governance meets availability, and multi-tenant makes naive limits dangerous. A global limit lets one tenant starve everyone; frozen per-tenant limits waste idle capacity. The workable design is token buckets enforced **at the session router**, keyed by resolved tenant, at three horizons:

“`json
{
“tenant”: “acme”,
“rps”: { “capacity”: 50, “refillPerSec”: 5 },
“concurrent”: { “max”: 20 },
“tokensPerDay”: { “capacity”: 500000 }
}
“`

A per-second burst cap; a concurrency cap that doubles as the memory-exhaustion defense (each concurrent session pins sandbox or connection resources); and a daily token budget that maps directly onto the billing ledger. Enforcement lives at the **router, before expensive work begins**, and an over-limit tenant gets a structured 429 with `Retry-After` so their agents back off gracefully. Limits must also be **adaptive, not static**: near the daily budget, throttle or queue rather than kill mid-task; with idle fleet capacity, let a well-behaved tenant burst. Static limits are a policy; adaptive limits driven by the same telemetry that feeds billing are a platform.

Close the loop by writing every rejection, throttle, and sandbox denial into the same immutable audit store as the security events — governance data and security data are the same data from different seats. A tenant spiking to 10x normal volume is either scaling up or exfiltrating, and the SOC and the billing team should be watching the same dashboard when it happens.

—

**Building a multi-tenant MCP platform — or retrofitting isolation into one that already shipped? Talk to the OnlineCloud architects.** We are a US-based engineering studio — MCP-native, senior engineers only, no offshore handoff — and we deliver session-bound secrets isolation, VPC-sandboxed data planes, and audited cost governance in production. Contact us at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/) for a free initial consultation. We respond within 24 hours.