Implementing Multi-Tenant Secret Isolation for Agentic Tool Chains

# Implementing Multi-Tenant Secret Isolation for Agentic Tool Chains


*By the OnlineCloud Security Architecture team*

—

## 1. Introduction: The Risk of Credential Bleeding in Shared Agent Environments

The agentic era has quietly collapsed one of the most reliable security invariants of the last two decades: **a process runs with the credentials of exactly one identity.** When we moved from monoliths to microservices, we painstakingly decomposed ambient authority into least-privilege service accounts. We built IAM policies, short-lived instance roles, and workload identity federation precisely so that no single credential could unlock more than it needed. Agentic tool chains — the Model Context Protocol (MCP) servers, the planners, the retrievers, the code executors — have started to reverse that progress.

Consider the typical deployment. A single orchestration process hosts N tenant sessions. Each session can call M tools: a SQL MCP server, a GitHub MCP server, a Slack MCP server, an internal ticketing system, a browser automation tool. To make this work, the platform loads an *environment* — a bag of API keys, OAuth refresh tokens, database DSNs, and signing secrets — and hands it to every tool invocation. The moment one tenant’s prompt-injection payload convinces a tool to `printenv`, or a shared cache stores a rendered template containing a token, or a tool crashes and dumps its context to a shared log, **tenant A’s credential is now readable by tenant B.**

This is **credential bleeding**: the involuntary lateral movement of secrets across trust boundaries that were never designed to share a process. It is not a theoretical risk. It is the predictable consequence of three design shortcuts that agent frameworks make by default:

1. **Process-level secret loading.** Secrets are injected as environment variables or a global config object that any in-process tool can read.
2. **Long-lived credentials.** API keys issued once and reused for the lifetime of the deployment, with no per-invocation binding.
3. **Undifferentiated audit trails.** The tool-call log and the secret-manager access log are separate systems with no shared correlation identifier, so no one can answer “which session’s tool call retrieved which secret?”

The blast radius is severe because agents are *explicitly designed to act autonomously on untrusted input*. An LLM that ingests a malicious document, a poisoned tool description, or a crafted user message is a confused deputy waiting to happen. When the credential is ambient, the deputy has nothing stopping it from using tenant A’s key to service tenant B’s request.

This post describes the architecture we use at **OnlineCloud** to make credential bleeding structurally impossible rather than merely unlikely. We treat every agent session as a hostile, multi-tenant workload and apply four controls in layers: **secret isolation** (per-tenant KMS key + vault path), **token scoping** (short-lived, audience-bound credentials minted per tool call), **auditing** (reconciling tool-call events with secret access logs), and a **per-session environment sandbox** that never persists a plaintext secret beyond the duration of a single invocation. We close with a reference Node.js implementation of secret propagation in an MCP server.

—

## 2. Secret Isolation: Designing a Per-Tenant KMS Key and Vault Architecture

The foundation of isolation is **key separation**. If two tenants’ secrets are encrypted under the same key, then any compromise of that key (or of the decryption path) collapses the boundary. Our rule is simple: **one tenant, one KMS master key, one vault namespace, one encryption context.**

### 2.1 The key hierarchy

We use a three-tier hierarchy:

– **Root KMS key (per region).** Holds no tenant data directly. Signs and derives nothing; it exists only to wrap tenant data keys.
– **Tenant KMS key (per tenant).** A customer-managed key with a strict key policy that grants `Encrypt`/`Decrypt` only to the tenant’s own service identity. Crucially, the key policy includes a **`kms:EncryptionContext:tenant_id` condition**, so even a caller who holds the identity cannot use the key to decrypt another tenant’s ciphertext.
– **Envelope data key (per secret version).** Generated locally, used once to encrypt a single secret payload, then discarded. Only the wrapped data key is persisted.

The encryption context is the subtle but essential control:

“`json
{
“TenantId”: “tnt_8f3a”,
“SecretPath”: “tools/github/pat”,
“SessionId”: “sess_9c21”,
“Purpose”: “tool-invocation”
}
“`

KMS treats encryption context as *additional authenticated data*. A request to decrypt with a mismatched `TenantId` fails cryptographically — not because of an authorization check that could be misconfigured, but because the ciphertext simply will not authenticate. This makes cross-tenant decryption a **hard cryptographic failure**, which is exactly the property we want.

### 2.2 Vault path topology

On top of KMS, secrets live in a vault (HashiCorp Vault, AWS Secrets Manager, or our managed equivalent) with a namespaced path that mirrors the tenant boundary:

“`
kv/tenants/{tenant_id}/tools/{tool_name}/{credential_name}
“`

Access is governed by a policy templated at tenant-onboarding time. The policy grants **read on exactly one path prefix** and nothing else. There is no wildcard that spans tenants, and no “admin” role short of break-glass, which itself requires two-person approval and emits a distinct audit event.

Two design decisions matter here:

– **No shared “service” namespace.** A tempting shortcut is a `kv/shared/` path for common credentials. We reject it. Shared secrets are shared blast radius. If two tenants need the same provider, each gets its own credential, ideally issued from the same upstream identity with distinct scopes.
– **Write-once, rotate-often.** Secret versions are immutable; rotation creates a new version and updates the tenant’s policy to the new version while keeping the previous version valid for a grace window measured in minutes, not days. This bounds the window in which a leaked credential is useful.

—

## 3. Token Scoping: Minting Short-Lived Scoped Credentials for Downstream Tool Execution

Isolation answers *who can read which secret*. Scoping answers *what that secret can do, for how long, and to whom*. The goal is to ensure that even if a credential leaks mid-flight, it is worthless outside a narrow, time-boxed, audience-bound context.

### 3.1 From long-lived secrets to derived tokens

We never hand a tool the raw provider credential. Instead, the secret broker performs a **token exchange** at invocation time:

1. The orchestrator authenticates the agent session using its workload identity (SPIFFE/mTLS), yielding a session token bound to `{tenant_id, session_id, agent_identity}`.
2. For each tool call, the orchestrator requests a scoped credential from the broker, passing the intended **audience** (e.g., `mcp://github.tools`) and the **required capability** (e.g., `repo:read`).
3. The broker verifies the session token, checks the tenant policy, decrypts the underlying secret, and *derives* a short-lived credential:
– **For OAuth providers:** a down-scoped access token via RFC 8693 token exchange, or an STS session with an inline policy limited to the requested capability.
– **For cloud providers:** temporary credentials from STS with a session policy and a 5–15 minute lifetime.
– **For static API keys:** a broker-held key never leaves the broker; instead the broker acts as a **reverse proxy** that injects the credential server-side, so the tool only ever sees a broker-scoped bearer token.

### 3.2 Properties we enforce

– **Short TTL.** Default 300 seconds, hard ceiling 900. The token’s expiry is recomputed per call; there is no “refresh and reuse” path.
– **Audience binding.** The token’s audience claim must match the MCP server it is presented to. A GitHub token will not satisfy a Slack tool’s audience check, defeating token replay across tools.
– **Capability minimization.** The derived token carries only the scopes the *current* call needs, computed from the tool’s declared schema — not the union of everything the agent might do.
– **Sender-constrained tokens.** Where supported, we use DPoP or mTLS-bound tokens so a stolen token is useless without the corresponding private key held in the session sandbox’s memory.
– **Single-use where possible.** One-shot tokens for high-risk operations (writes, deletes, financial actions) carry a nonce that the broker invalidates on first use.

The economic effect is that we convert a **long-lived, ambient, over-scoped** credential into a **short-lived, audience-bound, minimal** one. The compromise window shrinks by roughly two orders of magnitude, and lateral movement between tools is blocked by audience binding.

—

## 4. Auditing: Reconciling Tool-Call Events with Secret Manager Access Logs

Isolation and scoping are preventive controls; auditing is the detective control that proves they work and surfaces the moment they don’t. The core requirement is **correlation**: every secret access must be traceable to exactly one tool call, one agent session, and one tenant.

### 4.1 The correlation identifier

We generate a **`trace_id`** at the start of each agent session and a **`call_id`** per tool invocation. Both are propagated:

– Into the secret request as the KMS `EncryptionContext` and the vault request metadata.
– Into the tool-call event emitted by the orchestrator.
– Into the derived token’s `jti` claim.

Because the `call_id` is captured in the KMS access log (via encryption context), the vault audit log (via request metadata), and the orchestrator’s tool-call stream, we can **join** the three data sets on a single key. That join is what turns raw logs into a defensible narrative:

> At 14:02:11Z, session `sess_9c21` (tenant `tnt_8f3a`) invoked `github.createIssue` (`call_id=cl_5510`). The broker issued a 300s token (`jti=tk_88a1`) with scope `repo:write`. KMS `Decrypt` on tenant key `key_8f3a` succeeded with matching encryption context. No other secret was accessed in this session.

### 4.2 Reconciliation rules and anomaly detection

We run a nightly and near-real-time reconciliation that flags:

– **Orphan secret reads** — a vault read with no matching tool-call event (possible exfiltration or a misbehaving tool).
– **Orphan tool calls** — a tool call with no corresponding secret access (possible hard-coded credential or a bypass of the broker).
– **Context mismatch** — a KMS decrypt where `TenantId` in encryption context differs from the requesting session’s tenant (should be impossible; treat as a P1).
– **Scope creep** — a token issued with scopes broader than the tool’s declared schema requires.
– **Fan-out anomalies** — a single session reading secrets across more than K tool namespaces in a window.

Alerts route to the tenant’s SOC with the full correlated timeline. Critically, tenants can only query their own correlation trail, preserving the isolation boundary *in the observability plane as well as the data plane* — a detail many platforms miss.

—

## 5. Architecture: Implementing an ‘Environment Sandbox’ per Agent Session

The environment sandbox is the runtime construct that enforces everything above. Think of it as an **ephemeral capsule** wrapping a single agent session, with three guarantees: **no ambient secrets, no persistence, no cross-session memory.**

### 5.1 Sandbox composition

For each session the platform provisions:

– **A dedicated process tree** (or microVM/gVisor sandbox) with its own mount, PID, and network namespaces. Tools cannot see other sessions’ processes or sockets.
– **A secrets tray** that is *empty at rest*. The tray holds no plaintext; it holds only a broker client and the session’s workload identity. Secrets are pulled into the tray **only for the duration of a single tool call**, then zeroed.
– **Memory hygiene.** Decrypted secrets are written to page-locked buffers, used, and explicitly zeroed. We disable core dumps and enforce `mlock`/`madvise(MADV_DONTDUMP)` so secrets never reach swap or crash artifacts.
– **Egress allow-listing.** The sandbox can reach only the broker and the MCP servers in its tenant’s allow-list. This blocks the classic exfiltration path.
– **Per-session log redaction.** All stdout/stderr and tool output pass through a redaction filter seeded with the session’s live secret fingerprints, so a leaked value in a log dump is masked before it is written.

### 5.2 Lifecycle

1. **Provision.** Session starts with identity only. Broker issues a session token; no secrets are loaded.
2. **Invoke.** On each tool call, the sandbox requests a scoped token from the broker. The broker approves, mints the token, and the sandbox injects it into the tool’s invocation context — never into the environment block.
3. **Execute.** The tool runs, receives a token that expires in minutes and is audience-bound.
4. **Teardown.** On session end (or idle timeout, default 10 minutes), the sandbox is destroyed. All buffers are zeroed, the process tree is torn down, and the session token is revoked. There is no “warm pool” that carries state between tenants.

The sandbox ensures that isolation is a **runtime property**, not a configuration convention. Even a fully compromised tool inside the sandbox cannot reach another tenant’s secret, because the secret was never co-resident with it.

—

## 6. Implementation: A Reference Node.js Example of Secret Propagation in MCP

Below is a compact reference implementation of an MCP server that uses the broker pattern: it never reads ambient environment secrets, requests a scoped, short-lived token per tool call, and propagates it with an audience and correlation ID.

“`javascript
// mcp-github-server.js — broker-scoped secret propagation
import { Server } from “@modelcontextprotocol/sdk/server/index.js”;
import { StdioServerTransport } from “@modelcontextprotocol/sdk/server/stdio.js”;
import { CallToolRequestSchema, ListToolsRequestSchema } from “@modelcontextprotocol/sdk/types.js”;

const BROKER_URL = “https://broker.onlinecloud.dev/v1/token”;
const AUDIENCE = “mcp://github.tools”;
const WORKLOAD_TOKEN = process.env.OC_WORKLOAD_TOKEN; // identity only, NOT a secret

// Fetch a short-lived, audience-bound, minimally-scoped token for ONE call.
async function mintScopedToken({ tenantId, sessionId, callId, capability }) {
const res = await fetch(BROKER_URL, {
method: “POST”,
headers: {
“content-type”: “application/json”,
“authorization”: `Bearer ${WORKLOAD_TOKEN}`, // proves the session identity
“x-oc-call-id”: callId, // correlation into audit logs
},
body: JSON.stringify({
tenant_id: tenantId,
session_id: sessionId,
audience: AUDIENCE,
capability, // e.g. “repo:read” — least privilege for THIS call
ttl_seconds: 300, // hard, short-lived
}),
});
if (!res.ok) throw new Error(`broker refused token: ${res.status}`);
const { access_token, expires_at } = await res.json();
return { access_token, expires_at };
}

const server = new Server(
{ name: “github-tools”, version: “1.0.0” },
{ capabilities: { tools: {} } }
);

server.setRequestHandler(ListToolsRequestSchema, async () => ({
tools: [
{
name: “create_issue”,
description: “Create a GitHub issue in the tenant’s repo.”,
inputSchema: {
type: “object”,
properties: { repo: { type: “string” }, title: { type: “string” } },
required: [“repo”, “title”],
},
// Declared capability drives scope minimization server-side.
_meta: { requiredCapability: “repo:write” },
},
],
}));

server.setRequestHandler(CallToolRequestSchema, async (req, extra) => {
const { tenantId, sessionId, callId } = extra?.session ?? extractContext(extra);
const capability = “repo:write”;

// 1) Mint a scoped token for THIS invocation only.
const { access_token } = await mintScopedToken({
tenantId, sessionId, callId, capability,
});

try {
// 2) Use it immediately; never cache, never log the value.
const { repo, title } = req.params.arguments;
const r = await fetch(`https://api.github.com/repos/${repo}/issues`, {
method: “POST”,
headers: {
“authorization”: `Bearer ${access_token}`,
“accept”: “application/vnd.github+json”,
“x-oc-call-id”: callId, // same correlation id as the KMS/vault logs
},
body: JSON.stringify({ title }),
});
if (!r.ok) throw new Error(`github error: ${r.status}`);
const issue = await r.json();
return { content: [{ type: “text”, text: `Created issue #${issue.number}` }] };
} finally {
// 3) Token expires in minutes; we hold no reference to the raw secret.
// The broker’s reverse-proxy mode means we never even saw the PAT.
}
});

function extractContext(extra) {
// Populated by the session sandbox; identity claims, NOT secrets.
return {
tenantId: extra?.requestInfo?.headers?.[“x-oc-tenant”],
sessionId: extra?.requestInfo?.headers?.[“x-oc-session”],
callId: extra?.requestInfo?.headers?.[“x-oc-call-id”] ?? crypto.randomUUID(),
};
}

await server.connect(new StdioServerTransport());
“`

### Why this shape matters

– **No `process.env.GITHUB_TOKEN`.** The PAT never enters the process. The only ambient value is a workload identity token that grants *nothing* on its own.
– **Per-call minting.** Scope and TTL are recomputed for every invocation, so a leaked token is worthless in minutes and useless against another tool’s audience.
– **Correlation by construction.** `callId` rides along into the broker, the upstream API, and (via the broker) the KMS encryption context — closing the reconciliation loop from §4.
– **Fail-closed.** If the broker refuses, the tool errors; there is no fallback to a cached or ambient credential.

—

## Conclusion: Isolation as a Runtime Invariant

Multi-tenant agentic tool chains are only as safe as their weakest shared resource — and by default, that resource is the environment block holding everyone’s secrets. The disciplined answer is to make credential bleeding **structurally impossible**: per-tenant KMS keys with cryptographic encryption contexts, vault namespaces that mirror the tenant boundary, short-lived audience-bound scoped tokens minted per call, a correlation identifier woven through every audit plane, and an ephemeral environment sandbox that never lets two tenants’ secrets share a process.

None of these controls is exotic. Each is a well-understood primitive — envelope encryption, token exchange, structured audit joins, process isolation — applied with the assumption that **the agent is a hostile multi-tenant workload until proven otherwise.** Build for that assumption and your tool chain stays fast; skip it, and the first prompt injection becomes the first credential breach.

—

## Ready to isolate your agent secrets end to end?

OnlineCloud gives your team per-tenant KMS key isolation, broker-minted scoped tokens, correlated audit trails, and ephemeral session sandboxes out of the box — so you can ship agentic tool chains without shipping your customers’ credentials alongside them.

**→ [Start your free security assessment at onlinecloud.dev](https://www.onlinecloud.dev/)**

*Talk to our security architects about a multi-tenant MCP hardening review, and see how credential bleeding becomes a non-issue in your stack.*