Author: sandeepsingh

Scaling MCP Servers with Serverless AWS Lambda and SSE

# Scaling MCP Servers with Serverless AWS Lambda and SSE

*Deep-dive for cloud architects who are tired of babysitting a Kubernetes cluster so a handful of agent tools can talk to the CRM.*

![OnlineCloud — Agentic AI & MCP Engineering Studio](https://www.onlinecloud.dev/assets/hero-company-V8Xps1Ae.png)

—

## 1. Introduction: From Container Fleets to Event-Driven Tools

Most MCP (Model Context Protocol) servers shipped to production last year followed one template: take the SDK of your choice — FastMCP, the TypeScript or Python reference servers, a Go SDK — wrap it in Docker, and park it on ECS or EKS behind an ALB. One container per tool domain, a couple of replicas for “HA,” an autoscaler, and a promise that the cluster will figure itself out.

That template was the only thing the original MCP SDKs supported: a long-lived HTTP process speaking SSE. But it imports every operational cost of the container era into a workload that is fundamentally **event-shaped**. An agent issues a `tools/call`, waits seconds for a JSON-RPC result, and moves on. Between calls your deployment does nothing — yet you pay for idle pods, node pools that must never drop below two, and an EKS control plane that bills $73/month whether you execute one tool call or a million.

The architectural reframe: stop modeling MCP as a *service* and model it as a *stream of events*.

| Container mental model | Serverless mental model |
|—|—|
| Deploy N replicas of each MCP server | One function per transport concern |
| HPA scales pods with minutes of lag | Every event is an isolated invocation |
| Pay for idle capacity 24/7 | Pay only for GB-seconds actually executed |
| Session state lives in process memory | Session state lives in DynamoDB |
| “HA” means running 2+ of everything | HA is the platform’s default posture |

In this model, a client’s MCP session is a durable conversation record, a `tools/call` is a Lambda invocation, and the SSE channel the agent listens on is a thin, replaceable relay. This post walks that architecture end to end: the POST-to-SSE bridge on Lambda URLs with DynamoDB as the backbone, cold-start tactics for first-token latency, concurrency at thousands of parallel calls, a CDK implementation, and an honest ROI comparison against the EKS floor.

—

## 2. Lambda for MCP: The POST-to-SSE Bridge with Lambda URLs and DynamoDB

MCP’s classic HTTP transport has two asymmetric legs: the client holds open a **GET /sse** stream for server-pushed responses and notifications, and it **POSTs** client-to-server messages (`initialize`, `tools/call`) to a messages endpoint. The naive Lambda port — one function owning a session, handling both legs, keeping the SSE stream open in memory — collapses under its own weight: a held-open execution is billed for the whole session, and tool calls that could run in parallel get serialized behind one instance.

The production pattern splits the legs into independent functions joined by a durable bridge:

– **`mcp-session` (SSE relay).** A Lambda Function URL with response streaming answers `GET /sse?sessionId=…`, streams `text/event-stream` frames, emits heartbeats, and **drains** the session’s results from DynamoDB.
– **`mcp-executor` (worker).** A second Function URL serves `POST /messages`: authenticate, parse the JSON-RPC envelope, execute the tool, and **append** the result to the session’s outbox in DynamoDB.

DynamoDB is the bridge. Two tables carry the system: `mcp_sessions` (keyed by `sessionId`, holding negotiated capabilities, an activity flag, and a TTL so orphaned sessions self-destruct) and `mcp_event_outbox` (keyed by `(sessionId, seq)` with a monotonically increasing sort key — the executor writes one item per response, the relay queries forward from its last-read cursor). Because JSON-RPC responses correlate by request `id`, strict ordering across *different* tool calls is unnecessary — which is what makes a horizontally scaled executor safe: any warm instance picks up any POST, executes, and appends, while the single relay streams results in order.

“`
Agent mcp-session (relay) DynamoDB mcp-executor (workers)
│ GET /sse?sessionId=… │ │ │
│◄────────────────────────────┤ │ │
│ POST tools/call ───────────┼──────────────────────┼───────────────────────►│
│ │ │◄── append result ──────┤
│ │ drain cursor ──────►│ │
│◄──── SSE: jsonrpc result ───┤◄─────────────────────┤ │
“`

The relay polls the outbox every 300–500 ms only while the session has an in-flight request — tracked by an activity flag the executor sets with a short TTL — and backs off to heartbeat cadence when idle. That converts a naive always-on poller (~$1.30/month in reads per continuously connected session) into a cost that ticks only while an agent is actually waiting on a tool.

Design around one hard constraint from day one: **Lambda executions cap at 15 minutes**, so a relay holding a stream that long will be terminated mid-session. Treat the relay as a replaceable view, not a stateful process: SSE clients reconnect with `Last-Event-ID`, rejoin the same `sessionId`, pick up the cursor from DynamoDB, and never notice the handoff. The session outlived the relay — that is the entire point of the bridge.

—

## 3. Cold Start Optimization: Managing First-Token Latency

The objection every architect raises here is legitimate: *”My agent is waiting on this tool — I can’t afford a cold start in the critical path.”* Manage it; don’t pray about it. First-token latency decomposes into: TLS to the Function URL (~20–50 ms region-co-located), executor cold start when one occurs, DynamoDB append + relay drain (single-digit ms), and the poll interval. Only the cold start is under your control — and it is smaller than container folklore suggests. A trimmed Node.js or Python executor with the MCP SDK initializes in **200–600 ms** at 512 MB; Go and Rust MCP servers boot in **under 50 ms** (attractive, since an MCP server is a thin JSON-RPC layer); Java 11+ gets **SnapStart**, cutting startup to hundreds of milliseconds.

The structural mitigations matter more than the runtime choice:

– **Keep the executor out of a VPC unless the tools force it.** ENI attachment is the largest cold-start multiplier, historically adding seconds. If tools must reach internal databases, isolate narrowly — RDS Proxy, or one VPC-pinned function behind the hot path.
– **Exploit burst warmth.** Agent loops are self-warming: ten sequential tool calls keep one instance warm for the burst, so only the *first* call of an idle window absorbs startup cost. Warm-up pings are justified only during known business windows; otherwise you are buying capacity for traffic that never comes.
– **Provisioned concurrency is a scalpel, not a default.** Pin it on the relay tier only if always-connected agents need instant reconnect, and accept idle billing there. Never provision the executor — that tier must track demand, not forecasts.
– **Measure the real metric.** Response streaming opens the SSE connection immediately; only the result-bearing frame is latency-sensitive. That also gives you room to stream `notifications/progress` frames during long tools — exactly what a multi-minute Oracle export needs to keep the agent harness from timing out.

Budget honestly: the warm-path overhead of this stack is roughly **50–150 ms** of platform time beyond the tool’s own execution. A cold call adds a few hundred milliseconds once per idle window — a 15% tax on the first call of a burst when the tool itself runs two seconds, and 0% on the rest. Contrast that with a container deployment that scaled to zero: minutes to recover. Serverless recovers from zero in milliseconds; that asymmetry is the whole game.

—

## 4. Concurrency: Thousands of Parallel Tool Calls, Zero Infrastructure

Here the event-driven model stops being merely cheaper and becomes categorically more capable. Containers scale tool execution by scaling *whole replicas*: 2,000 concurrent calls means enough aggregate pod capacity, an autoscaler with 1–2 minutes of reaction lag, and headroom burned on that lag — or over-provisioning.

Lambda scales per invocation. The executor is invoked once per tool call and scheduled onto warm instances up to your account’s soft concurrency limit (1,000 concurrent executions per region by default, with 500–3,000 executions/minute of burst headroom depending on region — both raiseable by ticket in hours, and per-function caps are manageable via reserved concurrency). The ceiling stops being “how many pods can the cluster schedule?” and becomes “how many requests can the downstream API tolerate?” — the right problem to have.

Two properties fall out of the split design:

– **Relay and worker scale independently.** A thousand agents connected but idle cost a thousand tiny parked relays at 128 MB (~$0.0075/hour each, and sessions typically last minutes, not days). A thousand agents *actively calling tools* cost a thousand executor invocations that finish in seconds and release capacity. You never provision for the product of both.
– **Parallel tool calls are free.** Three independent `tools/call` requests in one planning turn become three invocations on whatever instances are warm — the executor acts as a shared elastic pool across every session in the account.

The failure modes move from infrastructure to application: enforce **concurrency budgets** per downstream system (your CRM’s rate limits are the real bottleneck — use per-tool throttles), carry **idempotency keys** on mutating tools (retries are cheap now, so make them safe), and add **dead-letter handling** on the outbox so a poisoned event can’t stall a session’s stream. These are the disciplines of any event-driven backend; MCP is no longer special.

—

## 5. Implementation: Serverless MCP Infrastructure in AWS CDK

A minimal but production-shaped CDK stack (TypeScript): two functions behind two Function URLs, DynamoDB as session store and outbox, TTL cleanup, pay-per-request billing.

“`typescript
import * as cdk from “aws-cdk-lib”;
import { Construct } from “constructs”;
import * as dynamodb from “aws-cdk-lib/aws-dynamodb”;
import * as lambda from “aws-cdk-lib/aws-lambda”;
import * as nodejs from “aws-cdk-lib/aws-lambda-nodejs”;

export class McpServerlessStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);

// Session registry + event outbox (the “bridge”)
const sessions = new dynamodb.Table(this, “McpSessions”, {
partitionKey: { name: “sessionId”, type: dynamodb.AttributeType.STRING },
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
timeToLiveAttribute: “ttl”,
});
const outbox = new dynamodb.Table(this, “McpEventOutbox”, {
partitionKey: { name: “sessionId”, type: dynamodb.AttributeType.STRING },
sortKey: { name: “seq”, type: dynamodb.AttributeType.NUMBER },
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
timeToLiveAttribute: “ttl”,
});

// Leg 1: SSE relay — GET /sse, streams events, drains the outbox
const relay = new nodejs.NodejsFunction(this, “McpSseRelay”, {
entry: “lambdas/relay.ts”,
memorySize: 128,
timeout: cdk.Duration.minutes(15),
environment: {
SESSIONS_TABLE: sessions.tableName,
OUTBOX_TABLE: outbox.tableName,
},
});
outbox.grantReadData(relay);
sessions.grantReadWriteData(relay);

const relayUrl = relay.addFunctionUrl({
authType: lambda.FunctionUrlAuthType.NONE, // front with CloudFront/JWT in prod
invokeMode: lambda.InvokeMode.RESPONSE_STREAM, // SSE needs streaming
});

// Leg 2: executor — POST /messages, runs the tool, appends the result
const executor = new nodejs.NodejsFunction(this, “McpExecutor”, {
entry: “lambdas/executor.ts”,
memorySize: 512,
timeout: cdk.Duration.minutes(5),
environment: {
SESSIONS_TABLE: sessions.tableName,
OUTBOX_TABLE: outbox.tableName,
},
});
outbox.grantWriteData(executor);
sessions.grantReadWriteData(executor);

const executorUrl = executor.addFunctionUrl({
authType: lambda.FunctionUrlAuthType.NONE, // enforce auth before this
cors: { allowedOrigins: [“*”], allowedMethods: [lambda.HttpMethod.POST] },
});

new cdk.CfnOutput(this, “RelayUrl”, { value: relayUrl.url });
new cdk.CfnOutput(this, “ExecutorUrl”, { value: executorUrl.url });
}
}
“`

The executor’s hot path is a registry lookup plus an atomic outbox append:

“`typescript
const seq = await outbox.updateItem({ // atomic next-seq
TableName: OUTBOX_TABLE,
Key: { sessionId: { S: sessionId } },
UpdateExpression: “ADD seq :one”,
ExpressionAttributeValues: { “:one”: { N: “1” } },
ReturnValues: “UPDATED_NEW”,
});
await outbox.putItem({
TableName: OUTBOX_TABLE,
Item: {
sessionId: { S: sessionId },
seq: seq.Attributes!.seq,
event: { S: JSON.stringify({ jsonrpc: “2.0”, id, result }) },
ttl: { N: String(Math.floor(Date.now() / 1000) + 300) },
},
});
“`

The relay opens the streamed response and loops: `Query` the outbox from its cursor, write frames as SSE `data:` lines, emit a `: ping` comment every 15 seconds, and exit cleanly when the session completes or the 15-minute ceiling approaches — the client’s SSE reconnect with `Last-Event-ID` rejoins the session from DynamoDB. Streamed responses cap at 20 MB; anything larger should return a presigned reference anyway. Two production notes the skeleton omits: Function URLs support only `NONE` or `AWS_IAM` auth, so put a CloudFront or API Gateway JWT authorizer in front of the executor; and use the `@modelcontextprotocol/sdk` server classes inside the executor rather than hand-rolling JSON-RPC.

—

## 6. ROI: EKS Node Costs vs. Lambda Execution Costs for Spiky AI Workloads

The ROI argument is not “serverless is cheaper than containers” — that claim is false in general, and be suspicious of anyone making it flatly. The correct claim is about **cost shape**. MCP tool traffic is bursty: business-hours peaks, nightly batch jobs, campaign spikes, with peak-to-average ratios routinely above 10:1. That is precisely the regime where a fixed infrastructure floor loses.

A minimum viable HA MCP estate on EKS (us-east-1, on-demand, three `m6i.large` across AZs so the HA story is real):

| Item | Monthly cost |
|—|—|
| EKS control plane ($0.10/hr × 730) | $73 |
| 3 × `m6i.large` ($0.0965/hr × 730) | ~$211 |
| ALB + NAT gateway + egress (approx.) | ~$60–90 |
| **Floor before a single tool call executes** | **~$350–375** |

That floor is paid on nights, weekends, and holidays whether the estate executes five calls or five million. The same workload on Lambda, assuming 512 MB average memory and ~1.2 s average execution per call (generous for CRUD-style tools, which usually finish in hundreds of milliseconds):

| Monthly tool calls | Lambda compute + requests |
|—|—|
| 0 (quiet month) | **$0.00** |
| 1,000,000 | ~$10.20 (600k GB-s × $0.0000167 + $0.20/M requests) |
| 10,000,000 | ~$102 |
| Crossover vs. the EKS floor | ~20–35M calls/month |

The crossover sits at 20–35 million calls per month — beyond most enterprises’ volume, and if reached, reached in spikes that container autoscaling would choke on anyway. Hidden costs widen the gap further: cluster upgrades, node patching, autoscaler tuning, per-service version sprawl, and the engineering hours spent explaining why an agent timed out because a node drained at the wrong moment.

The honest caveats cut the other way, and an architect should state them. If traffic is genuinely flat — a 24/7 baseline of millions of calls with no seasonality — a Savings Plan on ECS or EKS beats Lambda on unit economics. Don’t hold thousands of SSE connections open indefinitely on Lambda; reap idle sessions and let reconnect rebuild them. And downstream data costs are identical in both architectures — exclude them from the comparison. Choose based on your actual traffic distribution, not ideology.

For spiky AI workloads — agent fleets, RAG backfills, support-bot surges — the decision is rarely close. Lambda turns a $350/month always-on floor into a cost curve that starts at zero, tracks demand to the cent, and reaches thousands of concurrent calls without a single infrastructure ticket. Your workload is event-shaped. Stop forcing it into a pod.

—

## Conclusion

Container-based MCP servers were the correct first move — they let teams adopt the protocol with the SDKs as written. But they import an always-on cost and operations model into a naturally event-driven workload. Splitting the transport into an SSE relay and a stateless executor, bridged by DynamoDB behind Lambda Function URLs, yields a system that recovers from zero in milliseconds, scales to thousands of concurrent tool calls without provisioning, and bills only for what executes. The cold-start tax is a few hundred milliseconds on the first call of a burst; the EKS tax is hundreds of dollars every month even when nothing runs.

Designing this split well — session lifecycle across the 15-minute relay ceiling, outbox polling that tracks activity, honest concurrency budgets for downstream systems, an ROI model that matches your real traffic shape — is exactly the architecture work that looks simple on a whiteboard and gets subtle in production. That is what OnlineCloud does all day: MCP-native architectures, audited end to end, built by senior US-based engineers and shipped in weeks, not quarters. If you are moving your MCP tool surface off containers — or designing a multi-agent estate that must scale past what your cluster tolerates — **talk to the OnlineCloud architects** at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or [onlinecloud.dev](https://www.onlinecloud.dev/). We would rather argue about your cost curve than sell you a deck.

September 6, 2026
Building a Low-Code MCP Dashboard with React and Tailwind
# Building a Low-Code MCP Dashboard with React and Tailwind

*How to turn a stream of MCP JSON-RPC notifications into a real-time, human-checkpointed control room — without leaving the React ecosystem.*

![OnlineCloud AI engineering solutions](https://www.onlinecloud.dev/assets/hero-company-V8Xps1Ae.png)

## 1. Why Your Agent Needs a Dashboard It Never Asked For

The first time you watch a production agent run unattended, you’ll notice something uncomfortable: it’s a black box with a pulse. It plans, calls tools, mutates rows in your ERP, and reports back in terse JSON that no human reads until something breaks. That’s the fundamental trust problem with agentic systems — not model quality, but *observability of intent*. A plan/act/reflect loop that touches a production database needs a human who can answer one question at any moment: *what is the agent doing right now, and should it be allowed to finish?*

This is the human-in-the-loop gap. Model Context Protocol (MCP) gives you typed tools, declarative resources, and auditable transports, but it doesn’t hand you a UI. The transport is a contract, not a cockpit. If you’re running MCP servers in front of Oracle, MS SQL, Snowflake, or a CRM, an autonomous loop that can issue writes deserves a control surface between the LLM and your business logic — a surface that visualizes each tool call as it happens, makes the agent’s decision tree legible, and inserts a hard checkpoint before high-impact mutations.

Think about what actually goes wrong in production. An agent tasked with reconciling invoices chains five tools: a read query, a fuzzy match, a write to the accounting system, a notification to finance, and a retry path when a row is locked. Each step is individually reasonable; the *combination* is where risk hides. Without a live view, your only signals are a completed run or a timeout — you find out an hour later that the agent emailed 40 customers with a miscomputed total because a JOIN dropped a filter. A dashboard doesn’t prevent the bad idea; it compresses the time between the bad idea and a human seeing it, from hours to seconds. That compression is the entire value proposition of human-in-the-loop tooling.

The fastest way to build that surface, without dragging in a heavy BI stack, is a single-page dashboard: React for state and component modeling, Tailwind for the visual layer, and a thin adapter that bridges MCP’s streamable transports into React state. Low-code here doesn’t mean “no code.” It means the *dashboard* logic stays small enough that a team can ship it in a sprint, because the hard protocol work is already done by MCP. You’re not building an agent platform; you’re building a window onto one.

## 2. Dashboard Architecture: From MCP Transport to React State

MCP servers expose tools over two mainstream transports: the original SSE-based transport and the newer streamable HTTP transport. From a dashboard’s perspective, both collapse into the same shape: a client-to-server JSON-RPC channel for requests, and a server-to-client event stream for notifications. The notifications you care about are the ones that make agent behavior visible: `notifications/progress` for tool execution progress, tool call lifecycle messages, and server requests that need human input (which MCP models as a `sampling/createMessage` request).

The architectural rule that keeps this dashboard “low-code” is **never let a UI component talk to the wire**. Every transport event funnels through one place — a reducer that normalizes JSON-RPC into a flat, indexable domain model. Everything downstream (progress bars, decision trees, modals, the inspector) is a pure function of that state. That single choke point is also where you attach audit logging, which matters when the dashboard is your compliance story for “who approved this tool call.”

“`tsx
// bridge.ts — normalize raw MCP notifications into app events.
// One adapter per transport; the reducer never sees JSON-RPC.
export type McpEvent =
| { kind: “tool:started”; callId: string; tool: string;
parentCallId: string | null; args: unknown }
| { kind: “progress”; callId: string; pct: number | null; message?: string }
| { kind: “tool:finished”; callId: string; result: unknown }
| { kind: “checkpoint:requested”; callId: string; tool: string };

export function fromJsonRpc(msg: any): McpEvent | null {
if (msg.method === “notifications/progress”) {
const p = msg.params;
return {
kind: “progress”,
callId: p.progressToken, // our correlation id
pct: p.total ? Math.round((p.progress / p.total) * 100) : null,
message: p.message,
};
}
// … map tool:started / finished / sampling requests the same way
return null;
}
“`

With an `EventSource` (SSE) or a fetch-based reader (streamable HTTP) feeding `dispatch(fromJsonRpc(raw))`, React becomes a pure projection of the agent’s runtime state — the same trick that makes Redux DevTools useful, applied to your agent. A `parentCallId` field is the seed of your decision tree: because MCP tool calls nest (a plan tool calls a query tool), retaining the parent link on every event lets you render the agent’s reasoning structure for free.

## 3. Real-Time UI: Progress Bars and Decision Trees in Tailwind

Once events are normalized, the UI is almost mechanical. Tool-call progress is a per-call percentage plus a status enum — `queued`, `running`, `awaiting-approval`, `done`, `error`. A progress bar in Tailwind is a borderless rounded track with a gradient fill whose width is driven by a style binding, animated purely by `transition-[width]`. The status enum drives color: emerald for healthy execution, amber when the tool is parked at a human checkpoint, rose for failures.

“`tsx
// ToolProgress.tsx — one running tool call
export function ToolProgress({ tool }: { tool: ToolState }) {
return (
{tool.tool}

{tool.pct === null ? “running…” : `${tool.pct}%`}

);
}
“`

The decision tree is where a dashboard earns its keep. A flat log of fifty tool calls is unreadable; a tree that shows the plan spawning lookups, which spawn a write, which now waits on a human — that’s a story. Because every call carries `parentCallId`, rendering the tree is a recursive component: each node is a status dot plus a mono-spaced tool name, with children indented under a hairline left border. Recursion depth rarely exceeds four or five levels, so even naive recursion is fine.

“`tsx
// DecisionTree.tsx — recursive render of nested tool calls
export function DecisionTree({ nodes, rootId }: { nodes: Map; rootId: string }) {
const root = nodes.get(rootId);
if (!root) return null;
const children = […nodes.values()].filter((n) => n.parentCallId === rootId);
return (

{root.tool}
{latencyMs(root)}

{children.map((c) => )}

);
}
“`

On a dark control-room palette — slate panels, one accent gradient, mono for tool names — this reads like an observability product in an afternoon. Tailwind’s utility model keeps the whole UI in one file of small components rather than a design-system project. Two details matter for polish. First, keep bars *indeterminate* (a sliding pulse) when a call reports no total; a static 4% bar for a twenty-minute export reads as a hang. Second, animate tree edges only when a node *enters* — flashing the whole subtree on every status change is visual noise. Small motion rules like these are what separate a dashboard operators trust from one they ignore.

## 4. User Interaction: Human Checkpoints for High-Impact Tools

Visualization is half the loop; the other half is the ability to *stop*. In MCP terms, when an agent wants to call something risky, the server issues a `sampling/createMessage` request back to the client — the protocol’s native human-input channel. Your dashboard answers that request with a modal instead of letting the client auto-reply. That turns MCP’s sampling hook into a proper approval gate: the tool call parks in `awaiting-approval`, the decision tree goes amber, and a modal renders the args, the call path that led here, and Approve / Reject actions.

The trick is to make approval *policy-driven*, not ad hoc. A small declarative map decides which tools trigger a checkpoint: `create_refund` and `send_email` confirm; `read_only_query` auto-runs; destructive ERP calls confirm with a typed acknowledgment. Keeping policy in one object means the dashboard stays low-code while the blast-radius logic lives somewhere an architect can review it — a stance OnlineCloud takes seriously when wiring MCP servers to real databases with real auth boundaries.

“`tsx
// CheckpointModal.tsx — rendered when a call parks at a human gate
export function CheckpointModal({ pending, onApprove, onReject }:
{ pending: ToolState; onApprove: () => void; onReject: () => void }) {
return (
Approval required

The agent wants to call {pending.tool}
{” “}with the arguments below. Review before it proceeds.

{JSON.stringify(pending.args, null, 2)}
);
}
“`

Every decision — approve, reject, or the timeout that defaults to reject — should append an audit record to the same event log the dashboard renders. Now your “human checkpoints” feature doubles as an audit trail, which is precisely the difference between an agent demo and a system a compliance team signs off on. A useful pattern is to log the *reason* an operator gives when rejecting — a required text field on the reject path — so the agent’s error-correcting loop has something to learn from, and auditors have a record that the rejection was deliberate rather than a misclick.

## 5. Code Walkthrough: The MCP Tool Inspector

The centerpiece of a dashboard for engineers is a **Tool Inspector**: pick a server, browse its declared tools, inspect schemas, fire a call with custom arguments, and watch the live result — including the decision-tree context around it. It’s the MCP equivalent of a database query tool, and it’s where all the earlier pieces converge.

State lives in three places: the tool list (fetched once from the server’s `tools/list`), the invocation log (a `useReducer` over the event bridge — the same reducer powering the tree), and UI micro-state for the active tool, the argument draft, and the selected call. The schema panel needs no custom form generator: MCP tool inputs are JSON Schema, so render required fields, types, and descriptions from `inputSchema`, and let engineers edit raw arguments in a textarea with live JSON validation. That’s what keeps it low-code — you’re *reflecting* the server’s metadata, not maintaining a parallel form definition.

“`tsx
// ToolInspector.tsx — the working core
export function ToolInspector({ serverUrl }: { serverUrl: string }) {
const [tools, setTools] = useState([]);
const [active, setActive] = useState(null);
const [argsDraft, setArgsDraft] = useState(“{}”);
const [argsError, setArgsError] = useState(null);
const [log, dispatch] = useReducer(dashboardReducer, {});
const [selected, setSelected] = useState(null);

useEffect(() => {
fetch(`${serverUrl}/tools/list`).then((r) => r.json())
.then((res) => setTools(res.tools));
}, [serverUrl]);

const invoke = useCallback(async () => {
if (!active) return;
let parsed: unknown;
try { parsed = JSON.parse(argsDraft); }
catch { setArgsError(“Arguments are not valid JSON”); return; }
// Ask the server to emit progress under a token we control,
// then correlate incoming notifications with this invocation.
const token = crypto.randomUUID();
dispatch({ kind: “tool:started”, callId: token, tool: active.name,
parentCallId: null, args: parsed });
setSelected(token);
const res = await callTool(serverUrl, active.name, parsed, token);
dispatch(res.ok
? { kind: “tool:finished”, callId: token, result: res.value }
: { kind: “tool:failed”, callId: token, error: res.value });
}, [active, argsDraft, serverUrl]);

const activeCall = selected ? log[selected] : null;
return (

{/* Column 1: server tools */}

{tools.map((t) => (

))}

{/* Column 2: schema + raw args editor */}

{active && (
<>

setArgsDraft(e.target.value)} spellCheck={false} className=”h-64 flex-1 resize-none rounded-lg border border-slate-800 bg-slate-900 p-3 font-mono text-xs text-slate-200″ /> {argsError && {argsError} } <button onClick={invoke} className="rounded-lg bg-emerald-500 px-4 py-2 text-sm font-medium text-slate-950 hover:bg-emerald-400"> Invoke tool </button> </> )} </main> {/* Column 3: live call + decision tree */} <section className="overflow-y-auto rounded-xl border border-slate-800 bg-slate-900/50 p-4"> {activeCall ? <ActiveCallView call={activeCall} /> : Invoke a tool to inspect its run… } </section> </div> ); } “` Note what’s deliberate here: `useCallback` on `invoke` so the button never re-renders on log churn, `useReducer` so progress notifications trigger narrow updates instead of whole-tree re-renders, and a correlation token that lets one inspector session drive many concurrent calls. The same component, pointed at a different MCP server URL, inspects a database connector, a CRM wrapper, or a payment tool — that reusability is the payoff of building on the protocol instead of per-server glue. ## 6. Performance: Surviving Large-Scale Decision Logs A long-running agent session can emit thousands of events. Progress notifications are the worst offender — an LLM streaming through a file export can push dozens of updates per second, and if each one re-renders the whole decision tree, your “real-time” dashboard becomes a slideshow. Before reaching for tools, set a latency budget: the dashboard should reflect a state change within 100ms of the transport delivering it, and it should hold 60fps while doing so. Everything below is aimed at that budget. The fixes, in order of impact: **Coalesce high-frequency events.** Progress is ephemeral — only the latest percentage per call matters. Instead of dispatching every notification straight into the reducer, buffer them in a ref and flush once per animation frame. You trade a few milliseconds of displayed staleness for a hard cap on renders per second. “`ts // rAF coalescing: at most one state flush per frame, no matter the spam const buffer = useRef(new Map<string, number>()); useEffect(() => { let raf = 0; const flush = () => { if (buffer.current.size) { buffer.current.forEach((pct, callId) => dispatch({ kind: “progress”, callId, pct })); buffer.current.clear(); } raf = 0; }; return () => cancelAnimationFrame(raf); }, []); // On each SSE notification: buffer.current.set(callId, pct); raf ||= requestAnimationFrame(flush); “` **Memoize leaf components and virtualize the log.** Wrap the log row in `memo` so an update to one call only re-renders that row. Then cap the visible DOM: with thousands of completed calls, render only the viewport window with `react-window`’s `FixedSizeList`. Virtualization turns a 5,000-row log into ~20 mounted rows, and the decision tree’s depth naturally bounds its cost. One caveat: keep the visible row height constant (hence `FixedSizeList`, not the variable-size variant) — monospaced, single-line rows make that trivial and measurable. “`tsx const LogRow = memo(function LogRow({ call }: { call: ToolState }) { return ( <div className="flex items-center gap-3 border-b border-slate-800/60 py-1.5 font-mono text-xs"> {call.tool} {latencyMs(call)} </div> ); }); “` **Keep the store flat and indexed.** Store calls in a `Record<callId, ToolState>` keyed map, never an array you filter linearly. Index children by parent at insert time if you render trees frequently, and derive the tree inside a `useMemo` keyed on the log version. Finally, put a ring-buffer cap on completed calls — keep the last 2,000 and aggregate the rest into counters. No human reads 20,000 rows of history live; the dashboard’s job is the present, and the audit log’s job is the past. Follow the same discipline OnlineCloud applies to production MCP systems: measure latency of the dashboard itself ($/request and render time are both line items), keep a rollback path, and let the eval — in this case, “can an operator understand and stop the agent in under five seconds” — define done. ## Ship the Cockpit, Not Just the Agent The pattern is portable: a normalizing bridge, a reducer that turns MCP notifications into a flat state model, Tailwind components that project that state into progress bars and decision trees, and a policy map that decides when a human must step in. That’s a production-grade human-in-the-loop surface with maybe a few hundred lines of React — because the protocol did the heavy lifting. When you’re ready to go deeper — checkpoint policies wired to your actual ERP schema, audit-grade approval logs, streamable HTTP tuned for low-latency long-running calls — that’s exactly the territory OnlineCloud works in. We’re a US-based engineering studio building MCP-native, evaluated, observable systems that plug into Oracle, MS SQL, your CRM, and the tools your team uses daily. Senior engineers only, no offshore handoffs, production in 6–10 weeks. **[Get in touch →](mailto:contact@onlinecloud.dev)** — free initial consultation, response within 24 hours. Or see more at **[onlinecloud.dev](https://www.onlinecloud.dev/)**. </div> <div style="margin-top:var(--wp--preset--spacing--40)" class="wp-block-post-date has-small-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/building-a-low-code-mcp-dashboard-with-react-and-tailwind/"><time datetime="2026-09-06T13:19:45+00:00">September 6, 2026</time></a></div> </div> </li><li class="wp-block-post post-251 post type-post status-publish format-standard hentry category-ai-agents"> <div class="wp-block-group alignfull has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--60)"> <h2 class="wp-block-post-title has-x-large-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/designing-a-cost-aware-router-for-model-cascades-bedrock-vs-self-hosted/" target="_self" >Designing a Cost-Aware Router for Model Cascades (Bedrock vs. Self-Hosted)</a></h2> <div class="entry-content alignfull wp-block-post-content has-medium-font-size has-global-padding is-layout-constrained wp-block-post-content-is-layout-constrained"># Designing a Cost-Aware Router for Model Cascades: Bedrock vs. Self-Hosted *A FinOps Architect’s guide to paying for reasoning only where reasoning is required.* ![OnlineCloud — cost-aware model routing](https://www.onlinecloud.dev/assets/hero-company-V8Xps1Ae.png) — ## 1. Introduction: The Economics of Frontier Reasoning vs. Steady-State Inference Every LLM workload is really two workloads wearing the same API shape. There is the **reasoning tail** — the 10–20% of requests that genuinely need a frontier model to think: multi-step planning, adversarial code review, unstructured “figure out what I’m asking” prompts. And there is the **steady-state bulk** — the 80–90% of production traffic that is structurally repetitive: extraction from a known schema, classification, summarization of a fixed document type, routing a support ticket, rewriting a paragraph. The gap is not semantic; it is **pricing architecture**. Frontier providers meter reasoning separately from output: every chain-of-thought token the model “thinks” is billed, putting reasoning-class models at $15–$75 per million output tokens plus a surcharge. Meanwhile, a self-hosted 8B–70B model on amortized GPU capacity delivers $0.10–$0.50 per million tokens — a **10–30× spread** — with sub-second time-to-first-token and data that never leaves your VPC. The FinOps failure mode is treating these as one pool. Route every request to the frontier model because “it’s the best model” and you convert a 10–20% reasoning need into a 100% reasoning bill: the line item balloons, p95 latency climbs with queueing, and — counterintuitively — quality doesn’t improve. Frontier models are not measurably better at schema-constrained extraction than a tuned 8B model; they are just more expensive at it. A model **cascade** fixes this. Put a router in front of the model tier, classify each prompt by complexity, and send the bulk to self-hosted inference on EKS while the reasoning tail goes to Bedrock. Done well, a cascade cuts GenAI inference spend by 60–75% at a bounded, measurable accuracy cost — typically under one point on your eval set. This post walks through the router pattern, the decision heuristics that make it trustworthy, the Go architecture we run in production, and a worked ROI model at 50k requests/day. — ## 2. The Router Pattern: Classifying Prompts by Complexity and Routing by Cost/Latency A cascade router is an intercepting layer between your application and your model endpoints. Every request arrives at one ingress and is dispatched to one of two tiers: | Tier | Backend | Cost profile | Latency profile | Best for | |—|—|—|—|—| | **Frontier** | Bedrock (Opus/Sonnet class) | $3–$15+ per M tokens | 2–10s+ (reasoning) | Novel, ambiguous, multi-step reasoning | | **Self-hosted** | EKS + vLLM/TGI (8B–70B) | ~$0.10–$0.50 per M tokens, amortized | 150–800ms | Extraction, classification, known-shape transforms | The router performs three jobs, in order: 1. **Policy override.** Explicit client intent wins: an `X-Route: frontier` header, a `requires_reasoning: true` field, or a retry-after-failure escalation must always reach the frontier tier. The router is a cost optimizer, not a quality gate that can be silently skipped. 2. **Classification.** A cheap scoring step decides which tier the request belongs to (Section 3). It must cost under ~1% of the request’s downstream price — easy, since it’s a few milliseconds of CPU/GPU on a tiny model, not another LLM call. 3. **Dispatch with failover.** The chosen backend handles the request; on timeout, 5xx, or low-confidence output, the router escalates to the other tier exactly once and records the escalation as a metric. Two properties make the pattern production-grade rather than a demo: **Cascading is asymmetric.** Sending a request to the frontier model is always allowed and always safe; serving it from the self-hosted tier is only safe when you can bound the classifier’s error. Hence the design principle: *when in doubt, escalate.* The router’s default on uncertainty is the frontier tier, and savings come from making the *confident* cheap calls fast and correct. **Cost is a routing dimension, not just a bill.** Tier choice sets both price and latency, so routing is a two-objective optimization with a constraint: per-request accuracy stays above the product’s floor. You are not minimizing cost; you are minimizing cost *subject to an accuracy budget* — and everything downstream (thresholds, heuristics, dashboards) exists to keep that constraint observable and enforceable. — ## 3. Decision Heuristics: Logprob Variance and Classification The router’s entire value rests on one question: *can I cheaply predict that the self-hosted model will answer acceptably before I let it answer?* Two families of heuristics answer it, and production routers use both in sequence. ### 3.1 Logprob variance (a cheap “does the model know?” probe) When a model is confident, its sampled outputs converge; when it is guessing, they diverge. You can exploit this without touching the frontier tier: sample **K = 4 generations at low temperature** from the *self-hosted* model (nearly free at $0.10–$0.30/M) and measure agreement across the samples. The cleanest signal is **semantic entropy**: cluster the K outputs by normalized semantic similarity (or exact-match on forced-extracted answers) and compute the variance of the model’s token logprobs around the majority answer. Low variance + high agreement → the model has the answer → serve it. High variance → the model is improvising → escalate. “`text logprob_variance = E[ (log p(token_i) – mean_logprob)^2 ] agreement = (K – minority_count) / K route = self-hosted if logprob_variance < τ_var AND agreement > 0.75 route = frontier otherwise “` The probe is **self-calibrating per request** — no labeled data, because it measures the model’s own epistemic state. In practice it catches the classic failure modes: adversarial phrasings, out-of-vocabulary entity names, ambiguous instructions with two plausible readings. Its cost is 3 extra cheap generations (30–90ms of added latency), amortized against a 10–30× price differential. ### 3.2 Learned classification Probing multiplies cheap inference by K; for the 60–70% of traffic that is *obviously* steady-state — “extract the invoice number,” “classify this ticket,” “summarize this changelog” — you want a microsecond decision instead. That is the **small classifier** run before any generation: – **Features:** prompt length, reasoning verbs (“prove,” “compare,” “design,” “why”), number of explicit constraints, entity density, structured-output requests, instruction complexity, domain markers. – **Model:** a fine-tuned DeBERTa-v3-small (or gradient-boosted model over the feature vector) trained on 10–20k labeled exemplars from your traffic mix — single-digit milliseconds on CPU, emitting a score in [0, 1] plus calibrated confidence. – **Decision:** below a low threshold → confidently cheap → self-hosted, skip the probe. Above a high threshold → reasoning needed → frontier. In the middle (the *uncertainty region*) → run the logprob probe, or default to frontier if latency-bound. The two-stage design keeps the router **graceful under distribution shift**: when traffic drifts (a new product ships, a new template lands), the classifier’s scores shift and the probe stage absorbs the uncertainty instead of silently misrouting. You detect the drift on the dashboard (Section 6) and retrain. ### 3.3 Calibration and the accuracy budget None of this works without a **held-out eval set** — a golden set of 1–3k prompts with reference answers, run nightly against both tiers. You tune τ_var and the classifier thresholds on a single axis: *the measured accuracy delta between the cascade and the all-frontier baseline*, held under a committed budget (e.g., Δ ≤ 0.5–1.0 points). Thresholds are not tuned to maximize savings; they are tuned to spend the accuracy budget, and savings is whatever falls out. That inversion of priorities is what keeps a cascade from becoming a silent quality regression users notice three weeks later. — ## 4. Architecture: A Go Router in Front of Bedrock and EKS Go is the right substrate here: it is the lingua franca of ingress proxies (the router *is* a proxy), it streams HTTP without the GC pauses that break SSE, and it deploys as a single static binary on EKS beside the inference tier. “` ┌─────────────────────────────────────────┐ │ Go Cascade Router │ App traffic ───────► │ policy override → classifier → probe │ │ dispatch · retry · escalate · metrics │ └───────────────┬───────────┬─────────────┘ │ │ ┌───────────────▼──┐ ┌────▼──────────────┐ │ EKS (self-hosted) │ │ Bedrock (frontier)│ │ vLLM · 8B-70B │ │ Opus/Sonnet class │ └───────────────────┘ └───────────────────┘ “` The router is a plain `net/http` server whose decision core is deliberately small and pure — unit-testable without network: “`go type Route string const ( RouteSelfHosted Route = “eks” RouteFrontier Route = “bedrock” ) type Router struct { classifier *Classifier // feature-based scorer prober *ConsistencyProber // logprob variance probe cfg Config // thresholds, timeouts, retries } // Decide returns the tier for a request. It never returns an error // for an ambiguous request — ambiguity escalates to frontier. func (r *Router) Decide(ctx context.Context, req *Request) Route { // 1. Policy override — explicit intent always wins. if req.Header.Get(“X-Route”) == “frontier” { return RouteFrontier } // 2. Cheap classifier first. score, conf := r.classifier.Score(req.Messages) if conf >= r.cfg.Confident && score <= r.cfg.SelfHostedThreshold { return RouteSelfHosted } if conf >= r.cfg.Confident && score >= r.cfg.FrontierThreshold { return RouteFrontier } // 3. Uncertainty band — probe the cheap model’s own confidence. if variance := r.prober.LogprobVariance(ctx, req, 4); variance < r.cfg.VarianceThreshold { return RouteSelfHosted } return RouteFrontier // when in doubt, escalate } ``` The dispatch layer wraps that decision in the concerns that keep the system alive: - **Streaming.** Both tiers are proxied with `http.Flusher` passthrough. The classifier runs *before* streaming begins, so the added latency is one ~5ms round-trip, not a serialized LLM call. - **Failover.** Every upstream call gets a deadline (frontier: 120s, self-hosted: 30s). On timeout, connection error, or a 429/5xx burst, the router retries once on the *other* tier and stamps `x-cascade-escalated: true` so callers and dashboards both see it. - **Bedrock specifics.** Calls use Bedrock's streaming invoke API with IAM SigV4 signing and `contentBlockDelta` decoding; the EKS tier is a vLLM `OpenAI-compatible /v1/chat/completions` endpoint on an internal service. One shared request/response schema keeps the router thin. - **Deployability.** The router ships as its own Deployment behind the service mesh, scales on CPU (the classifier is the only compute), and exposes `/-/healthz` plus a `/-/routes` endpoint dumping live threshold state for on-call engineers. The highest-leverage operational detail is **making the decision observable on the request itself**: every response carries `x-cascade-route`, `x-cascade-score`, and `x-cascade-escalated`, so application teams can build product behavior on top of routing (forcing frontier for a user segment, surfacing "answered by fast model" badges) without touching router code. --- ## 5. ROI Calculation: Monthly Savings at 50k Requests/Day A worked model with stated assumptions, so you can re-run it with your own rates: **Assumptions** - Traffic: 50,000 requests/day, average 2,000 input tokens and 500 output tokens each. - Frontier (Bedrock, Sonnet-class blended): **$3/M input, $15/M output**. - Self-hosted: 8B model on vLLM, fleet of 6× A10G at **$0.60/GPU-hr effective** (savings plans/spot), all-in for the bulk tier. - Cascade accuracy budget: Δ ≤ 0.01 on the golden eval set. **Baseline — everything to the frontier model:** - Input: 100M tokens/day × $3/M = $300/day · Output: 25M tokens/day × $15/M = $375/day - **API spend ≈ $675/day ≈ $20,250/month** — with no GPU cost at all, because there is no self-hosted tier. **Cascade — 85% confidently self-hosted, 15% frontier (probes and escalations absorbed):** - Frontier tier: 7,500 req/day → 15M input ($45) + 3.75M output ($56) ≈ $101/day ≈ **$3,030/month** - Self-hosted tier: 42,500 req/day on the 6× A10G fleet → 6 × $0.60 × 24h ≈ **$86/day ≈ $2,590/month** (classifier, probes, router CPU included) - **Cascade total ≈ $5,620/month ≈ 72% below baseline → ~$14,630/month saved.** | Scenario | Monthly cost | vs. all-frontier | |---|---|---| | All traffic → frontier | ~$20,250 | — | | Naive 50/50 static split | ~$15,300 (half frontier spend + larger GPU fleet) | −24% | | **Classifier-driven cascade (85/15)** | **~$5,620** | **−72%** | Two findings worth internalizing. First, **a naive static split is a false economy**: 50% of traffic to each tier still burns ~$10k/month on frontier tokens *and* forces GPU over-provisioning for requests the classifier would have kept at home — the savings live in the *routing decision*, not the split ratio. Second, **the GPU fleet is the floor, not the ceiling**: below ~15–20k self-hosted requests/day, fixed fleet cost dominates and a pure-API cascade (small model on Bedrock's cheap tier, no EKS) can win on paper. Model the fixed floor against per-token savings, then decide whether EKS earns its place. Annualized, that is **~$175k/year** at 50k req/day — the router's dev cost (a few engineer-weeks plus the classifier training set) pays back in its first month, and the nightly eval keeps the accuracy budget honest so savings arrive without waiting for user complaints. --- ## 6. Monitoring: Cascade Hit-Rates and Accuracy Trade-offs A cascade you can't observe is a cost leak with a latency SLA. The stack has one north-star metric and three supporting views, all exported as Prometheus metrics from the Go router and rendered in Grafana, with cost data reconciled from CloudWatch/Bedrock usage and EKS node bills. **North star — accuracy delta (the constraint).** Every night a batch job replays the 2k-prompt golden set against (a) the frontier tier and (b) the live cascade, publishing `cascade_accuracy_delta`. The alert is blunt: if the trailing-7d Δ exceeds the 0.01 budget, page the owning team. This is the *only* metric that can veto a threshold change — savings gains are real only while Δ stays inside budget. **View 1 — Cascade hit-rate and routing telemetry.** Per-route counters and histograms labeled by route, model, and escalation flag: - `cascade_hit_rate = self_hosted_requests / total_requests` — the efficiency gauge; watch it per segment (product, prompt template) to catch segments silently degrading. - `cascade_escalation_rate` — how often router failover fired. A sustained rise usually means the self-hosted tier is degrading (slow TTFT, timeouts), *not* that the classifier is wrong. - `router_decision_us{stage="classifier|probe"}` and the *uncertainty-band share* — traffic landing between thresholds; a widening band over weeks means drift and a retraining trigger. **View 2 — Cost and latency per request, by tier.** `cost_per_request{route}` (token meters × contract rates, plus fleet $/hr spread across requests) alongside p50/p95 latency per tier. The money line: `monthly_spend = sum(rate(cost_per_request) * seconds_in_month)` compared against the all-frontier counterfactual — the view the FinOps review actually reads: *did the cascade save what the model said it would, this week?* **View 3 — Classifier health and drift.** The distribution of classifier scores per segment, the logprob-variance distribution of probe-stage requests, and per-threshold confusion on the nightly eval (`self-hosted-but-wrong` is the true cost of misrouting). Thresholds are only meaningful relative to the eval set they were calibrated on, so the dashboard pins current τ_var and cutoffs next to the eval Δ — one screen answering "is this router still doing its job?" **Operational loops worth wiring from day one:** - **Shadow mode for threshold changes.** Replay 48h of logged traffic through the proposed config before moving a threshold; diff the predicted Δ with zero production risk. - **Escalation as a quality canary.** A sustained rise in escalations *to* frontier is a leading indicator of self-hosted quality drift — investigate before users file bugs. - **Accuracy sampling in production.** Beyond the golden set, sample 0.5–1% of self-hosted responses for LLM-as-judge scoring. Golden sets go stale; production samples don't. --- ## Build the Cost-Aware Tier on OnlineCloud The pattern is proven: classify, probe, escalate, and police the accuracy budget — letting the frontier model do the thinking only where thinking is required. The remaining question is operational: you need managed Bedrock access, an EKS tier that scales your GPU fleet to the *cascade's* actual demand, and cost telemetry spanning both — without stitching five consoles together. That's what OnlineCloud delivers: unified access to frontier models and your own Kubernetes inference clusters, with per-cluster and per-route cost visibility out of the box. **Ship your first cascade this week — start free at [onlinecloud.dev](https://www.onlinecloud.dev/)**, and let the dashboards show you where the 72% went. </div> <div style="margin-top:var(--wp--preset--spacing--40)" class="wp-block-post-date has-small-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/designing-a-cost-aware-router-for-model-cascades-bedrock-vs-self-hosted/"><time datetime="2026-09-06T13:19:45+00:00">September 6, 2026</time></a></div> </div> </li><li class="wp-block-post post-252 post type-post status-publish format-standard hentry category-ai-agents"> <div class="wp-block-group alignfull has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--60)"> <h2 class="wp-block-post-title has-x-large-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/implementing-audit-logging-for-agentic-decisions-a-schema-first-approach/" target="_self" >Implementing Audit Logging for Agentic Decisions: A Schema-First Approach</a></h2> <div class="entry-content alignfull wp-block-post-content has-medium-font-size has-global-padding is-layout-constrained wp-block-post-content-is-layout-constrained"># Implementing Audit Logging for Agentic Decisions: A Schema-First Approach ![Agentic decision audit logging reviewed by a compliance architect — OnlineCloud](https://www.onlinecloud.dev/assets/hero-profile-BlpwYfK_.png) *By the compliance engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio* — ## 1. Evidence, Not Transcripts: Why Chat Logs Fail Audit Regulators do not ask what your agent *intended*; they ask what it *did* — which systems it touched, which records it read or modified, under whose authority, at what moment, and with what approval. When the SOC 2 assessor or the external auditor arrives, the most common defense I see is a database full of chat transcripts, and it collapses in the first fifteen minutes. An unstructured chat log is a conversational artifact, not an evidentiary record. It captures what the model *said*, paraphrased for a human reader — not what it actually executed. The refund the agent processed appears in the transcript only if the model happened to mention it, and models summarize, elide, and occasionally confabulate. The tool call that mutated a ledger row, the exact arguments passed, the row-level security context under which the query ran, the approval that authorized it: none of that was ever designed to be in the transcript. Three structural failures make chat logs unusable for audit. **Incompleteness:** an agentic run is mostly invisible machinery — planning, tool invocations, retries, aborted branches — so a transcript cannot attest to activity that was never recorded. **Non-verifiability:** transcript text cannot be cryptographically anchored, so you cannot prove it was not edited, reordered, or regenerated. **Non-reconcilability:** transcripts carry no stable identifier linking an utterance to the database transaction it caused, so you cannot trace an anomalous write back to the decision behind it. The fix is a change of frame: stop treating agent runs as conversations to be stored and treat them as *transactions to be journaled*. Every execution should emit a structured, versioned, append-only stream of decision events — a schema that defines, up front, what an auditable decision looks like. This post is the field guide: the schema, the log anatomy, immutable storage, the control mapping, and the replay tooling that turns raw events into evidence. ## 2. The Decision Schema: A Parent-Child Tree of Execution Schema-first means the audit contract is defined *before* the agent runs, not reconstructed after an incident. If every tool, sub-agent, and orchestration loop must emit events conforming to one versioned schema, completeness stops being a matter of developer discipline and becomes a property of the system: an event that does not conform cannot be written. The core insight: an agentic execution is not a flat sequence of log lines — it is a **tree**. A run begins with a goal; the orchestrator plans and spawns branches; each branch invokes tools that return results, trigger retries, request approval, or spawn sub-agents; some branches abort. Linear logs destroy that structure. A parent-child event model preserves it, so an auditor can walk from the initiating user request down to the tool call that changed a row — aborted branches included, because proving what was considered and rejected is often as important as what succeeded. Every event carries a common envelope: a globally unique `event_id`; a `run_id` grouping the whole decision; `parent_event_id` linking the event to the step that caused it; an `event_type` from a closed vocabulary; `occurred_at` in UTC with sub-second precision; the acting principal and its full delegation chain; and a monotonic sequence number for ordering within the run. “`json { “$schema”: “https://schemas.onlinecloud.dev/agentic-audit/event.schema.json”, “title”: “AgenticAuditEvent”, “type”: “object”, “required”: [ “schema_version”, “event_id”, “run_id”, “parent_event_id”, “event_type”, “occurred_at”, “sequence”, “principal” ], “properties”: { “schema_version”: { “const”: “1.0” }, “event_id”: { “type”: “string”, “format”: “uuid” }, “run_id”: { “type”: “string”, “format”: “uuid” }, “parent_event_id”: { “type”: [“string”, “null”], “format”: “uuid” }, “event_type”: { “enum”: [ “run_started”, “plan_created”, “sub_agent_spawned”, “tool_invoked”, “tool_completed”, “tool_failed”, “approval_requested”, “approval_granted”, “approval_denied”, “branch_aborted”, “run_completed”, “run_failed” ] }, “occurred_at”: { “type”: “string”, “format”: “date-time” }, “sequence”: { “type”: “integer”, “minimum”: 0 }, “principal”: { “type”: “object”, “required”: [“initiating_user”, “delegation_chain”], “properties”: { “initiating_user”: { “type”: “string” }, “delegation_chain”: { “type”: “array”, “items”: { “type”: “string” } } } }, “tool”: { “type”: “object”, “properties”: { “name”: { “type”: “string” }, “arguments”: { “type”: “object” }, “argument_hash”: { “type”: “string” }, “result_status”: { “enum”: [“success”, “error”, “timeout”, “rejected”] }, “result_ref”: { “type”: “string”, “description”: “Pointer to stored result in the evidence store” } } }, “model”: { “type”: “object”, “properties”: { “model_id”: { “type”: “string” }, “prompt_version”: { “type”: “string” }, “reasoning_ref”: { “type”: “string” }, “tokens_in”: { “type”: “integer” }, “tokens_out”: { “type”: “integer” } } }, “access_context”: { “type”: “object”, “properties”: { “tenant_id”: { “type”: “string” }, “database_role”: { “type”: “string” }, “effective_scope”: { “type”: “string”, “description”: “RLS predicate actually enforced” } } } } } “` Three design rules matter in practice. First, the schema is **closed for event types but open for attributes**: add fields by bumping `schema_version`, never by overloading existing ones, so replay tools migrate deterministically. Second, every node carries its `parent_event_id` — an event without a parent means a malformed run, and the stream should alarm. Third, the schema is owned by the compliance function, not the model team — a feature that cannot emit conforming events cannot ship. ## 3. Audit Log Anatomy: Tool Inputs, RLS Context, and Reasoning An envelope without payload is a timestamp. The evidentiary value lives in three payload classes, each with its own capture discipline. **Tool inputs and outputs.** The most audited fact in an agentic system is: *what did the tool actually receive, and what did it return?* Capture the validated arguments *after* policy enforcement — schema-validated, scoped, sanitized — not the raw model proposal, which may contain hallucinated or injected content. Record an `argument_hash` over the canonical serialization so payloads can be integrity-checked without re-reading them, and write full results to a separate evidence store behind a `result_ref` pointer. Redact secrets, tokens, and full PII at the boundary before writing; a compliant log must not become a data breach. Record the outcome truthfully — success, error, timeout, or policy rejection — because a rejected call is an audit event too, often the most interesting one in an incident review. **RLS context.** Row-level security is what makes agentic database access defensible, but the auditor needs proof the boundary was *actually enforced* at the moment of the call. Capture the live access context: the tenant or account ID, the database principal, the propagated delegation chain (initiating user → orchestrator → sub-agent), and the effective predicate the RLS layer appended — the `SYS_CONTEXT` values that drove an Oracle VPD policy, or the session context a Postgres or SQL Server row-security policy read. When you can show the auditor that a customer query ran under `tenant_id = 4421` with the initiating user’s delegation chain, you have converted “the agent can only see its own data” from an assertion into a demonstrated control. **Model reasoning steps.** The uncomfortable question is whether to log chain-of-thought. My guidance: capture *decision-relevant* reasoning, not the raw hidden trace. Log the plan the model committed to, the retrieved-context identifiers that informed each step (which documents and tool results entered the window), the model and prompt versions, and the outcomes at approval gates. If you persist full reasoning tokens — and there are legitimate forensic reasons to — treat them as the most sensitive material in the estate: separately encrypted, access-controlled to the compliance role, excluded from end-user transcripts, and bound to their own retention schedule, because reasoning traces routinely contain PII that was never meant to be stored. The schema models this with a `reasoning_ref`: a pointer, not an inline dump. ## 4. Immutable Storage: Object Lock or Write-Only RDS An audit trail you can edit is a narrative. Immutability has one job: make the record *survive the compromise it is designed to detect*. Two production patterns dominate, and both rest on the same rule — **the write path must be reachable only by credentials the agents cannot hold, and no code path in the agent runtime may delete or modify events.** **Pattern A: S3 with Object Lock in Compliance mode.** Ship events as objects keyed by `run_id/sequence`, versioned, into a bucket with Object Lock in Compliance mode and a retention period exceeding your records schedule — align it to legal counsel and any SOX-relevant retention. In Compliance mode, not even the root account can shorten retention or delete an object before the period lapses; that is the property you are buying. Give the agent runtime’s IAM role `s3:PutObject` only, with a bucket policy denying `s3:DeleteObject`, `s3:DeleteObjectVersion`, and `s3:PutBucketPolicy` to every role the model can assume, and enforce SSE-KMS with a key the agent role cannot use. Buffer events through SQS or Kinesis so the producer never talks to the bucket directly — the queue is the only door — and alarm on any write that bypasses the conforming path. Lifecycle-tier to Glacier after the active window; the retention clock still runs on the locked object. **Pattern B: write-only RDS.** For teams that want the trail beside the business data, use a dedicated audit schema where the application role holds `INSERT` only — `UPDATE`, `DELETE`, and `TRUNCATE` privileges simply do not exist on the role, `DROP` belongs to a separate compliance role, and a trigger rejects any mutation of the event payload. Anchor the chain cryptographically: each row stores a hash of the previous row’s canonical serialization, producing a tamper-evident hash chain that makes silent back-dating detectable on replay. A periodic integrity job recomputes the chain and pages compliance on any break. Choose between the patterns on your threat model and cost curve — object storage is cheaper at volume and carries native WORM semantics; write-only RDS keeps events queryable in place. The non-negotiable is identical: the audit store is independent infrastructure, its credentials are provisioned outside the agent’s identity graph, and its integrity is monitored as a control, not assumed as a feature. ## 5. Compliance: Reconciling Decision Trails with SOC 2 and SOX The schema and storage above are not abstractions; they map one-to-one onto the control frameworks your auditors will invoke. **SOC 2 CC-series.** The AICPA Trust Services Criteria are explicit that logical access must prevent unauthorized access to systems and data (CC6), that operations must be monitored for anomalies (CC7), and that change management must be evidenced (CC8) — and an agentic deployment strains all three. CC6.1 through CC6.3 require access to be granted by role and logged: the decision schema’s delegation chain is your evidence of which principal held which capability at decision time. CC6.6 requires the audit trail itself to be protected from alteration and unauthorized destruction — that is Section 4, verbatim. CC7.2 and CC7.3 require anomaly detection and response: your malformed-event alarms and hash-chain integrity jobs are the monitoring controls. When the assessor asks how you know the agent only touched approved data under approved identities, the answer is a replayable, immutable decision tree — not a screenshot of a chat window. **SOX.** Where agents touch financial processes — revenue recognition, refunds, journal entries, close activities, order-to-cash — the trail becomes evidence over internal control over financial reporting. Under Sections 302 and 404, management must certify, and auditors attest to, the design and operating effectiveness of controls around financial data, including the *automated* controls your agents now embody. An agent that posts a journal entry is a control activity; the question is whether its execution is auditable. The parent-child tree supplies the ITGC evidence the engagement team will demand: the initiating user’s authority (segregation of duties — the agent cannot approve its own irreversible action, because `approval_granted` events come from a human checkpoint identity), the change record (model and prompt versions pinned per event), and the data integrity chain (hash-linked events reconciling to the ledger). Retain SOX-relevant decision trails as long as the underlying financial records — commonly seven years — and design the Object Lock window accordingly, because retroactive retention is not a thing in object storage. The reconciling rule is simple to state and hard to fake: **every control assertion about the agent must cite a decision-tree artifact.** “Access was scoped” cites RLS-context events; “changes were authorized” cites approval-checkpoint events; “nothing was altered” cites the hash chain. A control that cannot cite an event does not exist. ## 6. Forensic Replay: Rebuilding the Trajectory for Incident Review The final layer pays for all the others: tooling that turns the event stream back into a decision. Replay takes a `run_id`, assembles the tree by resolving `parent_event_id` links, and renders the trajectory as an interactive timeline — goal, plan, each tool invocation with its validated arguments and outcome, approval gates, aborted branches, and reasoning pointers. Because every event carries the delegation chain and UTC timestamps, the tool answers the questions incident reviewers actually ask: *What did user X’s session do between 14:02 and 14:09? Which tool calls touched records outside tenant 4421? Which write operations ran without an approval checkpoint? Did this refund decision precede or follow Tuesday’s prompt-version change?* Build replay with four properties. **Full fidelity:** include failed and rejected events, not just successes — incidents hide in branches that were denied and retried with mutated arguments. **Role-based redaction:** the replay layer re-applies the ingestion redaction policy, so a Tier-1 investigator sees arguments with secrets masked while the compliance role can expand them under a documented access reason. **Integrity verification first:** validate the hash chain before rendering; if the store shows tampering, the tool says so before it shows anything else. **Exportable evidence:** every view exports to a signed, timestamped artifact (CSV, JSON, or PDF) that reaches an auditor or opposing counsel without another engineer in the loop. The benchmark of a good replay tool is boring: an external reviewer can reconstruct what happened, in order, with citations, in an afternoon — with no access to the production agent. ## The Audit Trail Is the Product The agentic era has not changed the first law of auditable systems: if you cannot prove what happened, from the outside, after the fact, it did not happen in the eyes of your auditors. Unstructured chat logs fail that test on day one. A schema-first decision tree — structured events, captured tool inputs, enforced RLS context, immutable storage, framework mapping, replay tooling — passes it, because it treats the audit trail as a first-class system property, not a logging afterthought. Organizations that deploy agents without it will learn the lesson in an incident review; those that ship it will hand the assessor a replayable decision tree and watch the room relax. Choose which meeting you want to be in. — **Designing an audit trail for agentic decisions? Talk to the OnlineCloud architects.** We are a US-based Agentic AI & MCP engineering studio — senior engineers only, no offshore handoff — and we ship evaluated, observable, audited systems into production in 6–10 weeks. From decision schemas and MCP tool surfaces to Object Lock pipelines and write-only audit stores, we build compliance into the architecture, not the slide deck. Contact us at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/) for a free initial consultation — we respond within 24 hours. </div> <div style="margin-top:var(--wp--preset--spacing--40)" class="wp-block-post-date has-small-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/implementing-audit-logging-for-agentic-decisions-a-schema-first-approach/"><time datetime="2026-09-06T13:19:45+00:00">September 6, 2026</time></a></div> </div> </li><li class="wp-block-post post-253 post type-post status-publish format-standard hentry category-ai-agents"> <div class="wp-block-group alignfull has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--60)"> <h2 class="wp-block-post-title has-x-large-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/building-a-custom-vector-search-engine-in-rust-performance-benchmarks/" target="_self" >Building a Custom Vector Search Engine in Rust: Performance Benchmarks</a></h2> <div class="entry-content alignfull wp-block-post-content has-medium-font-size has-global-padding is-layout-constrained wp-block-post-content-is-layout-constrained"># Building a Custom Vector Search Engine in Rust: Performance Benchmarks ![OnlineCloud — agentic AI and applied ML engineering studio](https://www.onlinecloud.dev/assets/hero-company-V8Xps1Ae.png) Every serious vector search project starts with the same seduction: grab the nearest database that ships an “extension,” call `CREATE EXTENSION`, and you’re searching in an afternoon. The extension is free, it speaks SQL, and your team already runs the database. What nobody tells you is that you have also signed up for a p99 latency ceiling you cannot tune, cannot prefetch around, and cannot thread. At a billion vectors under real concurrency, that ceiling is where products start to feel slow — and no amount of index tuning inside someone else’s extension will move it. This is the story of why we stopped renting a search index inside an OLTP database and built our own HNSW engine in Rust — how the proximity graph went together with `ndarray`, what AVX-512 and Rayon actually bought us on the tail, and where we landed benchmarking against pgvector and FAISS on a billion vectors. ## The p99 Ceiling You Inherit With Generic Vector Extensions Let’s be precise about what “generic vector extension” means and where it pinches. A Postgres extension like pgvector is not a search engine; it is a *guest* that borrows the host’s execution model. Your ANN query becomes a plan node in the Postgres executor. It competes with your OLTP workload for the same buffer pool, the same WAL, the same memory allocator, and — most painfully — the same per-backend execution context. HNSW is a pointer-chasing workload: each level of the graph is a random walk over node neighborhoods, where every hop is a dependent load into a cache line you only discover after the previous hop resolves. That pattern rewards *dedicated* structures: memory you own, a layout you control, prefetches you issue. Inside an extension, the graph lives in pages that are subject to the buffer manager’s eviction policy, because Postgres does not know your index is hot — it knows only that pages get touched. Under checkpoint pressure or a bulk load elsewhere in the database, your graph pages get written out, and the next search takes a page fault at p99. Then there is concurrency. Postgres gives you one backend per connection and no shared-memory compute. At 64 concurrent search clients, the graph walk’s memory-level parallelism per query collapses — everyone is thrashing the same TLB and the same LLC. The aggregate throughput story is fine; the *tail* story is not. We watched p99 drift from tens of milliseconds to hundreds under mixed load, with latency spikes correlating to vacuum, checkpoints, and nothing at all. That is the ceiling: you can buy more RAM, but you cannot buy control. A generic extension optimizes for “works everywhere,” which is the opposite of “fastest possible tail on *your* data, on *your* hardware, for *your* workload.” FAISS, meanwhile, is not an extension — it is a C++ library — but it is also not a serving system. It gives you the kernel, then leaves you to invent the threading model, the memory lifecycle, the graceful degradation under load. That gap — kernel performance without serving control, or serving control without kernel performance — is exactly the space a custom Rust engine occupies. ## Why Rust? If we were going to own the hot path, we were going to own everything on it. That pointed at Rust for four concrete reasons. **Memory safety without a GC.** A p99 tail is, more than anything, a statement about unpredictable pauses. Garbage collectors stop the world; `malloc` sometimes takes a lock. Rust gives you deterministic ownership and `no_std`-adjacent discipline — you know exactly when memory is allocated and freed, and you can route all hot-path allocation through a pre-sized arena. We measure allocations per query in single digits. **Zero-cost abstractions that survive monomorphization.** The distance kernel is generic over the storage type — `f32` for float, `i8` for quantized — and Rust monomorphizes that generic into separate, specialized functions. There is no virtual dispatch on the inner loop, no interface box, no indirection the optimizer cannot see through. A C++ template gives you the same thing; Rust gives you it with `cargo`-managed feature flags and an ecosystem that treats it as normal. **First-class SIMD and safe-enough intrinsics.** `core::arch` exposes `_mm512` intrinsics directly, `is_x86_feature_detected!` gives runtime CPU dispatch without build-time guessing, and the portable SIMD story keeps improving. You write one AVX-512 kernel, one AVX2 fallback, and the runtime picks per machine. **Rayon.** Parallelism on the serving path in Rust is a `.par_iter()` away with work-stealing that does not collapse under load — a sentence you cannot write about most languages without qualification. Rust is not magic. C++ with `-march=native` and careful allocator choice matches most of this. But Rust matches it while making the concurrency and memory errors that destroy p99 *at compile time*, and that trade — equivalent performance with a fraction of the debugging tax — is why the engine is Rust. ## Building the HNSW Graph With ndarray HNSW is conceptually simple: a multi-layer proximity graph where upper layers are sparser and longer-range, and layer 0 contains every node. Search descends greedily from the top; each layer is a best-first walk with a bounded candidate pool. The two operations that dominate are (a) neighbor selection during insert and (b) distance evaluation everywhere. Both are matrix problems in disguise, which is where `ndarray` earns its place. We store all vectors as rows of a single `Array2<f32>` (float path) or `Array2<i8>` (quantized path), so the “index” is one contiguous, cache-friendly allocation and a vector *is* a row view: “`rust use ndarray::{Array2, s}; use std::cmp::Reverse; use std::collections::BinaryHeap; struct HnswIndex { vectors: Array2<f32>, // one row per node, dims columns layers: Vec<Layer>, // layers[0] contains every node m: usize, // max outgoing edges per node ef_construction: usize, } struct Layer { // adjacency: node_id -> Vec<node_id>, kept sorted by distance neighbors: Vec<Vec<u32>>, } “` Insert is where HNSW earns its reputation for subtle bugs. The classic algorithm adds the new node to layer 0, finds its `ef_construction` nearest neighbors at each level via a greedy walk, then runs the *heuristic neighbor selection* — which deliberately keeps diverse neighbors rather than the closest ones, because a proximity graph of only-closest links degrades to a linked list. We implemented that as a loop over `vectors.rows()` with `ndarray`’s `dot`, then applied the distance-and-diversity prune against the running candidate set: “`rust fn search_layer(&self, query: &[f32], entry: u32, ef: usize) -> BinaryHeap<Reverse<(f32, u32)>> { let mut visited = vec![false; self.vectors.nrows()]; let mut best = BinaryHeap::new(); // max-heap by (distance) for pruning let mut candidates = BinaryHeap::new(); // min-heap for exploration // … greedy descent with distance computed as a row-view dot product … while let Some((dist, node)) = candidates.pop() { if dist > best.peek().map(|x| x.0).unwrap_or(f32::MAX) { break; } for &nb in &self.layers[0].neighbors[node as usize] { if visited[nb as usize] { continue; } visited[nb as usize] = true; let d = 1.0 – self.vectors.row(nb as usize).dot(query); // normalized: cosine ≡ IP if best.len() < ef || d < best.peek().unwrap().0 { candidates.push((d, nb)); best.push(Reverse((d, nb))); if best.len() > ef { best.pop(); } } } } best } “` `ndarray`’s real payoff shows up outside the graph walk: building the scalar-quantization codebooks (per-dimension min/max via `fold_axis`), normalizing a million embeddings with a fused `mapv` pass, and — critically — batch-scoring candidate lists as real matrix multiplies when `ef_search` is large. Graph traversal is pointer chasing and stays scalar; the *arithmetic* is SIMD-friendly and belongs in `ndarray`’s lane. Getting that split right is half the performance story. ## Benchmarking: Rust vs. pgvector vs. FAISS at a Billion Vectors Benchmarks lie by omission, so here is the whole rig. We used the Deep1B corpus — 1 billion 96-dimensional learned descriptors, the de facto standard for billion-scale ANN work — with recall@10 against the published ground truth. Hardware: a single 2-socket box with Sapphire Rapids Xeons (AVX-512, 60 physical cores total), 1 TB DDR5. Every engine stored vectors int8-quantized (SQ8) to keep the working set comparable: ~96 GB of vectors plus graph. pgvector ran on the same hardware as a dedicated Postgres 16 instance. FAISS used `IndexHNSWSQ` with 64 threads. Our engine used the same HNSW parameters (M=32, `efConstruction`=200, `efSearch`=64) and 60 Rayon threads. Queries: 128 concurrent clients, 10,000-query sample, p50/p99 reported per query. | Engine | Storage | Recall@10 | p50 | p99 | QPS @ 128 clients | |—|—|—|—|—|—| | **pgvector** (HNSW, SQ8) | 96 GB + WAL/graph | 0.89 | 24 ms | 512 ms | 410 | | **FAISS** (IndexHNSWSQ) | 96 GB + graph | 0.93 | 6.1 ms | 41 ms | 1,150 | | **Rust engine** (pre-tune) | 96 GB + graph | 0.93 | 7.4 ms | 68 ms | 980 | | **Rust engine** (AVX-512 + Rayon batch) | 96 GB + graph | 0.93 | 4.2 ms | 11 ms | 3,700 | Read pgvector’s row carefully. Its *median* is respectable — 24 ms for a billion-vector ANN query from inside Postgres is genuinely not bad. Its p99 is where the extension tax shows up: 512 ms is a 21× blowup over the median. Buffer-manager evictions, WAL churn from graph updates, allocator contention across 128 backends — the tail is not search time, it is *co-tenancy time*. If your product SLA is “p99 under 50 ms,” pgvector fails by an order of magnitude no matter how many replicas you throw at it, because each replica pays the same structural tax. FAISS is the honest baseline: with 64 threads hammering a shared index it delivers 41 ms p99. Our pre-tuned engine was actually *slower* on p50 — expected, we were a young implementation — but even then the tail was better behaved. After the optimization pass in the next section, the gap stopped being about kernels (FAISS’s distance kernels and ours are within noise of each other) and became about the serving layer: we control the threading, we batch across queries, we pin memory, and we refuse to let any single query’s graph walk monopolize a socket. That is what turned 68 ms of tail into 11 ms — and it is the exact thing a library or an extension cannot sell you. Honesty corner: if your corpus is a few million vectors, pgvector is the right call and this post is academic. At billion scale, the engine *is* the product, and renting one you cannot tune is a p99 liability. ## Optimization: Rayon and AVX-512 on the Distance Kernel The post-tune numbers came from two moves: vectorize the arithmetic, then parallelize the *right* granularity. **AVX-512 on the dot product.** A 96-dimension int8 dot product is 96 multiply-accumulates per comparison — trivially SIMD-shaped. The AVX-512 kernel processes 64 int8 lanes per instruction via `_mm512_maddubs_epi16` (unsigned×signed multiply-add pairs) followed by `_mm512_sad_epu8` to widen and accumulate: “`rust #[target_feature(enable = “avx512f,avx512bw”)] unsafe fn dot_i8_avx512(a: *const i8, b: *const i8, len: usize) -> i32 { let mut acc = _mm512_setzero_si512(); let mut i = 0; while i + 64 <= len { let va = _mm512_loadu_si512(a.add(i) as *const __m512i); let vb = _mm512_loadu_si512(b.add(i) as *const __m512i); // maddubs: pairwise multiply-accumulate of u8 * i8 -> i16 acc = _mm512_add_epi32(acc, _mm512_sad_epu8(_mm512_maddubs_epi16(vb, va), _mm512_setzero_si512())); i += 64; } // horizontal sum of acc, then scalar tail… hsum_epi32(acc) + scalar_tail(a, b, i, len) } “` Two practical notes from the trenches. First, `#[target_feature]` with `unsafe` is the *only* safe way to compile AVX-512 in the current stable toolchain without poisoning the whole crate with `-C target-feature` — and it keeps a clean scalar fallback for non-AVX-512 machines. Second, AVX-512 on server Xeons can downclock the core under heavy 512-bit load; because our comparisons are memory-latency-bound far more than compute-bound, the FMA throughput rarely matters. What matters is issuing 64-wide loads that fill cache lines in one pass. If you are ever tempted to obsess over FLOPs in an ANN kernel, don’t — obsess over memory-level parallelism. **Rayon at the right granularity.** The naive parallelization — `par_iter` over the candidate set inside one query’s graph walk — is wrong: graph walks are latency-critical, lock-free, and short, and cross-thread work-stealing per hop costs more than it saves. The two places parallelism actually pays are *construction* and *batch serving*. During index build, neighbor selection for different nodes is embarrassingly parallel, so `par_iter_mut()` over the layer-0 node range turns a 14-hour build into 3.5. During serving, we batch concurrent queries — 128 clients arriving on 128 async tasks funnel into a Rayon pool where each worker owns a thread-local query buffer and the AVX-512 kernel never contends for a shared scratch space. Query throughput scales nearly linearly to the socket count, and the tail stays flat because no single query can starve the pool. That batching layer — not the kernel, not the graph — is the real deliverable of writing your own engine. ## The OnlineCloud Stack: Low-Latency Search for E-commerce Intelligence This engine did not come from a “let’s build a vector database” itch. It came from a product requirement: e-commerce intelligence at OnlineCloud needs semantic search over product catalogs that can grow to hundreds of millions of SKUs — embeddings for titles, descriptions, images, and behavioral signals — queried from recommendation, personalization, and agentic shopping surfaces where every millisecond is a conversion decision. The production shape is the unglamorous one that makes the benchmarks matter. Embeddings (catalog + query-side) are refreshed by the applied-ML pipeline and normalized into the `Array2` codebook. The Rust engine runs as a lean gRPC service on dedicated instances — no co-tenancy, memory pinned, graph preloaded at boot — and is wrapped as a typed MCP tool, so the agentic surfaces (order-status bots, recommendation agents, the e-commerce intelligence layer) call `catalog_search(query, k, filters)` with the same discipline as any other tool: rate limits, audit logs, and a hard latency budget. On AWS, the whole thing sits behind the same VPC isolation and FinOps discipline OnlineCloud applies to every AI workload. The engine answers at p99 ≈ 11 ms on a billion vectors; the agents that call it never see a database checkpoint again. We learned the ceiling the expensive way so you do not have to: generic vector extensions are a compatibility layer, not a latency strategy. At the scale where your p99 is your product, you want the hot path in a language that lets you own it — and a team that has already built it. — **Ready to ship search that holds its p99 under real load?** OnlineCloud is a US-based engineering studio that builds MCP-native, evaluated, production-observable systems — including custom vector search for e-commerce intelligence — and ships them in 6–10 weeks, not quarters. We would rather argue about your `efSearch` than sell you a deck. [Contact us](https://www.onlinecloud.dev/) or write to contact@onlinecloud.dev — free initial consultation, response within 24 hours. </div> <div style="margin-top:var(--wp--preset--spacing--40)" class="wp-block-post-date has-small-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/building-a-custom-vector-search-engine-in-rust-performance-benchmarks/"><time datetime="2026-09-06T13:19:45+00:00">September 6, 2026</time></a></div> </div> </li><li class="wp-block-post post-249 post type-post status-publish format-standard hentry category-ai-agents"> <div class="wp-block-group alignfull has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--60)"> <h2 class="wp-block-post-title has-x-large-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/implementing-semantic-versioning-for-mcp-tools-managing-protocol-drift/" target="_self" >Implementing Semantic Versioning for MCP Tools: Managing Protocol Drift</a></h2> <div class="entry-content alignfull wp-block-post-content has-medium-font-size has-global-padding is-layout-constrained wp-block-post-content-is-layout-constrained"># Implementing Semantic Versioning for MCP Tools: Managing Protocol Drift ![Implementing semantic versioning for MCP tools with OnlineCloud](https://www.onlinecloud.dev/assets/hero-profile-BlpwYfK_.png) *By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio* — ## 1. Introduction: Why Breaking Changes in Tool Schemas Kill Production Agents Every team that has shipped an agent to production learns the same lesson the worst way. Your MCP server has run for months; ticket deflection is up, your CFO is happy. Then someone “tidies up” a tool schema on a quiet Tuesday: `create_ticket(customer_email)` becomes `create_ticket(contact_id)`. A human API consumer reads the changelog and fixes their integration in an afternoon. Your agent keeps working — badly. Here is what actually happens. The next customer inquiry arrives, and the model, still “knowing” the old surface from its prompts, training, and session context, calls with `customer_email`. The server rejects it. The model retries — models are stubborn about well-learned patterns — then burns tokens re-reading the schema and guesses at what `contact_id` means. It either fails the task or fabricates a value that validates but resolves to the wrong customer. The ticket is misrouted. PII lands in the wrong inbox. Nobody gets paged, because nothing *threw*; the agent just quietly delivered a worse outcome. This is the asymmetry between human and machine consumers. A human integrates against a contract once and reads release notes on change. A production agent re-discovers your tool surface every session, but it arrives carrying priors from everything it has ever seen — including your schema from three months ago. When the tool boundary drifts from what the model expects, the failure is not a clean error code. It is degraded behavior inside an unattended loop: retries, hallucinated arguments, and confidence miscalibration that logs can barely trace. We call this **protocol drift** — the gap between the surface your server advertises and the surface your agents believe they are calling. In fifteen-plus years shipping data systems at OnlineCloud — Oracle, MS SQL, Postgres, Snowflake, CRMs — we have never seen a system fail because a contract existed. We have seen systems fail because contracts changed without discipline. This post is the playbook for versioning MCP tool interfaces with semantic versioning, so schema evolution becomes routine instead of an outage in slow motion. — ## 2. SemVer for AI: Defining Major/Minor/Patch for Tool Interfaces Semantic versioning was designed for libraries, where the unit of compatibility is the function signature. In MCP, the unit is the **tool surface**: the `inputSchema` that defines every request an agent can validly send, and the `returns` — the output shape of a successful `tools/call`. Version the surface, and SemVer maps cleanly if you define the boundaries precisely. **Patch (x.y.***z***) — nothing about valid requests or responses changes.** You clarified a tool’s `description`, fixed an error message, or added an `example` to a schema property. Any agent that worked before works identically after. One warning: description edits are invisible to type systems but fully visible to models, so treat them as *behavioral* patches and run them through the same eval gate as code. **Minor (x.***y***.z) — backwards-compatible additions.** You added an *optional* input property with a sensible default, added an enum value that maps to an existing code path, added an output field old consumers will ignore, or shipped a brand-new tool. The discipline is ruthlessness about the word *optional*: adding a *required* property changes the set of valid requests, which makes it major. Output additions are minor only if consumers tolerate unknown fields; otherwise they are major too. Our standing rule: *new required field, new tool name.* **Major (x.***y***.z) — previously-valid calls stop working or semantics change.** Removing or renaming a tool; changing a type, `pattern`, or `minimum`; tightening an `enum`; adding a required property; removing an output field; changing error or side-effect semantics. Major is not a value judgment; it is the statement that existing agents will break, and you are buying the right to break them deliberately. Two properties of AI consumers make this trickier than human APIs. First, **the schema is not the only contract** — models also learn behavior from descriptions, examples, and listing order, so a technically-minor reword can be behaviorally major. Second, **models generalize**: tightening an enum from five values to three doesn’t just reject two strings — it changes the model’s willingness to attempt adjacent calls. When in doubt, call it major: an unannounced breaking change in an unattended loop costs an order of magnitude more than one extra version bump. Finally, version at the right granularity. A server with forty tools does not need one monolith version; it needs a version per tool surface, aggregated into a published manifest (section 4). — ## 3. Deprecation Patterns: Orchestrating Tool Upgrades Without Agent Downtime Knowing a change is major solves nothing. The hard part is moving agents — some long-running, some third-party — off the old surface without a cutover weekend. **Additive first.** MCP’s tool namespace is flat, so you cannot serve two `create_ticket` tools — but you can ship `create_ticket_v2` beside the original. That is additive, minor, and safe: both appear in `tools/list`, agents discover both, and nothing breaks because nothing changed. **Dual-run.** While both versions live, run the same canonical task set against each and compare tool-call validity, completion, and repair-loop frequency before any production traffic shifts. This surfaces the real breakage — not schema violations (CI covers those) but *which* tool the model picks and how often it fumbles new argument shapes. **Canary.** Shift production by the natural unit: new sessions and new agent instances, whose tool expectations crystallize early. Point 5–10% of new sessions at a profile that hides the deprecated tool, watch error rates and outcomes for two weeks, then expand. Rollback is a configuration flip. **The deprecation lifecycle.** Codify four states, each visible in `tools/list`: 1. **Active** — current, fully supported. 2. **Deprecated** — works identically, but the description opens with `DEPRECATED — use create_ticket_v2`. The model gets positive pressure toward the new tool *and* a working fallback. Log every deprecated call with session and agent version — that is your migration telemetry. 3. **Sunset** — reachable only by explicit opt-in; returns a structured error naming the replacement. Few models keep fighting once the failure is loud and gives the answer. 4. **Removed** — the major lands; the tool disappears and your golden suite confirms nothing in the wild still calls it. The cardinal rule: **never remove a tool that anything still calls.** Before the removal major, require a sustained window — at OnlineCloud, no fewer than four weeks — of zero deprecated-tool call volume across production and eval traffic. Sessions that cached the old surface can resurface weeks later. The deprecation window is not overhead; it is what makes major bumps boring. — ## 4. Metadata: Using Annotations and Version-Scoped URIs in MCP Versions only help if they are machine-readable where agents consume them. MCP gives you two native mechanisms — annotations and URIs — plus protocol signals, and disciplined teams use all three. **Annotations.** Every tool carries an `annotations` object (`title`, `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`) that shapes how hosts and models treat it — treat it as part of the versioned surface. Extend it with explicit version metadata, and mirror it in a canonical, parseable description block so a model reading the listing can act on it even if the host never surfaces the structured field: “`json { “name”: “create_ticket_v2”, “annotations”: { “title”: “Create ticket (v2)”, “readOnlyHint”: false }, “description”: “[version=2.1.0][deprecates=create_ticket][since=2026-08-01] Create a support ticket.”, “inputSchema”: { “type”: “object”, “required”: [“contact_id”, “subject”, “severity”], “properties”: { “contact_id”: { “type”: “string” }, “subject”: { “type”: “string” }, “severity”: { “type”: “string”, “enum”: [“low”, “medium”, “high”] } } } } “` **Version-scoped URIs.** MCP resources are addressed by URI, and the URI can be part of your contract. Serve each tool version’s JSON Schema at a stable address — `mcp://server/schema/create_ticket/v2` — so schemas become fetchable, cacheable, diffable artifacts that eval harnesses, audit tools, and golden-file suites consume from one source of truth. Keep alias URIs — `mcp://server/tools/create_ticket/latest` — for agents that want the current major, while cautious agents pin `@2`. **Protocol signals.** MCP negotiates `protocolVersion` at the transport layer, and servers can emit `notifications/tools/list_changed` whenever the surface mutates mid-session, prompting hosts to re-fetch `tools/list`. Emit it after every deploy that changes the surface; one message eliminates an entire class of stale-listing drift. Finally, publish a **version manifest resource** (`mcp://server/manifest`) listing every tool, its current version, all supported versions, and planned removals. This is what SREs query during incidents, what CI diffs against, and what audit logs join to when a regulator asks which version of `refund_order` touched a transaction. Version metadata is the join key between agent behavior and accountability. — ## 5. Automated Testing: Golden-File Testing to Validate Agent Compatibility SemVer promises are only as good as the tests that enforce them. The highest-leverage technique for tool surfaces is **golden-file testing**: checked-in, immutable snapshots that every change must be validated against. **Schema goldens.** Store the canonical `tools/list` response — every tool’s full schema, annotations, and description — as a JSON golden per server version. CI regenerates the live listing and diffs it against the golden, so any surface change becomes a *reviewable diff* and the pipeline forces you to classify it as patch, minor, or major. Tightening an enum without a major bump becomes impossible to ship silently. Keep per-version directories (`goldens/v1/`, `goldens/v2/`) so you can always reproduce what a historical agent saw. **Compatibility suites.** For every still-supported major, keep a corpus of **canonical valid requests** captured from real production transcripts — actual `tools/call` payloads, not hand-written examples. The test is brutally simple: every golden request for version N must still validate and succeed against the current server. This is the regression net against accidental majors — the day a “minor cleanup” breaks a v1 payload, CI fails, not production. Pair it with a **negative corpus** of payloads that must be rejected, so accidentally loosening a constraint also fails loudly. **Behavioral goldens.** Validation proves compatibility, not that models *behave* the same. Record agent sessions over a fixed set of canonical tasks, run them against a shadow server, and diff the tool-call transcript against a golden transcript. If a description rephrase makes the model switch tools, the diff shows the behavioral shift and a human decides whether it is intended. Two rules keep golden testing sustainable. First, **goldens change deliberately, never silently** — updating one requires a separate, labeled commit, never an auto-regeneration folded into an unrelated PR. Second, **releases train forward and test backward**: every deploy runs the full matrix — new-version schema goldens plus compatibility suites for every version you still promise to support. At OnlineCloud this matrix runs in under two minutes and has caught more would-be production incidents than every other layer combined. The suite does not prevent change; it forces change to be named, classified, and intentional — which is the entire point of versioning. — ## 6. Roadmap: The 6–10 Week Guide to Version-Controlled AI Systems You do not need a platform team for this. Here is the six-to-ten-week path we run with teams that need version-controlled agent surfaces — sequenced so every week leaves the system more defensible, with no big-bang rewrites. **Weeks 1–2 — Baseline and inventory.** Enumerate every tool in every server; snapshot live `tools/list` responses into `goldens/v1/`; start capturing real call transcripts (you need production payloads, not examples) and build the compatibility and negative corpora from them. Write the versioning policy on one page: the major/minor/patch definitions and the deprecation states above. Exit criterion: CI diffs live schemas against goldens. **Weeks 3–4 — Version machinery.** Add version metadata to every tool’s annotations and description; stand up version-scoped schema URIs and the manifest resource; wire `tools/list_changed` into deploys. Connect the golden diff to release so a surface change without a version classification cannot merge. Exit criterion: every surface change produces a reviewed golden diff and a version bump, end to end. **Weeks 5–6 — First real deprecation.** Pick one tool that genuinely needs to change. Ship the additive successor, dual-run the eval comparison, then canary 5–10% of new sessions. Instrument deprecated-call volume, repair-loop frequency, and outcome metrics. Exit criterion: a documented before/after showing the new surface is at least as good, with deprecation announced and telemetry live. **Weeks 7–8 — Hardening (take two to four weeks if your surface is large or agents are third-party).** Build the drift dashboard: validation-error rate, deprecated-call volume, repair-loop rate, and per-version call mix, with alerts on the first two. Write the incident runbook so “agent error rate spiked after a tool deploy” has a checklist, not a fire drill. **Weeks 9–10 — Removal and scale.** Execute the first fully-planned major: remove the sunset tool after its zero-call window, ship the major, confirm the compatibility suite passes for all remaining versions. Then institutionalize — fold version review into the eval cadence and publish the manifest where SREs and auditors can find it. Exit criterion: one full deprecation lifecycle completed, and the next team can do it from the runbook alone. The metrics that matter: **validation-error rate per tool** (drift surfacing), **deprecated-call volume** (migration progress), **repair-loop frequency** (agent suffering), and **time-to-resolution for tool-call failures** (maturity). When the first two trend down and the last two stay flat, you have version-controlled AI systems — not as governance theater, but as the boring, reliable substrate that lets agents change without breaking. — ## 7. Conclusion: Versioning Is How Agents Stay Honest Protocol drift is not a bug you fix once. It is a force that acts on every deploy, every description edit, every tightened enum — and it acts hardest on the agents that run unattended, where nobody watches when a schema quietly moves. Semantic versioning for MCP tools converts that silent drift into a visible, reviewable, scheduled event. The tool surface becomes a first-class artifact: versioned, annotated, URI-addressable, golden-tested, and deprecated on a timetable instead of by accident. At OnlineCloud, we build MCP-native systems for teams that need agents doing real work against real systems — Oracle, MS SQL, Postgres, Snowflake, CRMs — with versioned tool surfaces, eval suites, and observability built in from the first commit. If your agents are drifting, or you are about to change a schema that production traffic depends on, talk to our architects. We will help you turn your tool boundary into a contract that evolves on your schedule — not on the day your logs light up. **Ready to put version control on your MCP tool surface?** Contact the OnlineCloud architects today for a free initial consultation — [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) — or learn more at [https://www.onlinecloud.dev/](https://www.onlinecloud.dev/). US-based senior engineers, response within 24 hours. </div> <div style="margin-top:var(--wp--preset--spacing--40)" class="wp-block-post-date has-small-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/implementing-semantic-versioning-for-mcp-tools-managing-protocol-drift/"><time datetime="2026-09-06T13:19:44+00:00">September 6, 2026</time></a></div> </div> </li><li class="wp-block-post post-246 post type-post status-publish format-standard hentry category-ai-agents"> <div class="wp-block-group alignfull has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--60)"> <h2 class="wp-block-post-title has-x-large-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/building-a-custom-tool-discovery-service-for-large-scale-agent-fleets/" target="_self" >Building a Custom Tool Discovery Service for Large-Scale Agent Fleets</a></h2> <div class="entry-content alignfull wp-block-post-content has-medium-font-size has-global-padding is-layout-constrained wp-block-post-content-is-layout-constrained"># Building a Custom Tool Discovery Service for Large-Scale Agent Fleets ![Tool discovery service architecture for large-scale MCP agent fleets](https://www.onlinecloud.dev/assets/hero-company-V8Xps1Ae.png) *By the infrastructure engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio* — ## 1. Introduction: The 1,000-Tool Problem — Scaling Beyond Model Context Limits Every agent platform hits the same wall. A few tools — ticket lookup, refund action, warehouse query — and the model handles them effortlessly. Then the integrations multiply: CRM, ERP, billing, each customer’s bespoke connectors. Before long the fleet can reach a thousand tools, and the platform collapses under the weight of its own capability. The collapse is quantitative. A typical MCP tool definition costs 200 to 800 tokens serialized, so a thousand tools means 300,000 to 700,000 tokens of schema if you mount them all into context — beyond most production models’ entire window. Even where it fits, it is ruinous: function-calling accuracy degrades as the visible tool count passes a few dozen, the model confuses similarly named tools, and every request pays latency and dollar cost for schemas it never uses. The mistake is treating the context window as the catalog. It is a *working set* — small, relevant, loaded on demand. A thousand-tool fleet needs a **discovery layer**: infrastructure that answers, per task, *which of these capabilities do I need, and how do I reach them?* Search engines made this shift decades ago — when the corpus outgrows any client, you stop shipping the corpus and start shipping retrieval. Below is the blueprint we use at OnlineCloud, built on Model Context Protocol (MCP): a semantic search layer, a centralized tool catalog, dynamic runtime registration, tool-level RBAC, and HNSW-based performance engineering. The recurring theme: **discovery is a control-plane service, never a prompt-engineering trick.** ## 2. Tool Discovery: Implementing a Semantic Search Layer for Tool Registration Tool selection is retrieval wearing an LLM costume. The naive implementation asks the model to choose from everything; the correct one embeds every tool into a vector space and retrieves the top-k candidates for the task, letting the model choose among a handful of well-matched options. Registration is where the semantic layer is born. When an MCP server starts, it declares its tools via the `tools/list` handshake. The discovery service intercepts the declaration and turns each tool into a searchable document — the **name**, the **description**, and the **input schema**, whose field names and enums carry latent semantic signal — embedded with a fixed text-embedding model and stored beside the tool’s metadata. Freeze the embedding model per index version — re-embedding is a scheduled event, not a surprise. The query side mirrors registration. At task time, the orchestrator embeds the agent’s goal — user request, current sub-task, safe conversation context — into the same space and runs nearest-neighbor search. Production retrieval is never pure vectors: semantic search misses exact identifiers, so we run a **hybrid** pipeline — dense vectors for meaning, BM25-style lexical match for names and IDs, and metadata filters for hard constraints — fused by reciprocal rank fusion. Cold start is solved with logs and description-quality review: precise descriptions retrieve; “do refund stuff” does not. One rule is non-negotiable: **discovery is advisory; invocation is authoritative.** Retrieval proposes candidates; every call is validated against the catalog at invoke time — schema, version, endpoint, authorization. Never conflate the two. ## 3. Architecture: Designing a Centralized MCP Tool Catalog with Metadata Tags The semantic layer needs a home: a **centralized tool catalog** — a control-plane service that owns each tool’s lifecycle and records what exists, who owns it, who may call it, and where it runs. The data model separates a tool’s *definition* from its *deployment*. A **definition** is a versioned record: name, description, schemas, embedding, tags, owner, risk tier, data classification. A **deployment** is a live instance: transport, resolved endpoint, health state, and the schema hash currently served. Three healthy deployments of one definition are a single logical capability the gateway load-balances across; when a server’s live schema drifts from the catalog — teams deploy faster than they document — the hash mismatches and the catalog raises a drift alert instead of serving stale contracts. **Tags are the metadata spine**, drawn from controlled vocabularies because free-form tags decay into noise within a quarter: “`json { “domain”: “billing”, “risk_tier”: “high”, “access”: “write”, “data_class”: “pii”, “cost_tier”: 2, “owner”: “team-billing”, “status”: “active” } “` Tags feed the retrieval filters, define the RBAC policy surface (Section 5), and let platform teams answer fleet-level questions — “which tools touch PII?” — in one query. Risk tier and data classification are load-bearing: a discovery service that cannot flag tools touching expensive, regulated data cannot protect the budget or the compliance posture. Every catalog mutation is auditable, so the change log doubles as the compliance record for why an agent could call that tool on a given day. Run it as a stateless API over Postgres with pgvector (comfortable at tens of thousands of definitions) and treat availability as critical: when discovery is down, agents fall back to a cached, authorized toolset — never an unconstrained mode. ## 4. Dynamic Registration: How Agents Discover and Connect to New Tools at Runtime A static catalog is a glorified OpenAPI registry. The payoff is **dynamic registration** — agents picking up new tools, versions, and entire MCP servers mid-task, with no redeploys or prompt rewrites. The runtime flow: the orchestrator builds a **discovery query** from the current step; the catalog runs **hybrid search scoped by the agent’s authorization** and returns candidate tool stubs with deployment metadata; the gateway **reconciles schemas** (via schema hash) so the model only sees contracts the executor will enforce; the agent opens a **lazy, pooled MCP session** to the resolved endpoint bound to a short-lived, audience-scoped credential (Section 5); and **feedback** reinforces success while repeated schema failures or timeouts demote a tool and page its owner. New tools join the fleet the moment their server completes registration — a **push event, not a polling chore**. Servers register via signed webhook; the catalog validates the manifest, embeds the definitions, and publishes a change event that subscribed agents use to warm connections proactively. The hard part is **version transitions**. We version tools semantically and gate rollout through the catalog: when `create_refund` v4 publishes, v3 stays active for in-flight sessions while v4 serves new discovery queries, with a deprecation deadline attached. A mid-task agent finishes its current call against v3 — the session binding is pinned at connection time — while the next step discovers v4. No mid-call contract swaps, no orphaned sessions. Revocation is the same mechanism with urgency: the tool vanishes from discovery and sessions drain. Graceful degradation completes the design. Each agent role pins a **baseline toolset** — a small, curated, always-authorized set covering most routine tasks. If the catalog is unreachable, agents run against baseline and flag the session as degraded instead of failing open. ## 5. Security: Enforcing Tool-Level RBAC in Dynamic Discovery Environments Dynamic discovery is a double-edged sword: done right, it shrinks the model’s attack surface to the tools a task needs; done naively, it becomes a **capability disclosure service** — every query leaks tool existence, and every surfaced tool is one a prompt injection could aim. The founding rule: **never return a tool the caller is not authorized to invoke.** Authorization applies at query time, inside the catalog, as a hard filter — never as post-processing. If an agent cannot call a tool, it must not know it exists. That is the difference between filtering and *scoping*: filtering hides results after the fact; scoping constrains the search space so even the embeddings of unauthorized tools never cross the trust boundary. Authorization is two-layered. **RBAC** maps principals — agent identities, or the humans whose authority an agent inherits — to roles, then roles to allowed tool IDs or tag patterns. **ABAC** adds attribute conditions: “only 09:00–18:00,” “only cost tier ≤ 2,” “only when the delegated user holds `refund_approve` in the source system.” Tool tags feed both layers — the Section 3 vocabularies are security infrastructure, not documentation. “`json { “principal”: “agent:refund-workflow”, “effect”: “allow”, “match_tags”: { “domain”: “billing”, “access”: “read” }, “conditions”: { “data_class”: [“internal”, “pii”], “cost_tier”: { “lte”: 2 }, “delegated_user_entitlement”: “refund_approve” } } “` Connections minted through dynamic registration carry **capability-bound credentials**. Instead of a fleet-wide API key, the catalog issues a short-lived token per session whose audience is exactly the resolved endpoint, scopes exactly the authorized tool IDs, and TTL outlives the task by minutes, not months. A leaked token is useless against every other tool in the fleet — blast radius is one session, one endpoint. Three threat models deserve attention. **Prompt injection:** scoped discovery plus a whitelist-only tool view means an injected instruction cannot aim the agent at tools it was never shown, and discovery results must never widen privileges mid-task. **Endpoint spoofing:** the catalog resolves endpoints only from its own registration records and agent traffic flows through the gateway, so outbound calls to unregistered endpoints are blocked at the transport layer. **Discovery as an oracle:** discovery queries are logged, rate-limited, and audited like invocations — *who searched for what they could not have* is often the earliest signal of a compromised agent. ## 6. Performance: Optimizing Discovery Latency with Vector Caching and HNSW Discovery sits on the hot path — a single turn may issue several queries as sub-tasks decompose. Our budget is unforgiving: **p50 under 10 ms, p95 under 30 ms**, end-to-end — an order of magnitude faster than the LLM call around it, so discovery is never the bottleneck. Latency engineering is a caching problem first, an index problem second. Fleets repeat themselves — same roles, task archetypes, and hot tools hour after hour. We cache in three tiers. **L1, in-process:** keyed by (principal scope, task class, embedding bucket) so the top-20 tools for “refund escalation” resolve in microseconds. **L2, distributed:** a shared cache so a cold orchestrator inherits what its siblings computed minutes ago. **Index:** the vector index, hit only on genuine misses — the long tail of novel tasks. Invalidation is event-driven, not TTL-approximate: registration, schema change, and revocation events invalidate affected keys synchronously, because a cache serving a revoked tool for thirty extra seconds is a security bug wearing a performance costume. For index misses, **HNSW** earns its place. It builds a hierarchical navigable small-world graph over the embedding space and walks it coarse-to-fine, trading a controllable slice of recall for roughly two orders of magnitude less latency than brute force. At 50,000 definitions with 1,024-dimension embeddings, exhaustive scan costs tens of milliseconds; a tuned HNSW index answers in 2–8 ms at 95–99% recall. “`json { “index”: “hnsw”, “dims”: 1024, “m”: 16, “ef_construction”: 200, “ef_search”: 128, “quantization”: “scalar_int8” } “` Configuration discipline matters: `m` and `ef_construction` trade build cost for graph quality, `ef_search` trades query latency for recall — tune it per traffic class, since a batch job tolerates higher latency than an interactive chat turn. Memory is the silent killer: float32 embeddings for 100,000 tools × 1,024 dimensions exceed 400 MB before a single tag; scalar int8 quantization cuts that fourfold with negligible recall loss. The subtle trap is **filtered search**. HNSW shines on pure nearest-neighbor queries, but metadata filters (namespace, risk tier, authorization scope) interact badly with graph traversal — a walk can exhaust its budget where every neighbor fails the filter. Our approach is two-phase: small per-scope indexes for common hot scopes, enlarged `ef_search` with post-filtering for the long tail. Recall is a measurable SLO: periodic audits compare approximate results against exhaustive ground truth, and a filter change that silently drops recall from 97% to 91% is a paged incident — a missed tool is a failed task. Finally, measure the whole path: discovery telemetry — cache hit rate, index latency, recall audits, invalidation lag — belongs on the same dashboard as invocation metrics, because slow discovery means slow agents. — ## Conclusion: Discovery Is the Control Plane of the Agent Fleet The thousand-tool problem is not a context-window problem; it is an architecture problem. Teams that win with agents at scale are not the ones that squeeze the most schemas into a prompt — they are the ones that built a control plane where **capability is registered once, discovered semantically, authorized explicitly, and invoked through validated contracts.** Semantic search makes a thousand tools navigable; the catalog makes them governable; dynamic registration makes them deployable; tool-level RBAC makes them safe; vector caching over HNSW makes them fast enough that agents never feel the machinery between them and the work. That is the architecture we ship at OnlineCloud, wired into real estates — Oracle, SQL Server, Postgres, Snowflake, CRMs — with audited authorization and production observability from day one. **Ready to scale your agent fleet past the context window? Talk to the OnlineCloud architects.** We are a US-based engineering studio — MCP-native, senior engineers only, no offshore handoff — and we deliver evaluated, observable discovery infrastructure in production within 6–10 weeks. Contact us at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/) for a free initial consultation. We respond within 24 hours. </div> <div style="margin-top:var(--wp--preset--spacing--40)" class="wp-block-post-date has-small-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/building-a-custom-tool-discovery-service-for-large-scale-agent-fleets/"><time datetime="2026-09-06T13:17:01+00:00">September 6, 2026</time></a></div> </div> </li><li class="wp-block-post post-247 post type-post status-publish format-standard hentry category-ai-agents"> <div class="wp-block-group alignfull has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--60)"> <h2 class="wp-block-post-title has-x-large-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/optimizing-token-usage-in-rag-implementing-dynamic-context-pruning/" target="_self" >Optimizing Token Usage in RAG: Implementing Dynamic Context Pruning</a></h2> <div class="entry-content alignfull wp-block-post-content has-medium-font-size has-global-padding is-layout-constrained wp-block-post-content-is-layout-constrained"># Optimizing Token Usage in RAG: Implementing Dynamic Context Pruning ![Data engineer optimizing RAG token usage with dynamic context pruning at OnlineCloud](https://www.onlinecloud.dev/assets/hero-profile-BlpwYfK_.png) *By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio* — ## 1. Introduction: The Linear Cost of Long-Context Retrieval Every RAG pipeline I audit has the same hidden line item. Not in the vector database bill, not in the embedding batch job, but in the prompt assembly code: `context = “\n\n”.join(chunks[:top_k])`. That single slice — usually `top_k=10` or `top_k=20` — decides how many tokens the generation model re-reads on every single request. And because a chunk that added retrieval value in week one rarely gets removed in week twelve, the context grows fat, the latency creeps up, and the cost-per-query line on the FinOps dashboard doubles before anyone asks why. The economics are brutally linear. A knowledge-base query that matches twenty five-hundred-token chunks ships ten thousand tokens of context to the model — before the question, before the instructions, before the answer. At GPT-4-class input pricing, that is roughly $0.10 per request in context alone, and it compounds: users ask follow-ups, agents run multi-turn loops, and every turn re-encodes the same bloated window. Long-context models made this *possible*; they did not make it *cheap*. The 200K-token window is a safety margin, not a budget to spend. Worse, the waste is not neutral — it is actively harmful. Research on the “lost in the middle” phenomenon shows models systematically under-weight tokens in the middle of their context window. Stuff the window with marginal chunks and you are not just overpaying; you are diluting the signal that the high-value passages carry. Pruning is therefore a **quality** optimization disguised as a **cost** optimization. This post walks through the four layers where we cut token spend in production RAG — dynamic pruning at retrieval time, semantic compression before generation, resource-level pruning in the Model Context Protocol (MCP) transport layer, and the ROI math that justifies each — then closes with a working Python/Pydantic implementation of a context optimizer you can adapt today. — ## 2. Dynamic Pruning: Rerankers and Relevance Scoring The first and highest-leverage cut happens before generation: **stop shipping chunks that do not answer the question.** Vector search retrieves by embedding proximity, which is a fuzzy proxy for relevance. A bi-encoder embeds query and document independently, then measures cosine distance — fast enough to scan millions of vectors, but blind to the lexical and syntactic interactions that actually decide whether a passage answers a question. Top-20 by cosine similarity reliably contains *some* answer-bearing chunks; it rarely contains *only* them. The fix is a second-stage **cross-encoder reranker**. Unlike a bi-encoder, a cross-encoder takes the query and a candidate passage *together* through the transformer, producing an interaction-aware relevance score in (0, 1). It is too slow to run over the whole corpus — that is what the approximate-nearest-neighbor index is for — but running it over the top 20–50 candidates costs only a few milliseconds per chunk. The standard recipe: 1. **Recall stage:** embed the query, retrieve the top 40 candidates by cosine similarity (or hybrid BM25 + vector fusion via reciprocal rank fusion — dense vectors alone routinely miss exact identifiers like part numbers and policy IDs). 2. **Precision stage:** score all 40 candidates with a cross-encoder reranker (e.g., a Cohere or BGE reranker, or a fine-tuned `cross-encoder/ms-marco-MiniLM` for on-prem deployments). 3. **Prune stage:** drop everything below a relevance floor *and* enforce a hard token budget, whichever binds first. The reranker gives you something the vector index cannot: a **calibrated cutoff**. With raw similarity you are guessing; with cross-encoder scores you can set `min_score = 0.35`, measure precision on your eval set, and move the threshold until false-positive context stops appearing in prompts. In our production builds this step alone routinely removes 60–75% of retrieved tokens while holding answer accuracy flat — and frequently *improving* it, because the model no longer has to reason past ten irrelevant passages to find the one good answer. Two engineering cautions. First, rerank *after* metadata filtering and row-level access enforcement, never before — you do not want to spend ranking compute on chunks the caller cannot legally see. Second, log the scores. The distribution of reranker scores across your traffic is the single best early-warning signal for embedding drift and corpus decay; if the mean relevance score of served chunks is sliding down month over month, your ingestion pipeline is rotting even though nothing is erroring. — ## 3. Semantic Compression: Summarize Before You Generate Pruning removes what is irrelevant. **Semantic compression** shrinks what remains. The insight is that a retrieved passage usually contains one load-bearing fact surrounded by prose, examples, and boilerplate — and the generation model does not need the boilerplate to answer correctly. The pattern is map-reduce summarization applied *inside* the retrieval path. First, group the surviving chunks (optionally by source document or by cluster) and run a cheap summarizer over each group with a strict instruction: *”Preserve every fact, number, name, date, and caveat; remove restatement, examples, and filler. Output at most N tokens.”* Then concatenate the compressed groups into the generation context. Compression ratios of 3:1 to 5:1 are routine on enterprise documents, which are optimized for human skimming, not model ingestion. The design decision that matters is **what gets compressed versus what stays verbatim**. We classify each reranked chunk into tiers: | Tier | Reranker score | Handling | |—|—|—| | High | ≥ 0.7 | **Verbatim.** Quote-bearing, compliance-critical passages go to the prompt untouched. | | Medium | 0.35 – 0.7 | **Compressed.** Summarize to ~25% length, preserving all atomic facts. | | Low | < 0.35 | **Dropped.** Not retrieved at all. | This tiering exists because compression is lossy, and loss is not evenly distributed. A summarizer can faithfully reduce a two-hundred-word eligibility explanation to fifty words; it will occasionally mangle a precise dollar amount or a statutory date. High-value chunks earn the right to full fidelity; medium-value chunks trade a bounded fidelity risk for a 4× token reduction. Quantify that risk on your eval set (see Section 5) before you trust it in production — if compression drops factual-recall on your benchmark by more than ~1–2%, lower the verbatim threshold. Compression also has a **latency tax**: each summarization call is a small LLM round-trip. Amortize it by caching compressed forms keyed on the chunk hash (documents change rarely; their summaries should not be recomputed per query), and by batching — one call compressing three chunks beats three calls compressing one. Done right, the summarization cost lands at 10–20% of the token savings it unlocks, which is the trade you want. --- ## 4. Protocol Layer Optimization: Partial Resources in MCP If you are building RAG on top of Model Context Protocol — as we increasingly do at OnlineCloud — there is a fourth layer most teams miss, and it is the most elegant one. **The bloat can be cut before a single token ever leaves your server.** MCP servers expose data to agents as **resources**, addressed by URI and returned as typed contents — the contract layer between LLMs and your business logic. The naive pattern is to expose a document as one resource: the agent requests `mcp://kb/policy-42`, and the server returns the whole document. That is prompt-stuffing wearing a protocol hat, and it reproduces the linear-cost problem at the transport boundary. The fix is what we call **partial resources**: resource templates that let the client request precisely the span it needs, with the server doing the pruning against the *real* retrieval pipeline rather than the model doing it against a wall of text. Concretely: - **Offset/limit templates.** A resource template `mcp://kb/{doc_id}` accepts a query parameter block — `mcp://kb/policy-42?start=1200&limit=600` — and the server returns only that byte/token span, mirroring HTTP Range semantics. The agent's context-manager, not the model, decides the window. - **Projection queries.** For structured knowledge (Oracle, MS SQL, Snowflake), expose `mcp://kb/{doc_id}?fields=eligibility,effective_date` and let the server emit a row-level-filtered, column-projected view. The database does the pruning; the wire and the model never see the columns the task does not need. - **Scored slices.** For semantic resources, `mcp://kb/{doc_id}?query={question}&top=3&min_score=0.35` runs the retrieval + rerank pipeline *server-side* and returns only the pruned, ranked passages — the exact pipeline from Sections 2 and 3, enforced at the protocol seam. Why this matters architecturally: MCP's design puts the *application* in control of what gets loaded, when, and with what caching. That control is wasted if the only granularity your server offers is "the whole document." Partial resources move context optimization from an emergent property of prompts — unpredictable, unobservable, different for every model — into a deterministic, typed, auditable contract. Every resource read is a log line with the span requested, the spans dropped, and the score threshold applied. Your CFO gets a token-accounting trail; your eval suite gets a stable interface to test against; and your agents stop paying for the middle of documents they never needed. --- ## 5. Cost ROI: Token Savings vs. Accuracy Trade-offs None of this is worth doing if it breaks answers. The discipline is to treat context optimization as an experiment with two measured axes — **cost per resolved query** and **answer quality on a frozen eval set** — and to make the trade-off explicit before shipping. Here is a representative before/after from a production knowledge-base deployment (200–400 token chunks, GPT-4-class model, ~$10/1M input tokens): | Stage | Tokens sent per query | Cost per 1k queries | Latency added | |---|---|---|---| | Baseline (top-20 chunks, no rerank) | ~8,000 | ~$80 | — | | + Rerank & prune (top-5, score floor) | ~2,000 | ~$20 | +40 ms | | + Semantic compression (tiered) | ~800 | ~$8 | +250 ms (cached: ~+30 ms) | | + MCP partial resources | pays only for the span served | ~$6–8 | −network/parse overhead | The 75% token reduction from pruning is nearly free accuracy-wise: cross-encoder rerankers consistently beat bi-encoder top-k on passage-ranking benchmarks, so your *precision* goes up as your *volume* goes down. The honest trade-off lives in the compression tier, where lossy summarization can shave factual recall. The measurement protocol we use: 1. Freeze a labeled eval set of ~500 real queries with expected answer facts. 2. Measure baseline factual-recall and exact-match on served-chunk text. 3. Turn on pruning; confirm recall does not regress beyond noise (±1%). 4. Turn on compression; if factual-recall drops more than 1–2%, raise the verbatim threshold or drop the compression ratio. 5. Compute **net ROI = (tokens saved × blended price) − (rerank + summarization compute)** and track it on a dashboard, per corpus, per quarter. In our deployments the arithmetic lands almost monotonically positive: a system answering 100k queries/month saves $5–8k/month in context tokens against a few hundred dollars of reranker inference — before counting the secondary wins of lower latency, higher throughput per rate-limit, and smaller KV-cache footprints on self-hosted GPUs. The remaining accuracy tax, when it exists at all, sits in the 0.5–1.5% band and is directly tunable. That is a trade you can take to a budget meeting with a straight face. --- ## 6. Implementation: A Python / Pydantic Context Optimizer Here is the optimizer we actually run — compact, typed, and eval-friendly. It implements the full pipeline: rerank-and-prune against a token budget, tiered compression, and deterministic prompt assembly. Pydantic gives us validated boundaries between stages, so a malformed chunk or an out-of-range budget fails loudly at the seam instead of corrupting a prompt. ```python """Dynamic context pruning for RAG: rerank -> budget -> compress -> assemble.””” from __future__ import annotations import hashlib from dataclasses import dataclass, field from typing import Callable, Protocol, Sequence from pydantic import BaseModel, Field, field_validator # ————————————————————————— # # Stage 1: typed data contracts # ————————————————————————— # class RetrievedChunk(BaseModel): “””A candidate passage returned by the vector recall stage.””” doc_id: str text: str tokens: int = Field(ge=0) @field_validator(“text”) @classmethod def reject_empty(cls, v: str) -> str: if not v.strip(): raise ValueError(“chunk text must not be empty”) return v def cache_key(self) -> str: return hashlib.sha256(self.text.encode()).hexdigest() class ScoredChunk(RetrievedChunk): “””A chunk after cross-encoder relevance scoring.””” relevance: float = Field(ge=0.0, le=1.0) class CompressedChunk(BaseModel): “””A chunk after semantic compression, preserving atomic facts.””” doc_id: str text: str source_tokens: int tokens: int verbatim: bool = False class ContextPlan(BaseModel): “””The final, budgeted context handed to the generation model.””” chunks: list[CompressedChunk] total_tokens: int = 0 dropped_count: int = 0 @field_validator(“total_tokens”, mode=”before”) @classmethod def auto_sum(cls, _: int, info) -> int: # pragma: no cover – defensive return sum(c.tokens for c in info.data.get(“chunks”, [])) # ————————————————————————— # # Stage 2: pluggable scorers and summarizers (swap per deployment) # ————————————————————————— # class RelevanceScorer(Protocol): def score(self, query: str, chunk: RetrievedChunk) -> float: … class Summarizer(Protocol): def summarize(self, chunk: ScoredChunk, max_tokens: int) -> str: … @dataclass class BudgetedContextOptimizer: “””Rerank candidates, enforce a token budget, compress, and assemble.””” scorer: RelevanceScorer summarizer: Summarizer max_context_tokens: int = 3_000 min_score: float = 0.35 verbatim_threshold: float = 0.70 compression_ratio: float = 0.25 # medium-tier chunks shrink to 25% compression_cache: dict[str, str] = field(default_factory=dict) # — public API ——————————————————– # def optimize(self, query: str, candidates: Sequence[RetrievedChunk]) -> ContextPlan: “””Full pipeline: score -> prune to budget -> compress -> assemble.””” scored = self._score(query, candidates) pruned = self._prune(query, scored) kept, dropped = self._fit_budget(pruned) compressed = [self._compress(c) for c in kept] return ContextPlan(chunks=compressed, dropped_count=dropped) # — internals ——————————————————— # def _score(self, query: str, candidates: Sequence[RetrievedChunk]) -> list[ScoredChunk]: “””Run the cross-encoder over recall-stage candidates only.””” return [ ScoredChunk(**c.model_dump(), relevance=self.scorer.score(query, c)) for c in candidates ] def _prune(self, query: str, scored: Sequence[ScoredChunk]) -> list[ScoredChunk]: “””Drop chunks below the calibrated relevance floor.””” return [c for c in scored if c.relevance >= self.min_score] def _fit_budget(self, pruned: list[ScoredChunk]) -> tuple[list[ScoredChunk], int]: “””Greedy fill: keep the most relevant chunks until the budget binds.””” ordered = sorted(pruned, key=lambda c: c.relevance, reverse=True) kept, budget = [], self.max_context_tokens for chunk in ordered: if chunk.tokens <= budget: kept.append(chunk) budget -= chunk.tokens return kept, len(pruned) - len(kept) def _compress(self, chunk: ScoredChunk) -> CompressedChunk: “””Verbatim for high-relevance chunks; cached summary for the rest.””” if chunk.relevance >= self.verbatim_threshold: return CompressedChunk( doc_id=chunk.doc_id, text=chunk.text, source_tokens=chunk.tokens, tokens=chunk.tokens, verbatim=True, ) key = chunk.cache_key() summary = self.compression_cache.get(key) if summary is None: budget = max(64, int(chunk.tokens * self.compression_ratio)) summary = self.summarizer.summarize(chunk, max_tokens=budget) self.compression_cache[key] = summary return CompressedChunk( doc_id=chunk.doc_id, text=summary, source_tokens=chunk.tokens, tokens=len(summary) // 4, # ~4 chars/token; replace with a tokenizer ) def assemble(self, plan: ContextPlan, question: str, system: str) -> str: “””Deterministic prompt assembly — the only place the model sees text.””” blocks = [ f”[{c.doc_id} | {‘verbatim’ if c.verbatim else ‘summary’}]\n{c.text}” for c in plan.chunks ] return f”{system}\n\nCONTEXT:\n” + “\n\n—\n\n”.join(blocks) + \ f”\n\nQUESTION: {question}\nANSWER:” def count_tokens(text: str) -> int: “””Bare-bones fallback token estimate (replace with tiktoken in prod).””” return len(text) // 4 “` A few implementation notes worth carrying into your codebase: – **Budget before compression.** Prune to the token budget *first*, then compress what survives. Compressing chunks you are about to drop wastes summarizer spend — the expensive operation in this pipeline. – **Boundaries are Pydantic-validated.** `Relevance` outside [0,1], negative token counts, empty chunk text — all rejected at construction. When the optimizer is fed from a live retrieval service, this catches schema drift at the seam, not as a garbled prompt three hundred tokens later. – **Cache by content hash, not doc_id.** The same passage appearing under two document versions compresses once. Invalidation is trivial: corpus re-ingestion bumps the hash. – **Return the plan, not just the string.** `ContextPlan` carries `dropped_count` and per-chunk provenance so your observability layer can log *why* each prompt looks the way it does — the audit trail your eval and your finance team both need. – **The scorers and summarizers are protocols.** In production these are a Cohere/BGE reranker over HTTP and a small summarization model; in tests they are stubs returning fixed values. The optimizer itself never changes when the model vendors do. — ## Conclusion: Prune at Every Layer, Measure at Every Step Token optimization in RAG is not a single knob — it is a stack of cuts, each with its own cost and its own risk. Prune aggressively at retrieval with a calibrated reranker and you remove the 60–75% of context that was never going to help. Compress what remains, tiered by relevance, and you shrink the final prompt another 60% at a bounded, measurable fidelity cost. Push the pruning into the protocol layer with MCP partial resources and the savings become deterministic and auditable instead of prompt-dependent. Then close the loop with an ROI measurement that tracks tokens saved against factual recall — because an optimization you cannot measure is a regression you cannot see. The systems that survive contact with production are the ones where every token that reaches the model has earned its place: scored, budgeted, compressed, and accounted for. That discipline is exactly what we build at OnlineCloud — evaluated, observable RAG and agentic systems wired into your real data estate, from Oracle and MS SQL to Postgres and Snowflake, delivered by senior engineers in 6–10 weeks. **Ready to cut the token waste in your retrieval pipeline?** Talk to the OnlineCloud architects. We will map your data, pick the model class, and write the eval that defines “done” — with the cost and accuracy numbers on the table before you commit. [Contact us](https://www.onlinecloud.dev/) or email [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) to start the conversation. </div> <div style="margin-top:var(--wp--preset--spacing--40)" class="wp-block-post-date has-small-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/optimizing-token-usage-in-rag-implementing-dynamic-context-pruning/"><time datetime="2026-09-06T13:17:01+00:00">September 6, 2026</time></a></div> </div> </li><li class="wp-block-post post-248 post type-post status-publish format-standard hentry category-ai-agents"> <div class="wp-block-group alignfull has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--60)"> <h2 class="wp-block-post-title has-x-large-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/securing-the-confused-deputy-identity-propagation-in-agent-chains/" target="_self" >Securing the ‘Confused Deputy’: Identity Propagation in Agent Chains</a></h2> <div class="entry-content alignfull wp-block-post-content has-medium-font-size has-global-padding is-layout-constrained wp-block-post-content-is-layout-constrained">![OnlineCloud — Agentic AI & MCP Engineering Studio](https://www.onlinecloud.dev/assets/hero-company-V8Xps1Ae.png) # Securing the “Confused Deputy”: Identity Propagation in Agent Chains **By the OnlineCloud security architecture team** In 1988, Norman Hardy named the confused deputy problem. His compiler — a trusted utility with the file owner’s privileges — could be tricked into overwriting the billing file, not because it was malicious, but because it held *ambient* authority far beyond its caller’s. The deputy never doubted its integrity; it simply could not tell whose authority it was exercising in any given moment. We have rebuilt that architecture inside our agent chains, and this time the deputy is an LLM. A user says “reconcile last quarter’s invoices,” and one instruction fans out into dozens of tool calls across a database, a CRM, an email system, and a reporting service. The orchestrator authenticates once with a broad service identity, and every downstream tool trusts it unconditionally. A prompt injection buried in one vendor invoice becomes an exfiltration path with database credentials behind it. The blast radius is every system the agent can reach. As a security architect, I’ve watched this failure mode multiply across client engagements. The fix is not to make agents “safer” in the abstract; it is to make the *identity architecture* honest: propagate the human’s identity through every hop, delegate it with narrowing scope, and audit every link. This post walks through exactly that — OAuth 2.1 scoped delegation, hop-by-hop least privilege, forensic auditability, and a reference architecture built on the Model Context Protocol (MCP). ## 1. The Confused Deputy in Autonomous Multi-Step Tasks The confused deputy had three ingredients: a privileged program, a less-privileged caller, and a design in which the program’s authority did not depend on the caller’s. Hardy’s insight: authority should travel with the *request*, not sit on the *process*. Capability-based systems were the proposed cure; industry standardized on ambient principals instead — a workable trade-off while the deputy was a compiler running as one user on one machine. Autonomous agents broke that trade-off in three ways: **First, one identity now spans many systems.** The orchestrator holds a token (or worse, a long-lived service-account key) that grants access to every tool in the fleet. A “read CRM notes” step that reaches the “delete from Snowflake” endpoint is indistinguishable to the database — the deputy’s authority is the caller’s, full stop. **Second, the caller is an unreliable narrator.** Prompt injection is the normal adversarial condition of LLM tool loops: any hop that concatenates untrusted content into a prompt can steer subsequent tool calls. One successful injection inherits everything. **Third, the chain is deep.** Host → orchestrator → planner → tool server → data connector spans four or five trust boundaries; context about the human erodes at each one — unless identity is explicitly carried. The discipline that fixes all three is the one Hardy pointed at: **never let a deputy act on authority it cannot attribute to a caller.** Every tool invocation must answer “on whose behalf, through what path, and with what scope?” That takes three mechanisms — identity propagation (who), scoped delegation (how authority passes), and least-privilege enforcement at each hop (what can be done) — with audit making all three verifiable. ## 2. Identity Propagation: From Service-Account Auth to User-Scoped Tokens Service accounts were never a security strategy; they were a *convenience* strategy — one credential, one token refresh, no per-user state. The costs were merely deferred: – **No attribution.** “The service account deleted the row” is not an answer to “who asked for this?” – **No granular revocation.** A compromised session revokes the whole agent, not one user. – **Unbounded blast radius.** The principal is scoped to the *union* of everything the agent might ever need — nobody wants to re-issue credentials when the agent gains a tool. – **A confused deputy by construction.** The deputy’s authority is fixed and ambient; it is not tied to any caller at all. The remedy is user-scoped, propagating identity — the “on-behalf-of” pattern behind Kerberos constrained delegation, Windows S4U, and cloud role-chaining. The shape is always the same: 1. The **user authenticates at the host** (OIDC + PKCE), proving *who* they are. 2. The host exchanges that identity for an **access token bound to the orchestration session**, carrying the user as `sub`, the orchestrator as `aud`, and a tight `exp`. 3. Every downstream hop receives a token that **preserves the user’s subject** while naming the current actor — the CRM sees “user U, acting through orchestrator O,” never just “the agent.” In practice, access tokens should be JWTs (RFC 9068) carrying at minimum `sub` (the human), `iss`/`aud`, `exp`/`iat`, and a unique `jti`. Propagation is not copying one fat token everywhere — it is *re-deriving* a token at each boundary so context (subject, actor, audience, scope) is explicit and verifiable. Reserve true service identity for genuinely headless work — scheduled batch jobs with no user in the loop — and even then narrow it to the job, mint it short-lived, and log every use. The rule is simple: **ambient authority is a bug; attributed authority is a feature.** ## 3. Scoped Delegation: OAuth 2.1 Token Exchange for Agent Swarms Propagation gives you a subject that survives the trip; delegation gives you a way to *pass* authority without widening it. For agent swarms, the mechanism is **token exchange**, standardized in RFC 8693 and sitting cleanly on top of OAuth 2.1. OAuth 2.1 is the consolidation of everything we learned the hard way: PKCE required for public clients, the implicit and password grants gone, per-client redirect URI registration, refresh-token rotation. It matters here because it assumes *short-lived, audience-bound, single-purpose* tokens as the baseline — exactly what a multi-hop agent chain needs. RFC 8693 adds a new grant type, `urn:ietf:params:oauth:grant-type:token-exchange`, where a client presents a `subject_token` (the authority it holds) and optionally an `actor_token` (who is causing the exchange), and requests a *new* token for a specific `audience` with a specific `scope`: “`json POST /token HTTP/1.1 Host: idp.onlinecloud.dev Content-Type: application/x-www-form-urlencoded grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Atoken-exchange &subject_token=eyJhbGciOiJSUzI1NiIs… &subject_token_type=urn%3Aietf%3Aparams%3Aoauth%3Atoken-type%3Aaccess_token &actor_token=eyJhbGciOiJSUzI1NiIs… &actor_token_type=urn%3Aietf%3Aparams%3Aoauth%3Atoken-type%3Aaccess_token &audience=https%3A%2F%2Fmcp.crm.internal &scope=crm%3Aread%3Aorders%20crm%3Aread%3Acustomers &requested_token_type=urn%3Aietf%3Aparams%3Aoauth%3Atoken-type%3Aaccess_token “` The response is a fresh, short-lived token whose claims encode the delegation: `sub` remains the original user, `act` names the immediate actor, and `aud`/`scope` are **narrowed** to the next tier. A JWT-delegation chain records itself: token *n* carries the actor that minted token *n−1*, so the full path from human to deepest tool is reconstructible from any single token. For an agent swarm, I enforce five non-negotiable rules on every exchange: 1. **Monotonic narrowing.** An exchange may only reduce scope or keep it equal — never widen. If a request asks for more than it holds, the authorization server denies it. This single invariant stops a compromised mid-tier hop from escalating. 2. **Audience is a hard boundary.** Each token names exactly one audience (an MCP server URI, per RFC 8707 resource indicators). A CRM token presented to the database connector is rejected before any code runs. 3. **Explicit actor.** A chain of `act` claims is the difference between “the user did this” and “the user asked an agent, which asked another agent, which did this.” 4. **Delegation depth limits.** The authorization server stamps each token with a delegation counter and refuses to exchange beyond a configured depth (we run orchestrators at 2–3). This kills recursion and cycles. 5. **Short TTLs, no refresh across boundaries.** Hop tokens live minutes, not hours. A refresh token never crosses a trust boundary; each tier re-authenticates to the policy layer rather than trusting an inherited credential. ## 4. Enforcing Least-Privilege at Each Hop in the Tool Chain Propagation and delegation make identity *available* at the deepest tier; least privilege makes it *safe* there. A propagated token with unrestricted scope is just a faster confused deputy. The discipline is to treat scope as a property of the *tool call*, not of the *session*. The MCP authorization model was designed with exactly this in mind: the spec’s authorization flow builds on OAuth 2.1, uses resource indicators so each token is bound to one MCP server’s URI, and supports per-tool authorization decisions at the server. That gives you a concrete enforcement lattice for a typical chain: | Hop | Audience (MCP server) | Example scope | TTL | |—|—|—|—| | Host → Orchestrator | `orchestrator.internal` | `session:plan`, `session:checkpoint` | 15 min | | Orchestrator → CRM tool | `mcp.crm.internal` | `crm:read:orders`, `crm:read:customers` | 5 min | | Orchestrator → Fulfillment tool | `mcp.fulfillment.internal` | `fulfill:ship:create` | 5 min | | CRM tool → Warehouse DB | `pg.warehouse.internal` | `db:select:orders` (row-level filtered) | 5 min | Note what is *not* in that table: a token that can read the CRM cannot write to it, cannot touch fulfillment data, and the database connector never sees a token wider than `db:select` on orders — even though the human could authorize far more at the host. Each exchange produced a narrower token, so each hop’s worst case is bounded by its own scope. Three enforcement details matter in practice: – **Enforce at the server, not the client.** Token introspection before every tool execution, verifying `aud` matches the server’s own URI, `exp` is live, and the requested tool is in scope. Client-side filtering is convenience; server-side checks are security. – **Push subject claims into the data layer.** This is where OnlineCloud’s row-level access policies pay off: the connector maps token claims (`sub`, group memberships) onto row-level security predicates in Oracle, MS SQL, Postgres, or Snowflake. The database refuses rows the user may not see — the agent cannot retrieve what the user could not, even if it *wants* to. – **Deny by default.** A tool not explicitly in scope is refused. Scope drift requires a code review and a policy change, never an ad-hoc token with a wider `scope`. The mental model for every engineer on the chain: **each hop is a deputy with a fixed, caller-derived mandate. If a tool cannot state whose authority it is using and what subset of that authority applies, it should not execute.** ## 5. Audit: Replaying Propagation Logs for Forensic Analysis Least privilege limits the damage a confused deputy can do; audit ensures you can find out what happened when one does. The requirement is not “we log tokens” — it is **forensic replayability**: the ability to take any observed tool call and reconstruct the full delegation path that produced it. That demands a propagation log at every exchange point. Each event should carry: “`json { “event”: “token_exchange”, “ts”: “2026-09-06T14:32:07.418Z”, “trace_id”: “3f9c1a…”, “parent_jti”: “a1b2…”, “issued_jti”: “c3d4…”, “subject”: “user:uid-4471”, “actor_chain”: [“orchestrator:ord-882”, “planner:agent-12”], “audience”: “mcp.crm.internal”, “granted_scope”: [“crm:read:orders”], “requested_scope”: [“crm:read:orders”, “crm:write:orders”], “decision”: “narrowed”, “exp”: 300 } “` Linked by `jti` (parent → issued) and a `trace_id` surviving across HTTP hops, these events form a tree you can walk in either direction. Replaying a log means answering four questions: 1. **Who authorized this?** Walk the `actor_chain` to the root `subject`. 2. **Through what path?** Rebuild the token lineage — via the sanctioned orchestrator, or via a sideways delegation from a sibling agent? 3. **Was scope honored?** Every exchange records requested vs. granted scope. A run of “narrowed” decisions is healthy; a single “widened” decision is an incident. 4. **Was anything replayed?** `jti` + `aud` + `exp` reveal the same token presented to two audiences, or after its window closed. Replay is the signature of a stolen token — and it should page someone. For the logs to be trustworthy evidence, apply tamper-evident practices: append-only storage, hash-chaining each record to its predecessor, and WORM retention that outlives your token TTLs by months. Because an agent chain spans services, ship every propagation event to a single audit bus with the shared `trace_id` — per-service logs that cannot be correlated are, forensically, not logs at all. This is the “audited end-to-end” bar we hold every MCP deployment to: if you cannot replay the delegation path behind a given invoice read on demand, the identity architecture is unfinished. ## 6. Reference Architecture: Identity Flow from Host to Deep-Tier MCP Tools Putting it together, this is the identity flow we ship for production agent chains — user at the top, row-level-secured data connectors at the bottom, every boundary enforcing the rules above: “` ┌──────────────────────────────────────────────────────────────────┐ │ HOST (chat / IDE / portal) user authenticates (OIDC+PKCE) │ └───────────────┬──────────────────────────────────────────────────┘ │ identity token → exchange #1 (aud=orchestrator) ┌───────────────▼──────────────────────────────────────────────────┐ │ AUTHORIZATION SERVER / POLICY LAYER (RFC 8693 exchange, │ │ introspection, delegation-depth, scope-narrowing policy) │ └───────┬──────────────────┬───────────────────────┬───────────────┘ │ exchange #2 │ exchange #2 │ exchange #2 ┌───────▼────────┐ ┌──────▼─────────┐ ┌───────▼──────────────┐ │ ORCHESTRATOR │ │ PLANNER / │ │ MCP GATEWAY │ │ agent (depth1) │ │ sub-agents │ │ (broker, SSE/HTTP) │ └───────┬────────┘ └──────┬─────────┘ └───────┬──────────────┘ │ exchange #3 (aud=MCP server URI, narrow scope) ┌───────▼──────────────────────────────────────────▼──────────────┐ │ TIER-1 MCP SERVERS per-tool authz, introspection, rate limits│ └───────┬───────────────────────────────┬─────────────────────────┘ │ exchange #4 (aud=connector) │ exchange #4 ┌───────▼──────────────┐ ┌───────▼─────────────────────────┐ │ CRM / ERP connectors │ │ DATA CONNECTORS: Oracle, MS SQL,│ │ (user-scoped tokens) │ │ Postgres, Snowflake — row-level │ └──────────────────────┘ │ security from propagated claims │ └─────────────────────────────────┘ every hop → AUDIT BUS (jti-linked propagation events, trace_id) “` Walking the flow: the **host** authenticates the human and holds the only long-lived session. The **authorization server** is the single place that understands delegation policy — it narrows scope, binds audience, checks depth, records every exchange. The **orchestrator** never sees the user’s host token; it receives a derived token naming the user as `sub` and itself as `aud`. When it invokes a tool, the **MCP gateway** brokers the call over streamable HTTP or SSE; the **tier-1 MCP server** introspects the presented token, verifies audience and tool scope, then enforces its own per-tool policy. Deep connectors — Oracle, MS SQL, Postgres, Snowflake — receive narrower tokens still and map propagated claims onto row-level security predicates. If any single hop is compromised — an injected prompt, a buggy tool, a stolen token — the damage is bounded by that hop’s scope, the subject remains attributable to a human, and the full path is reconstructible from the audit bus. Two deployment notes. First, the authorization server’s policy tables are the *real* product: scope catalogs per tool, delegation-depth ceilings, and audience allow-lists should be versioned and reviewed like firewall rules. Second, make failure loud: exchange denials, scope-narrowing decisions, and replay detections should be observable metrics — a confused deputy that fails silently is one you will meet again in the incident report. ## Build This Into Your Agent Stack Identity propagation is not a compliance checkbox; it is the difference between an agent that can safely touch production data and one that *is* a production incident waiting for a prompt injection. At OnlineCloud, we design MCP servers around real auth boundaries — stateless servers with exchange-based auth, rate limits, and jti-linked audit logs out of the box, wired to Oracle, MS SQL, Postgres, Snowflake, and the CRMs your team actually uses. We ship evaluated, observable agent systems in 6–10 weeks, not quarters — and every one can answer “whose authority ran this tool call, and why?” If you are wiring autonomous agents to real business data, let’s talk about the identity architecture before the agents do. **[Contact OnlineCloud](https://www.onlinecloud.dev/)** — free initial consultation, US-based senior engineers, response within 24 hours. Or email us directly at **contact@onlinecloud.dev**. </div> <div style="margin-top:var(--wp--preset--spacing--40)" class="wp-block-post-date has-small-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/securing-the-confused-deputy-identity-propagation-in-agent-chains/"><time datetime="2026-09-06T13:17:01+00:00">September 6, 2026</time></a></div> </div> </li><li class="wp-block-post post-244 post type-post status-publish format-standard hentry category-ai-agents"> <div class="wp-block-group alignfull has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--60)"> <h2 class="wp-block-post-title has-x-large-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/designing-a-self-healing-agent-loop-automated-error-recovery-and-retries/" target="_self" >Designing a Self-Healing Agent Loop: Automated Error Recovery and Retries</a></h2> <div class="entry-content alignfull wp-block-post-content has-medium-font-size has-global-padding is-layout-constrained wp-block-post-content-is-layout-constrained"># Designing a Self-Healing Agent Loop: Automated Error Recovery and Retries *When the code path is generated per-run by a stochastic model, your retry policy can’t be a static snippet. Here is how an SRE makes an agent classify failures, recover from them, and stop amplifying outages.* ![Self-healing agent loop error recovery engineering at OnlineCloud](https://www.onlinecloud.dev/assets/hero-profile-BlpwYfK_.png) ## 1. Introduction: The Fragility of Autonomous Agents in a Brittle API World Every engineer who has run an agent against real APIs knows the death scene: the tool times out, and the agent calls it again with the same arguments, again and again, while the upstream stays down. Nobody wrote a retry loop; the model *became* it. A deterministic service survives production entropy because its error handling was written once, in code: backoff, jitter, idempotency keys, a circuit breaker. An agent has none of that by default. Its “error handling” is improvised at runtime, under token pressure, by a model that treats every failure as a puzzle rather than a signal to stop. **In an agentic system, the retry policy is generated per-run by the thing that is already failing.** Production APIs punish this: p99 spikes, 429 rate limits, 5xx during deploys, 2 a.m. maintenance resets. The failure shapes are consistent: the **retry spiral** (identical calls until a ceiling or a human intervenes), the **confident drift** (after two errors the model stops calling tools and invents answers), and the **cascade** (one poisoned result invalidates every downstream step). These failures are behavioral, not technical — the system returns success while doing the wrong thing, or quietly burns budget. The fix is not to prompt the model to “be more careful.” It is to move error handling out of the context window and into the execution machinery around the model: **classify the failure, pick a recovery strategy, and enforce limits the model cannot override.** That is the same architectural move the Model Context Protocol made for tooling: taking invariants out of prose and putting them into structure. A self-healing loop is what you get when the runtime cares about failure as much as the model cares about the task. ## 2. Error Classification: Model Errors vs. Transient API Failures You cannot heal what you cannot classify — and you cannot classify with a model. Asking an LLM “what went wrong?” on a 500 costs tokens and produces prose instead of policy. **Classification belongs at the tool boundary, in the runtime**, where every tool wrapper returns a structured error envelope before the model ever sees the failure: “`json { “error”: { “class”: “transient”, “code”: “RATE_LIMITED”, “retryable”: true, “retry_after_ms”: 30000, “tool”: “crm.account.update”, “hint”: “honor Retry-After; do not regenerate arguments” } } “` Three classes cover nearly every production failure, and each maps to one family of recovery: – **Transient API failures** — 429s, 5xx, timeouts, resets. A retry *can* succeed: the request was fine, the infrastructure was not. Recovery is deterministic — exponential backoff with jitter, honor `Retry-After`, retry the identical payload under an idempotency key. Never ask the model to “improve” a request that was never the problem. – **Contract failures** — 400/422 schema rejections, missing resources, expired auth. Also deterministic, in the opposite direction: retrying an identical payload is pure waste. Recovery means repairing the plan — fix the arguments, refresh the credential — then re-executing; a contract failure that recurs after repair is a stop-and-escalate signal. – **Model errors** — wrong tool chosen, arguments that pass schema but violate business semantics, a plan that contradicts verified results. Not a network class; retrying is meaningless because the same model repeats the same decision. This is the expensive class, because detecting it needs a **verification step** — a read-only check that output matches the goal — not a status code. Two rules keep it honest. Classify on code and envelope, never payload heuristics: a 500 with an `UPSTREAM_TIMEOUT` body is transient; a 200 with an empty result may be a model error. Treat the classifier as policy, not model judgment — version it with the tool schema, change it without shipping a prompt. ## 3. Recovery Patterns: Backtrack, Plan-Repair, and Tool-Switching With errors classified, the executor dispatches to one of three patterns from a policy table. They are deliberately few: the runtime must reason about them, and the model must be constrained by them — let the model choose among *visible, eligible* actions; let the runtime decide what is visible and eligible. **Backtrack — rewind, don’t replay.** Every completed step appends to a checkpoint: plan state, tool result, idempotency key. When an error invalidates the branch, the executor rewinds to the last validated checkpoint — you checkout a known-good state instead of performing surgery on a broken one. The SRE detail that makes this safe: **rewinding context does not unwind side effects.** An order created before the backtrack does not un-create itself. Every side-effecting tool must be idempotent (keyed by step) or compensated (a paired cancel call), or backtracking is just a fancier way to corrupt data. **Plan-Repair — amend, don’t regenerate.** For contract failures and shallow model errors, the cheapest correct recovery is a surgical amendment of the *remaining* plan — fix the argument that failed validation, drop the step that contradicts verified results. Repair is bounded to two or three rounds with a token budget: an unbounded repair loop is the retry spiral in a nicer jacket. The model may only edit **unexecuted** steps (executed history is immutable, keeping the decision log replayable), and repaired steps pass the same schema and policy validation as the original. **Tool-Switching — capability pluralism.** The same capability — update an account, enqueue a job — usually lives behind several tools: a synchronous REST tool, a queue-backed async variant, a read-only verification view. The loop needs a **fallback registry**: an ordered list of substitutes per tool, maintained by policy, not improvised mid-crisis. When the primary tool’s breaker is open or its retries are exhausted, the executor promotes the next tool in the registry. The model never “decides” to switch — the runtime simply filters the unavailable tool from its toolset. Switching becomes routing with deterministic preferability. The dispatch loop tying the patterns together: “` while plan.has_steps() and run_budget.remaining(): result = executor.call(step.tool, step.args, idempotency_key=step.key) if result.ok: checkpoint(step, result); continue error = classify(result) if error.transient and step.attempts < max_retries: # backoff retry sleep(backoff(step.attempts, jitter=True)); step.attempts += 1; continue if breaker.is_open(step.tool): # tool-switching step.tool = policy.fallback_for(step.tool) if step.tool is None: escalate(error); break step.attempts = 0; continue if error.contract or error.model: # plan-repair plan, repaired = repair_plan(plan, error, repair_budget=2) if not repaired: # backtrack plan = last_valid_checkpoint().plan ``` The property that makes this self-healing rather than self-harming: **every path terminates** — retries have a max, repairs have a budget, backtracks land on validated state, and exhaustion ends in a structured escalation: a human handoff, never a silent success. ## 4. Circuit Breakers: Preventing Agentic Retry Storms on Downed Upstreams A single agent retrying a downed API is a nuisance; a fleet of them is an outage amplifier. When the upstream goes down, every concurrent run discovers it at once and starts its own retry campaign against a service that cannot recover under the load they generate. No individual agent misbehaves, yet the aggregate is a self-inflicted DDoS with a token bill attached: the **agentic retry storm**. The arithmetic is brutal. An upstream sheds load with 429s and a 30-second `Retry-After`; a disciplined client retries every 30 seconds, while a model improvising its own policy retries in two — fifteen times the request rate. Across a hundred concurrent runs, that is a 1,500× amplification of the exact load the outage was designed to shed. The breaker makes disciplined backoff an *enforced property of the runtime*, not a suggestion. Agentic breakers follow the classic tri-state pattern with specifics bolted on: - **Per-tool and per-upstream.** State is keyed by tool name and underlying endpoint so one failing tool doesn't poison unrelated ones. Open on five consecutive transient errors or a sustained error rate above ~40%. - **Open, half-open, closed.** While open, calls are refused immediately and the **planner never sees the tool** — it is filtered from the toolset, so the model spends zero tokens proposing calls the breaker rejects. After backoff, a half-open breaker admits one probe: success closes it, failure re-opens it longer. - **Fleet-wide, not per-pod.** In-process state means every instance trips its own breaker while the fleet keeps hammering. Share state via Redis or etcd so the platform presents one breaker per tool, and stagger half-open probes with jitter. - **Break on the right class.** Open on *transient* failures only. Contract and model errors will never succeed on retry; tripping on them only masks a broken schema or a drifting model behind "service unavailable." The same logic extends to the model layer: when a provider degrades in one region, break to a secondary model or region. The breaker does not care whether the thing being hammered is an ERP endpoint or an inference API — and because it enforces limits the model cannot see past, a behavioral problem becomes a solved infrastructure problem. ## 5. Observability: Tracking Recovery Success Rates and "Time to Heal" A self-healing loop that is not instrumented is just a loop with extra steps — you cannot tell whether it is healing errors or converting them into confident wrong answers. Recovery observability adds two metrics to decision logging; both deserve SLOs. **Recovery success rate (RSR)** — the share of erroring steps that resolve to a verified successful outcome without human intervention, broken down by error class and healing pattern. RSR is the loop's health signal: a falling transient-class RSR tracks upstream fragility and miscalibrated retry budgets; a rising model-error RSR means the loop heals model mistakes so efficiently that your eval suite stops noticing the model is drifting. Track per class, per tool, per prompt version — alert on deltas, not absolutes. **Time to heal (TTH)** — wall-clock from first failed attempt on a step to that step completing successfully, or to human handoff. TTH is the direct cost of the uncertainty the loop absorbs and the best single number for comparing agent designs: a retry healed in 400 ms is noise; a model error needing four replans and a backtrack is a design finding. Track p50 and p95 and gate releases on them like latency. Every recovery decision emits an event that makes the healing replayable: ```json { "ts": "2026-09-06T09:14:02.118Z", "trace_id": "7c2e…", "step": 7, "tool": "crm.account.update", "error_class": "transient", "error_code": "UPSTREAM_TIMEOUT", "recovery": { "pattern": "retry_then_switch", "attempts": 2, "fallback_tool": "crm.account.update.async" }, "healed": true, "ttheal_ms": 5400, "cost_usd": 0.031 } ``` Build supporting panels around them — retry counts by error code, repair-budget exhaustion, breaker opens per tool — plus the chart most teams skip: **cost of recovery versus cost of starting over.** When a plan is fourteen steps deep and three are poisoned, healing can cost more than killing the run and restarting fresh. Then write SLOs that hold the loop accountable. The ones we run in production: **≥ 95% of transient errors heal automatically with p50 TTH under five seconds; ≥ 90% of contract errors heal via plan-repair; escalation under 2% of erroring steps; zero silent successes after an exhaustion path.** Alert on RSR falling below target or TTH p95 breaching — never on raw error counts, because a healthy healing loop is supposed to see errors. ## 6. The OnlineCloud Standard: Building Agents That Survive Production Entropy None of this is exotic, and that is the point. A self-healing agent is not a smarter model; it is a runtime that treats the model as the least reliable component and engineers around it. The discipline reduces to a short list — the one we hold every engagement to: **Typed tools with clean error semantics** — every wrapper classifies failures into the envelope before the model sees them. **Idempotency as a precondition** — without an idempotency key, retries and backtracking are off the table. **Recovery policy as data** — retry budgets, fallback registries, breaker thresholds, and repair caps live in versioned configuration, never improvised in a system prompt. **Limits the model cannot override** — breakers, per-run dollar ceilings, and human checkpoints on irreversible actions are enforced in the runtime. **Instrumented from day one** — RSR, TTH, and escalation rate ship with the first vertical slice, not the postmortem. One practice separates evaluated systems from demos: **failure injection.** Part of the eval suite that defines "done" is a chaos pass — kill the upstream mid-run, ratelimit the agent, return a schema-valid but poisoned response, rotate a credential — and the agent must heal within its SLOs. If it cannot survive a simulated outage in week three, it will not survive a real one in month six. That is the difference between an agent that demos well and one your operations team can sleep through: the second has already failed, in a controlled environment, and gotten better at it. Production entropy is not a bug to be eliminated; it is the environment. Upstreams degrade, schemas drift, credentials rotate, and models change their minds. The agents worth running are designed for that world — classifying errors, healing within budget, escalating cleanly when they cannot. At OnlineCloud, we build those agents: US-based senior engineers shipping evaluated, observable agentic systems against Oracle, MS SQL, Postgres, Snowflake, and the CRMs and APIs your teams use every day, typically in 6–10 weeks, with dashboards and on-call owned by the people who built it. If your agents are one 429 away from a thousand-dollar spiral, [contact the OnlineCloud team](https://www.onlinecloud.dev/) and bring the workload you are tired of babysitting — we will show you a loop that heals itself. </div> <div style="margin-top:var(--wp--preset--spacing--40)" class="wp-block-post-date has-small-font-size"><a href="https://blog.onlinecloud.dev/2026/09/06/designing-a-self-healing-agent-loop-automated-error-recovery-and-retries/"><time datetime="2026-09-06T13:17:00+00:00">September 6, 2026</time></a></div> </div> </li></ul> <div class="wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--60)"> </div> <div class="wp-block-group alignwide has-global-padding is-layout-constrained wp-block-group-is-layout-constrained"> <nav class="alignwide wp-block-query-pagination is-content-justification-space-between is-layout-flex wp-container-core-query-pagination-is-layout-4dea2dca wp-block-query-pagination-is-layout-flex" aria-label="Pagination"> <a href="https://blog.onlinecloud.dev/author/sandeepsingh/page/4/" class="wp-block-query-pagination-previous">←Previous Page</a> <div class="wp-block-query-pagination-numbers"><a class="page-numbers" href="https://blog.onlinecloud.dev/author/sandeepsingh/">1</a> … <a class="page-numbers" href="https://blog.onlinecloud.dev/author/sandeepsingh/page/3/">3</a> <a class="page-numbers" href="https://blog.onlinecloud.dev/author/sandeepsingh/page/4/">4</a> 5 <a class="page-numbers" href="https://blog.onlinecloud.dev/author/sandeepsingh/page/6/">6</a> <a class="page-numbers" href="https://blog.onlinecloud.dev/author/sandeepsingh/page/7/">7</a> … <a class="page-numbers" href="https://blog.onlinecloud.dev/author/sandeepsingh/page/13/">13</a></div> <a href="https://blog.onlinecloud.dev/author/sandeepsingh/page/6/" class="wp-block-query-pagination-next">Next Page→</a> </nav> </div> </div> </main> <footer class="wp-block-template-part"> <div class="wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained" style="padding-top:var(--wp--preset--spacing--60);padding-bottom:var(--wp--preset--spacing--50)"> <div class="wp-block-group alignwide is-layout-flow wp-block-group-is-layout-flow"> <div class="wp-block-group alignfull is-content-justification-space-between is-layout-flex wp-container-core-group-is-layout-cf54d0a6 wp-block-group-is-layout-flex"> <div class="wp-block-columns is-layout-flex wp-container-core-columns-is-layout-794e3cfa wp-block-columns-is-layout-flex"> <div class="wp-block-column is-layout-flow wp-block-column-is-layout-flow" style="flex-basis:100%"><h2 class="wp-block-site-title"><a href="https://blog.onlinecloud.dev" target="_self" rel="home">Online Cloud</a></h2> </div> <div class="wp-block-column is-layout-flow wp-block-column-is-layout-flow"> <div style="height:var(--wp--preset--spacing--40);width:0px" aria-hidden="true" class="wp-block-spacer"></div> </div> </div> <div class="wp-block-group is-content-justification-space-between is-layout-flex wp-container-core-group-is-layout-bf19e0b3 wp-block-group-is-layout-flex"> <nav class="is-vertical wp-block-navigation is-layout-flex wp-container-core-navigation-is-layout-831b2db5 wp-block-navigation-is-layout-flex" aria-label=" 2"><ul class="wp-block-navigation__container is-vertical wp-block-navigation"><li class="wp-block-navigation-item wp-block-navigation-link"><a class="wp-block-navigation-item__content" href="#">Blog</a></li><li class="wp-block-navigation-item wp-block-navigation-link"><a class="wp-block-navigation-item__content" href="#">About</a></li><li class="wp-block-navigation-item wp-block-navigation-link"><a class="wp-block-navigation-item__content" href="#">FAQs</a></li><li class="wp-block-navigation-item wp-block-navigation-link"><a class="wp-block-navigation-item__content" href="#">Authors</a></li></ul></nav> <nav class="is-vertical wp-block-navigation is-layout-flex wp-container-core-navigation-is-layout-831b2db5 wp-block-navigation-is-layout-flex" aria-label=" 3"><ul class="wp-block-navigation__container is-vertical wp-block-navigation"><li class="wp-block-navigation-item wp-block-navigation-link"><a class="wp-block-navigation-item__content" href="#">Events</a></li><li class="wp-block-navigation-item wp-block-navigation-link"><a class="wp-block-navigation-item__content" href="#">Shop</a></li><li class="wp-block-navigation-item wp-block-navigation-link"><a class="wp-block-navigation-item__content" href="#">Patterns</a></li><li class="wp-block-navigation-item wp-block-navigation-link"><a class="wp-block-navigation-item__content" href="#">Themes</a></li></ul></nav> </div> </div> <div style="height:var(--wp--preset--spacing--70)" aria-hidden="true" class="wp-block-spacer"></div> <div class="wp-block-group alignfull is-content-justification-space-between is-layout-flex wp-container-core-group-is-layout-2ab8c7fb wp-block-group-is-layout-flex"> Twenty Twenty-Five Designed with <a href="https://wordpress.org" rel="nofollow">WordPress</a> </div> </div> </div> </footer> </div> <script type="speculationrules"> {"prefetch":[{"source":"document","where":{"and":[{"href_matches":"/*"},{"not":{"href_matches":["/wp-*.php","/wp-admin/*","/wp-content/uploads/*","/wp-content/*","/wp-content/plugins/*","/wp-content/themes/twentytwentyfive/*","/*\\?(.+)"]}},{"not":{"selector_matches":"a[rel~=\"nofollow\"]"}},{"not":{"selector_matches":".no-prefetch, .no-prefetch a"}}]},"eagerness":"conservative"}]} </script> <script data-wp-router-options="{"loadOnClientNavigation":true}" fetchpriority="low" id="@wordpress/block-library/navigation/view-js-module" src="https://blog.onlinecloud.dev/wp-includes/js/dist/script-modules/block-library/navigation/view.min.js?ver=1bf28ded04f9f188bdcb" type="module"></script> <script id="wp-emoji-settings" type="application/json"> {"baseUrl":"https://s.w.org/images/core/emoji/17.0.2/72x72/","ext":".png","svgUrl":"https://s.w.org/images/core/emoji/17.0.2/svg/","svgExt":".svg","source":{"concatemoji":"https://blog.onlinecloud.dev/wp-includes/js/wp-emoji-release.min.js?ver=7.1.3"}} </script> <script type="module"> /*! This file is auto-generated */ var e="script#wp-emoji-settings",t=document.querySelector(e);if(!(t instanceof HTMLScriptElement))throw new Error("Element missing: "+e);const r=JSON.parse(t.text),s=(window._wpemojiSettings=r,"wpEmojiSettingsSupports"),o=["flag","emoji"];function i(e){try{var t={supportTests:e,timestamp:(new Date).valueOf()};sessionStorage.setItem(s,JSON.stringify(t))}catch(e){}}function c(e,t,n){e.clearRect(0,0,e.canvas.width,e.canvas.height),e.fillText(t,0,0);t=new Uint32Array(e.getImageData(0,0,e.canvas.width,e.canvas.height).data);e.clearRect(0,0,e.canvas.width,e.canvas.height),e.fillText(n,0,0);const r=new Uint32Array(e.getImageData(0,0,e.canvas.width,e.canvas.height).data);return t.every((e,t)=>e===r[t])}function p(e,t){e.clearRect(0,0,e.canvas.width,e.canvas.height),e.fillText(t,0,0);var n=e.getImageData(16,16,1,1);for(let e=0;e<n.data.length;e++)if(0!==n.data[e])return!1;return!0}function u(e,t,n,r){switch(t){case"flag":return n(e,"\ud83c\udff3\ufe0f\u200d\u26a7\ufe0f","\ud83c\udff3\ufe0f\u200b\u26a7\ufe0f")?!1:!n(e,"\ud83c\udde8\ud83c\uddf6","\ud83c\udde8\u200b\ud83c\uddf6")&&!n(e,"\ud83c\udff4\udb40\udc67\udb40\udc62\udb40\udc65\udb40\udc6e\udb40\udc67\udb40\udc7f","\ud83c\udff4\u200b\udb40\udc67\u200b\udb40\udc62\u200b\udb40\udc65\u200b\udb40\udc6e\u200b\udb40\udc67\u200b\udb40\udc7f");case"emoji":return!r(e,"\ud83e\u1fac8")}return!1}function f(e,t,n,r){let a;const s=(a="undefined"!=typeof WorkerGlobalScope&&self instanceof WorkerGlobalScope?new OffscreenCanvas(300,150):document.createElement("canvas")).getContext("2d",{willReadFrequently:!0}),o=(s.textBaseline="top",s.font="600 32px Arial",{});return e.forEach(e=>{o[e]=t(s,e,n,r)}),o}function a(e){var t=document.createElement("script");t.src=e,t.defer=!0,document.head.appendChild(t)}r.supports={everything:!0,everythingExceptFlag:!0},new Promise(t=>{let n=function(){try{var e=JSON.parse(sessionStorage.getItem(s));if("object"==typeof e&&"number"==typeof e.timestamp&&(new Date).valueOf()<e.timestamp+604800&&"object"==typeof e.supportTests)return e.supportTests}catch(e){}return null}();if(!n){if("undefined"!=typeof Worker&&"undefined"!=typeof OffscreenCanvas&&"undefined"!=typeof URL&&URL.createObjectURL&&"undefined"!=typeof Blob)try{var e="postMessage("+f.toString()+"("+[JSON.stringify(o),u.toString(),c.toString(),p.toString()].join(",")+"));",r=new Blob([e],{type:"text/javascript"});const a=new Worker(URL.createObjectURL(r),{name:"wpTestEmojiSupports"});return void(a.onmessage=e=>{i(n=e.data),a.terminate(),t(n)})}catch(e){}i(n=f(o,u,c,p))}t(n)}).then(e=>{for(const n in e)r.supports[n]=e[n],r.supports.everything=r.supports.everything&&r.supports[n],"flag"!==n&&(r.supports.everythingExceptFlag=r.supports.everythingExceptFlag&&r.supports[n]);var t;r.supports.everythingExceptFlag=r.supports.everythingExceptFlag&&!r.supports.flag,r.supports.everything||((t=r.source||{}).concatemoji?a(t.concatemoji):t.wpemoji&&t.twemoji&&(a(t.twemoji),a(t.wpemoji)))}); //# sourceURL=https://blog.onlinecloud.dev/wp-includes/js/wp-emoji-loader.min.js </script> </body> </html>

Author: sandeepsingh

Scaling MCP Servers with Serverless AWS Lambda and SSE

Building a Low-Code MCP Dashboard with React and Tailwind

Approval required