Author: sandeepsingh

# Implementing Rate Limiting and Cost Ceilings in MCP Server Middleware


> Platform Engineer’s field notes on the middleware that keeps an MCP server from spending next quarter’s AI budget before lunch.

## 1. Introduction: The $1,000 Recursive Loop — Why Governance Must Be Architectural

Every MCP team has the same origin story. The agent calls a tool, the tool returns an error, the agent retries with different arguments, the tool errors again — and because each attempt *looks* like progress to the model, the loop never self-terminates. Meanwhile every iteration appends its own output to an ever-growing context window, and per-token pricing quietly multiplies: forty steps of a “three-step task,” each one reading an escalating transcript, adds up to a conversation that bills like a small data center. No malice, no prompt injection, no rogue employee. Just one tool with ambiguous error semantics and no ceiling above it.

That is the $1,000 recursive loop, and the reason it keeps happening is that most teams treat cost as a *reporting* problem. They discover it in the provider’s billing export. Governance that arrives after the invoice is not governance — it is archaeology.

The fix is architectural: put the limits in the path that every model interaction already crosses. In a Model Context Protocol (MCP) deployment, that path is the server. Every tool call — from Claude, GPT-4-class models, or an open-weights deployment in your own VPC — arrives as a JSON-RPC `tools/call` over a streamable HTTP or SSE transport and lands in one dispatch layer. If rate limits, budget ceilings, and audit live in that layer as middleware, then *no client can route around them* — the middleware does not ask nicely. That seam is why we treat MCP as a governance boundary, not another API gateway: done badly, an MCP server is just a slower way to call your database; done well, it is the only place agents cross into your business logic, and it enforces the rules there.

This post is the playbook we use at OnlineCloud when we wire MCP servers to real systems — Oracle, MS SQL, Postgres, CRMs — with real budgets attached. Three layers: **token-weighted rate limiting** so no tenant can monopolize the pipeline, **hard cost ceilings** enforced before a single token is spent, and **observability** that turns spend and throughput into two lines on a live dashboard instead of a surprise in a billing export.

## 2. Token-Based Rate Limiting: Leaky Bucket and Fixed Window per Tenant

Request-level rate limits are the first thing everyone reaches for, and the first thing that fails. Ten requests per second tells you nothing about load when one request asks the model to reason over a 40,000-token context and the next is a two-line classification. Agents amplify the skew: a single planning loop can fire dozens of tool calls in a burst, each carrying a different payload weight. **Rate limiting for agentic traffic has to be denominated in tokens, not requests.**

That means counting what each tool call will plausibly consume before admission — an estimate derived from the arguments it carries and the model class it targets — and charging the tenant’s bucket by that weight. That way, flooding the pipeline with tiny calls and running amok on huge context become the same accounting problem.

Two algorithms dominate for per-tenant control, and they answer different questions:

**Fixed window** is brutally simple: a counter per tenant per window (say, 60 seconds). One atomic increment against a TTL’d Redis key — but with a well-known seam: a tenant that exhausts the window at second 59 can immediately spend a fresh window at second 60, doubling its limit in adjacent bursts.

**Leaky bucket** smooths that out. Tokens drip out of the bucket at a constant refill rate, and a call that would overflow the bucket is rejected (or queued) regardless of where it lands in the clock. Bursts are bounded by bucket capacity; sustained load is bounded by the drip rate. For agentic workloads the leaky bucket is usually the right default: agent loops produce *clusters* of calls, and you want each cluster bounded without penalizing steady background traffic.

A per-tenant leaky bucket in Node.js is small enough to inline:

“`ts
type TenantBucket = { tokens: number; lastRefill: number };

class TokenLeakyBucket {
constructor(
private capacity: number, // max burst, in tokens
private refillPerSec: number, // steady-state drip
) {}

tryAcquire(tenant: string, weight: number, now = Date.now()): boolean {
const b = (this.buckets.get(tenant) ??= {
tokens: this.capacity,
lastRefill: now,
});
// Drip: refill is proportional to elapsed time.
b.tokens = Math.min(
this.capacity,
b.tokens + ((now – b.lastRefill) / 1000) * this.refillPerSec,
);
b.lastRefill = now;
if (b.tokens < weight) return false; // 429 → retry after b.tokens -= weight; // atomic in Redis return true; } private buckets = new Map();
}
“`

In a multi-instance deployment, the same logic moves into a Redis `EVAL` script so the bucket is shared across replicas; the algorithm is identical, only the store changes. What *must not* change is that the token weight is computed in one place — the governance layer — so a tool that fans out a sub-agent or streams a large document gets charged its true weight, not the weight of a cheap request.

| Algorithm | Burst behavior | Smooths load? | Implementation cost | Best for |
|—|—|—|—|—|
| Fixed window | Boundary bursts (2× limit) | No | Trivial (INCR + TTL) | Coarse quotas, alerting thresholds |
| Leaky bucket | Bounded by capacity | Yes | Simple (refill + subtract) | Agentic, bursty per-tenant traffic |
| Token bucket | Bounded, allows credit accumulation | Partial | Simple | APIs with periodic batch jobs |

Whichever you pick, per-tenant state must be keyed by an authenticated tenant identity — never by IP or connection — because an agent session can hop transports and reconnect. And rejections must return a *structured, retryable* error the model can actually handle: an MCP error result with code and a `Retry-After`-style hint, so the agent backs off instead of looping harder. Rate limiting that fails closed with an unparseable error just moves the recursion from your bill to your error logs.

## 3. Cost Ceilings: Real-Time Token Counting and Hard Budgets at the Protocol Layer

Rate limiting answers “how fast may this tenant go?” Cost ceilings answer a different question: “how much may this tenant *cost* before someone human gets involved?” A well-behaved tenant at modest throughput can still burn a shocking daily bill if it is running a wide agent over a long conversation — no individual request was excessive, so throughput limits never see the problem.

The enforcement unit is the token, and the discipline is **check before you spend, not after.** Counting happens in the request path, where the middleware knows three numbers for every call: input tokens, output tokens (or a strict cap), and the price per token for the model actually serving the tenant. For the input side you can count deterministically — arguments are fully known before dispatch — using a tokenizer matched to the model family (the `cl100k_base` / `o200k_base` BPE encodings for the GPT/Claude-class models). For the output side you cannot know the future, so you enforce a *bound*: a per-call `max_tokens` ceiling derived from the tenant’s remaining budget. Every call is then:

1. **Pre-check** — estimate input tokens, take the bounded output allowance, and ask the tenant ledger: *does the remaining budget cover this step?* If not, refuse with a structured error and an escalation path. The agent cannot nickel-and-dime its way past the ceiling, because every call is checked against what *remains*.
2. **Execute** — the call runs with a hard output cap.
3. **Post-charge** — actual tokens observed on the response are debited from the ledger. In streaming mode, charge incrementally: accumulate tokens from the stream as they arrive and abort the stream the moment the step budget is exhausted, so a runaway generation dies mid-stream rather than completing and billing.

The ledger is a per-tenant, per-period budget — `{ tenant, period, model, spentTokens }` in Redis with a daily rollover — and the middleware treats it like a resource limit on a container. Memory limits do not ask the process nicely to stop allocating; the kernel kills it. Your cost ceiling should have the same temperament.

“`ts
interface BudgetGate {
remaining(tenant: string): number; // dollars, this period
reserve(tenant: string, estCost: number): boolean; // pre-check
settle(tenant: string, actualTokens: number): void; // post-charge
}

const budgetGate: BudgetGate = {
reserve(tenant, estCost) {
if (ledger.remaining(tenant) < estCost) { // Hard stop: structured MCP error → agent escalates to a human. throw mcpError("BudgetExceeded", `Tenant ${tenant} has ${ledger.remaining(tenant).toFixed(2)} left ` + `for this estimated ${estCost.toFixed(2)} call`); } ledger.debit(tenant, estCost); // optimistic hold return true; }, settle(tenant, actualCost) { ledger.adjust(tenant, actualCost); // reconcile hold vs. reality }, }; ``` Two properties make this a *hard* ceiling rather than an aspiration. First, the pre-check runs at the protocol layer, in the middleware — not in the tool implementation, and certainly not in the prompt. No tool author can forget to check the budget, because they never see the request until the gate has passed it. Second, ceilings compose: per-call, per-run (an agent loop gets a max-iteration cap and a dollar cap), and per-tenant daily caps. A loop that hits the run cap terminates *gracefully* — a structured handoff to a human checkpoint — not an agent retrying a refund call against an empty ledger. ## 4. Middleware Design: Wrapping MCP Tool Calls with Governance Logic MCP gives you a clean place to stand: the server owns the tool registry, and every client — any agent framework, any host — reaches business logic only through `tools/call`. Wrap that dispatch in a middleware chain and governance becomes a property of the *server*, not of the clients. One chain, in order: **authenticate → rate-limit → budget-gate → audit → invoke → settle**. With the TypeScript MCP SDK the pattern is a higher-order function around tool registration: ```ts import { Server } from "@modelcontextprotocol/sdk/server/index.js"; type GuardedTool = { name: string; costClass: "cheap" | "standard" | "expensive"; // → token weight + price run: (args: unknown, ctx: CallContext) => Promise;
};

export function registerGuarded(server: Server, tool: GuardedTool) {
server.registerTool(tool.name, {
description: tool.description,
inputSchema: tool.inputSchema,
execute: async (args, extra) => {
const tenant = resolveTenant(extra.session); // auth first
rateLimiter.tryAcquire(tenant, weight(tool, args)); // tokens/sec
const reserved = budgetGate.reserve(tenant, estCost(tool, args));
const started = now();
audit.log({ tenant, tool: tool.name, phase: “start”, args }); // decision log
try {
const result = await tool.run(args, { tenant });
const cost = settleFromUsage(result, tool.costClass);
budgetGate.settle(tenant, cost); // debit actual
audit.log({ tenant, tool: tool.name, phase: “done”, cost });
emitMetrics(tenant, tool, cost, now() – started); // → dashboards
return result;
} catch (err) {
audit.log({ tenant, tool: tool.name, phase: “error”, err });
throw err;
}
},
});
}
“`

The same shape ports to Go (`github.com/mark3labs/mcp-go` or the official SDK): a wrapper that takes a tool handler and returns a guarded one, with the chain expressed as composable middleware structs. What matters is not the language but the invariants. **Every tool gets the same chain** — no tool can declare itself exempt, because exemption would just become the bug that produces the incident. **Weights and prices are declared per tool** (`costClass`) so policy lives with the schema, not scattered across handlers. And **every refusal and every charge is an auditable event** with a trace ID, so when finance asks why tenant X was throttled at 14:02, the answer is a query, not a meeting.

Keep the middleware *fast and pure*: state in Redis (shared across instances), synchronous decisions, single-digit milliseconds added to a model round-trip that costs hundreds. One chain, one seam, one place where the rules are enforced and audited.

## 5. Observability: Visualizing Spend vs. Throughput in Real Time

Governance without evidence is a policy document. The third layer makes the first two *visible*: a live view that plots the two variables every platform engineer cares about — **spend** (dollars per minute, per tenant, per tool) against **throughput** (tokens and calls per minute) — with budget ceilings drawn as horizontal lines that you can watch the burn approach.

The metrics come straight out of the middleware in Section 4. Each guarded call emits one event on completion (and one on refusal) with the fields that make the dashboard meaningful:

“`json
{
“ts”: “2026-09-06T14:02:31.004Z”,
“tenant”: “acme-corp”,
“tool”: “orders.lookup”,
“decision”: “allowed”,
“tokens_in”: 1843,
“tokens_out”: 512,
“cost_usd”: 0.0211,
“latency_ms”: 844,
“budget_remaining_usd”: 142.08,
“trace_id”: “8f3a1c…”
}
“`

Feed those events through OpenTelemetry into a Prometheus/ClickHouse-plus-Grafana stack (or any metrics backend you already run), and build four panels:

| Panel | Series | What it tells you |
|—|—|—|
| Spend vs. throughput | $/min (area), tokens/min (line), per-tenant | Are we getting busier or just more expensive? |
| Ceiling burn | Cumulative $ per tenant + ceiling line | Which tenant is sprinting at their budget? |
| Refusals by reason | 429s (rate) vs. budget refusals | Rate refusals → sizing issue; budget refusals → ceiling tuning |
| Step latency & size | p50/p95 latency, tokens/call distribution | Context bloat and slow tools, before they become bills |

The dashboard’s real job is to make *shapes* recognizable. A tenant whose spend line goes vertical while throughput stays flat is running a recursive loop with growing context — the $1,000 story happening live, in time to stop it. A sawtooth of budget refusals every morning means your ceiling is tuned to yesterday’s traffic. Alert on the shapes: spend rate crossing an anomaly band, any tenant above 80% of ceiling, refusal-rate spikes. And keep the panel honest with a **cost-per-outcome** view — dollars per resolved ticket, per order lookup, per deflection — because it is the number the business renews on — and the line that justifies the whole middleware layer in a quarterly review.

## 6. The OnlineCloud Standard: Production-Grade Governance in 6–10 Weeks

None of this is exotic, but all of it is *exacting* — and the difference between a demo and a deployment is whether the middleware exists before the first production tenant connects. At OnlineCloud we ship MCP servers with rate limiting, budget gates, and dashboards as part of the baseline architecture, on a 6–10 week delivery model:

– **Week 1 — discovery.** Map the data systems and auth boundaries, pick the model class per workload, and write the cost model: token weights per tool, prices per model, tenant tiers. If you cannot price the system in week one, you cannot ceiling it in week nine.
– **Weeks 2–7 — vertical slices.** Ship one usable, instrumented slice every 1–2 weeks against real data — middleware chain and dashboard panels included from day one. Governance is not a phase; it ships with the first tool.
– **Weeks 8–10 — harden and hand over.** Load-test the failure paths: a simulated recursive loop dies at the ceiling and pages someone; an over-limit tenant gets clean retryable errors. Hand over the runbooks, alert tuning, and on-call.

That last part is the OnlineCloud difference: the engineers who scope the system push the commits and own the dashboards after launch. We instrument outcomes — accuracy, latency, dollars per request — and share live numbers rather than a slide deck. Evaluated, observable, cost-bounded systems are the deliverable; the middleware is how we keep the promise.

## Conclusion: Make the Ceiling the Architecture

Rate limiting, cost ceilings, and observability are one stance applied three times: *an agentic system should not be able to surprise you.* The token-weighted bucket bounds how fast any tenant can move; the protocol-layer budget gate bounds how much any tenant can spend; the dashboard makes both visible before they become incidents. None of it depends on the model behaving — models will not always behave. Recursive loops, context bloat, and runaway spend are not bugs you can prompt your way out of; they are resource problems that need resource enforcement in the one seam every interaction crosses: MCP server middleware.

If you are standing up MCP against real systems — Oracle, MS SQL, Postgres, Snowflake, your CRM — with real budgets and real tenants, the OnlineCloud engineers build this in from week one: typed tools, auditable transports, per-tenant rate limits and cost ceilings, and dashboards your operations team can sleep through. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) to start the conversation — and bring the workload you are tired of babysitting.

# Fine-Tuning Llama 3 for Tool Selection Accuracy: A Comparative Accuracy Study


## 1. Introduction: Why Prompting Frontier Models Hits a Tool-Call Ceiling

There is a moment every agentic-AI team hits around week six. The orchestration loop is solid, the tool schemas are typed, the eval harness is green — and the model still calls the wrong tool. It reaches for the Oracle `ORDER_UPDATE` endpoint when the user asked for an invoice status. It formats a Salesforce `Lead` insert with `Account` fields. The system prompt is a work of art, few-shot examples have been curated to death, and accuracy still plateaus between 70% and 85% on the trajectory-level metric that predicts production behavior.

This plateau is structural, not a prompt-engineering failure. Frontier models like GPT-4o and Claude 3.5 Sonnet are trained on web-scale general function calling, not on *your* API dialect. They are excellent at guessing what a plausible tool call looks like; they are mediocre at remembering that one Oracle endpoint wants a SOAP envelope while another expects a REST payload. Every token of prompt spent disambiguating your API is budget the model cannot spend reasoning about user intent — and once your surface holds 40, 60, or 200 tools, the few-shot prompt becomes the bottleneck before the model does.

At OnlineCloud, we kept hitting this ceiling building MCP-connected agents for Oracle and Salesforce estates, so we ran a controlled study: instead of fighting the prompt, we fine-tuned an open-weights Llama 3 model on golden tool-call trajectories and benchmarked it head-to-head against GPT-4o and Claude 3.5 Sonnet on the same held-out eval set. This post is the full write-up: methodology, dataset curation, the accuracy numbers, deployment on Bedrock and EKS, and an honest ROI read on when fine-tuning pays for itself.

## 2. Fine-Tuning Methodology: SFT, DPO, and LoRA for API Dialects

Fine-tuning for tool selection is not chat fine-tuning. Chat tuning optimizes for fluent text; tool-selection tuning optimizes one narrow decision — given a conversation state, which tool from a fixed registry, with which arguments, in which serialization? The output grammar is small; the decision boundary is everything.

**Stage one: SFT.** Supervised fine-tuning teaches the skeleton. For a surface of ~60 Oracle and Salesforce operations, we generated roughly 12,000 SFT examples teaching exact tool names, argument casing (`recordTypeId`, not `record_type_id`), required-field sets, and serialization conventions. SFT alone takes a model from “plausible but often wrong” to “mostly right on seen shapes.” What it does not fix is *discrimination* — preferring the correct tool when two candidates are both syntactically valid. That is a preference problem, so we add a second stage.

**Stage two: DPO.** Rather than run full RLHF — reward models, PPO rollout farms, instability — we used Direct Preference Optimization, which reuses the SFT model as its reference and optimizes the policy against paired preferred/rejected trajectories. Our pairs came from production logs: the same request answered by a trajectory that ended in successful tool execution (preferred) versus one that selected the wrong operation or hallucinated arguments (rejected). DPO sharpened exactly the axis SFT leaves blunt, contributing roughly half the total accuracy gain over the base model at a fraction of RLHF’s cost.

**Stage three: LoRA.** Full fine-tuning of a 70B model is a capital project. Low-Rank Adaptation freezes the base weights and trains small adapter matrices (we used rank 16 on attention, rank 32 on MLP, alpha at 2× rank). Our adapter for the 8B model was ~160 MB — a few percent of the base weights — and trained on a single A100-class instance in under 24 hours. Because the base model stays intact, one deployment can host multiple dialect adapters behind a router and swap them per request. That adapter multiplexing drives most of the ROI math in Section 6.

## 3. Dataset Curation: Generating Golden Tool-Call Trajectories for Oracle and Salesforce

Fine-tuning quality is dataset quality, and tool-call datasets are the hardest kind to curate because ground truth is a *procedure*, not a string. We built ours in four layers.

**Schema-derived scaffolding.** For each Oracle EBS and Salesforce operation, we generated synthetic user intents from the operation’s own metadata — description, parameters, typical calling context — yielding thousands of cheap, unambiguous examples. Scaffolding establishes coverage; it does not establish realism.

**Production-log distillation.** The highest-value data came from logged production traffic on existing MCP servers: real utterances, attempted tools, argument values, and execution outcomes. We kept only trajectories that ended in verified success and deduplicated aggressively — 60% of requests map to roughly 15% of the operation space, and without dedup the model overfits the popular path.

**LLM-assisted labeling with verification.** We used frontier models to propose trajectories for the hard tail — ambiguous requests, cross-system intents like “sync this account’s new orders from Oracle into Salesforce” — then verified every proposal against the target APIs’ OpenAPI/WSDL schemas and, for anything with a side effect, an execution check in a sandbox tenant. Proposals that failed were rejected, not repaired. Labels were canonicalized into one strict grammar per dialect, so the model never had to learn two serializations for one decision.

**Adversarial negative mining.** We deliberately constructed the base model’s failure cases — requests where an entity like “order” maps to a different operation in context, requests naming a field from the wrong object, cross-dialect confusions — and used them as DPO rejected pairs. The final corpus: ~12,000 SFT examples and ~4,000 preference pairs, split 90/10 with the eval stratified so no tool appears only in training. Every number in Section 4 comes from that held-out set.

## 4. Accuracy Benchmarks: Llama 3 (Fine-Tuned) vs. GPT-4o vs. Claude 3.5 Sonnet

We benchmarked four configurations on 800 held-out real-world-derived requests (400 Oracle, 400 Salesforce), scored at the trajectory level: a call is correct only if the *operation*, *arguments*, and *serialization* all match the golden trajectory. Models: GPT-4o and Claude 3.5 Sonnet via hosted APIs (zero-shot tool-use prompting), base Llama 3 8B (zero-shot), and our SFT+DPO LoRA-tuned Llama 3 8B. Temperature 0, five runs per request, majority vote on disagreements.

| Configuration | Oracle tool-select | Salesforce tool-select | Argument accuracy | Composite trajectory |
|—|—|—|—|—|
| Llama 3 8B (base, zero-shot) | 61.4% | 64.2% | 58.7% | 56.9% |
| GPT-4o (zero-shot) | 82.1% | 84.6% | 81.3% | 78.4% |
| Claude 3.5 Sonnet (zero-shot) | 84.8% | 86.9% | 83.5% | 80.2% |
| **Llama 3 8B (SFT + DPO LoRA)** | **93.6%** | **94.8%** | **91.2%** | **89.7%** |

Three findings matter more than the headlines. **First, the tuned 8B beats both frontier models on its own dialect by roughly nine to eleven points on composite accuracy.** That is the expected result of training on the actual distribution: the frontier models are generalists; our adapter is a specialist in one narrow decision. This is not a claim that fine-tuned Llama 3 is smarter — it is a claim that it is *more calibrated to this tool surface*.

**Second, the gap widens exactly where prompting hurts most.** On the adversarial subset, GPT-4o fell to 71.2% and Claude 3.5 Sonnet to 73.8%, while the tuned model held 88.4%. Few-shotting closed maybe two points of the frontier gap; it never closed the structural one. Moving from 20 tools to the full 60-tool registry cost the frontier models three to four points and the tuned model less than one — prompting degrades with surface size; fine-tuning does not.

**Third, the base-model delta is the entire story.** Base Llama 3 8B at 56.9% is unusable here; the tuned model at 89.7% is production-grade. The whole gain — 33 points — came from ~160 MB of adapters and a training run measured in hours, at roughly 3–5× lower per-request cost than the hosted frontier models. We ran the same pipeline on Llama 3 70B; it buys one to two points for 8× the inference cost. The 8B adapter is the economic optimum.

## 5. Implementation: Deploying LoRA Adapters on AWS Bedrock or EKS

An adapter that lives in a notebook is a demo. Shipping it means putting it behind the same infrastructure as the rest of the agent — typed tool surfaces, auth boundaries, observability, rollback. We deploy on two AWS paths, and the choice is usually dictated by existing platform posture.

**Option A: AWS Bedrock (managed, minimal ops).** Bedrock supports importing custom models: the LoRA adapter is merged with the base weights at export, uploaded to S3, and registered through the API. You get managed inference with IAM access, CloudWatch logging, and auto scaling — no GPU fleet to babysit. The trade-offs: managed-inference margins, Bedrock’s import cadence (slower adapter iteration than a hot-swap), and no multi-adapter multiplexing behind one endpoint. One dialect, one region — Bedrock is the right call.

**Option B: EKS with vLLM (self-hosted, adapter-native).** For multi-dialect estates — Oracle in production, Salesforce in the same control plane, MS SQL next quarter — we deploy on EKS inside the client’s VPC. vLLM natively serves LoRA adapters, so one base-model deployment exposes multiple adapter endpoints and swaps them per request with near-zero overhead. The stack is standard: an inference Deployment behind a service, an MCP server as the tool-orchestration layer in front, OIDC-auth’d ingress, and a HorizontalPodAutoscaler on GPU utilization. Oracle traffic routes to its adapter; Salesforce traffic to its own; the base weights are shared, so memory is dominated by one model plus small adapter slabs.

Our rule of thumb: **Bedrock if you have one dialect and want someone else to own the GPU; EKS + vLLM if you have multiple adapters, regulated data that must stay in your VPC, or a retraining cadence measured in weeks.** Either way, three non-negotiables apply. The endpoint sits behind the same MCP tool layer as any hosted model, so swapping GPT-4o for the adapter is a routing change, not a rewrite. Every tool call is logged as a structured event for eval replay. And the previous adapter version stays live for one release cycle, so a regression is a routing rollback, not a redeploy. Instrument production accuracy the same way you ran the eval — sample traffic, replay against the golden set, and page on drift when the Oracle or Salesforce schemas change.

## 6. ROI: Performance Gains vs. Training and Inference Cost Trade-Offs

The honest question is never “does fine-tuning work?” — Section 4 answers that. It is whether fine-tuning pays for itself on *your* volume, *your* surface, and *your* accuracy requirement — a function of four numbers.

**Training cost is a rounding error.** Our SFT + DPO LoRA run on Llama 3 8B took roughly 12 GPU-hours on an A100-class instance, eval and ablations included — a few hundred dollars at current rates, and the low thousands even with a 70B experiment and a second dialect. That is less than one week of GPT-4o spend for a moderately busy agent, and it is capital cost amortized over every subsequent request. The real constraint is dataset curation labor, not compute.

**Inference cost is where the trade-off lives.** The tuned 8B runs at roughly 20–30% of GPT-4o’s per-token cost and lower still than Claude 3.5 Sonnet. But self-hosting has a floor: a GPU pod costs the same at one request a minute or a thousand. Below roughly 5,000–10,000 tool-call requests per day, hosted frontier models are often cheaper — you pay for consumed tokens, not reserved silicon. Above that volume the crossover favors the self-hosted adapter, and the gap widens linearly. The tuned model also removes the giant few-shot system prompt from every request and eliminates prompting’s worst failure mode: silent accuracy cliffs when the prompt drifts or a tool is added.

**The accuracy-point valuation is the number most teams skip.** A mis-selected tool costs one of three things: a failed API call retried (cheap), a wrong side-effect manually reversed in Oracle or Salesforce (expensive — the reason sandbox verification and human checkpoints before irreversible actions matter), or a silent wrong answer that reaches the user (most expensive). At 8,000 requests per day, moving from 80% to 90% trajectory accuracy is 800 fewer failures daily. If even 10% are wrong side-effects costing fifteen minutes of admin time to reverse, the tuned model pays for its entire training run before lunch. That is why we write the eval before the engagement: baseline accuracy and failure cost are both knowable in week one.

**When NOT to fine-tune.** Three conditions push us back toward prompting: low volume (a few thousand requests daily or less, where hosted models win at the margin); a tool surface still changing weekly (adapter retraining churns); and accuracy requirements a router already satisfies — if 85% of traffic is three well-handled tools, a small classifier routing those to a cheap prompt path beats fine-tuning on ROI. Fine-tuning is for the workload that is high-volume, stable-shaped, and where a wrong call has real cost. That matches most production Oracle and Salesforce agents we see — which is exactly why the prompting ceiling, not model capability, is what our clients keep hitting.

## Conclusion: When Prompting Ceils, Fine-Tune the Decision

Frontier models will keep improving at general tool use, and your API dialect will keep getting more specific. That asymmetry is permanent — which is why a small, fine-tuned specialist that knows your Oracle envelope and your Salesforce object model cold is not a workaround; it is the correct architecture for a stable, high-volume tool surface. In our study, a ~160 MB LoRA adapter on Llama 3 8B outperformed GPT-4o by 11 points and Claude 3.5 Sonnet by 9 points on composite trajectory accuracy, for a training cost in the hundreds of dollars and inference at 8B pricing. The bottleneck was never the model — it was the dataset, and dataset curation is an engineering problem with a known solution.

If your agent team is staring at a tool-selection plateau — Oracle, Salesforce, MS SQL, or a custom surface — bring us the eval and we will tell you in one discovery week whether fine-tuning earns its keep. OnlineCloud builds the MCP tool layer, curates the golden trajectories, runs the SFT + DPO + LoRA pipeline, and deploys the adapter on Bedrock or in your VPC on EKS — with the accuracy dashboards to prove it. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) and let’s measure your baseline: prompting where it works, fine-tuning where it doesn’t, and an honest cost model for the gap in between.

# Vector Database Sharding: Architecting for Billion-Scale Agentic Retrieval


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: When the Single Node Stops Being Enough

The pattern is always the same. Somewhere in month three of a production agent rollout, a 2 a.m. alert fires: vector search p95 latency has tripled, recall is silently degrading, and the ingest queue is backing up. Your Pinecone pod — or self-hosted pgvector instance — sits at 95% memory. The HNSW graph has outgrown its cache-friendly working set, every query is a disk-page fault, and the agent that answered in 300 milliseconds now carries the slowest dependency in its own loop.

This is not a tuning problem. A single-node vector store has hard ceilings: an HNSW graph must be largely resident in RAM to deliver its advertised latency, pgvector’s IVFFlat must keep inverted lists on disk without turning scans into random-I/O storms, and both absorb every write and query from every tenant on the box. Once the working set — vectors, graph metadata, tenant filters — exceeds one node, the answer is the one every database eventually reaches: **shard it**.

But sharding a vector store is not like sharding a relational database: there is no primary key to range-scan for a point lookup, and every similarity search is a *global* read — to find true neighbors you must examine enough of the corpus that you cannot miss them. That property changes everything downstream. The rest of this post takes each problem in turn: partitioning, index tuning, routing, tenant isolation, and the path to trillion-scale retrieval.

## 2. Sharding Strategies: Metadata Partitioning vs. Random Hash Distribution

Every sharding conversation starts with one question: *on what key do we split the corpus?* Two families dominate, and the choice is the highest-leverage decision you will make — it fixes your query fan-out, index sizes, and isolation story for the life of the system.

**Metadata-based partitioning.** Pick a high-cardinality attribute every query carries — tenant ID, product category, collection, region — and assign each value to a fixed shard. All vectors sharing the attribute land on one node. The payoff: a scoped query touches exactly one shard, the graph stays small and hot, and latency matches a single node. For agentic workloads this is enormous, because agents issue *many* retrieval calls per task — a support agent filters every `search_knowledge_base` call by tenant and SKU family. Ten scoped calls mean ten cheap local searches, not ten cluster-wide scatter-gathers.

The tax is skew. Tenants are never uniform; the largest may hold 40% of vectors while the long tail shares the rest, and the node holding the anchor tenant becomes the bottleneck you sharded to escape. You mitigate with two-level placement — coarse shard per attribute, replication or re-sharding of hot shards — but you are now operating a placement system, not a database. And cross-attribute queries (“find similar incidents across all tenants for compliance review”) fan out to every shard.

**Random hash distribution.** Assign every vector to a shard by hashing its ID, yielding statistically uniform partitions, perfect load balance, and trivial elasticity — add a node, rehash a slice. This is the strategy for genuinely global queries: semantic search over a public corpus, deduplication, “find anything like this anywhere.”

The cost: *every* query is cluster-wide. With hash distribution you cannot know which shard holds the true neighbors, so a top-k search fans out to all N shards and merges. Recall now depends on per-shard over-fetch — each shard must return 4–10× the final k so boundary neighbors survive the merge — and per-shard graphs are larger than under attribute partitioning at equal total data, since each covers a random sample rather than a semantic slice.

The architect’s heuristic is blunt: **partition on the attribute your queries always carry; hash when they carry nothing.** Mature systems do both — hash the corpus into balanced shard groups, then partition within the group on tenant or collection so scoped queries resolve to one shard inside a balanced group. The routing layer is what makes the hybrid viable.

## 3. Retrieval Performance: Optimizing HNSW and IVFFlat for Distributed Search

Sharding does not remove index tuning — it multiplies it, and adds a new failure mode: **perfect local indexes, broken global recall.** Decide what quality means across shards before touching a single parameter.

With hash distribution, asking each shard for its local top-k and merging loses recall at the boundaries: the true nearest neighbor may rank fifth on its home shard and never make the cut. The fix is oversampling — query each shard for `k × factor` candidates and merge with a global distance re-rank. Per-shard candidate count grows with roughly the *log* of shard count, not linearly. Measure it: run a labeled eval set against 1, 4, 16, and 64 shards and plot recall against the oversampling factor until the curve flattens. That curve is your capacity plan.

**HNSW distributed.** HNSW’s navigable small-world graph delivers logarithmic search, but neighbor lists add 30–100+ bytes per vector on top of the embedding, all of it wanting RAM. Distribution gives two levers a single node lacks. First, *right-size the graphs*: a 50-million-vector shard does not need `M = 32` links per node — drop to `M` 12–16 and `efConstruction` 100–200, shrink memory, and let oversampling cover the local recall loss at merge. Tune per-shard recall to the *merged* target, not 0.99 locally. Second, *exploit partition locality*: under attribute partitioning, each shard’s graph covers one semantic region — smaller, hotter, often more accurate for scoped queries, since long-range links no longer bridge unrelated neighborhoods. Attribute sharding quietly buys more recall per unit of memory.

**IVFFlat and the write path.** pgvector-style IVFFlat splits vectors into `nlist` inverted lists via k-means; a query probes `nprobe` lists. Rules of thumb: `nlist ≈ √n` (roughly 1,000 lists per million vectors), with `nprobe` as the latency/recall dial. Distributed IVFFlat inherits every single-node hazard and adds one of its own: **staleness under rebalancing.** IVFFlat is batch-built, and new vectors land in the nearest *existing* cluster center, which drifts as the corpus grows. Rebalancing moves vectors between nodes whose centroids were trained on different slices — and recall collapses silently. Budget for periodic per-shard rebuilds, or move to HNSW once write volume makes batch rebuilds untenable.

Two rules hold regardless of index type: hybrid metadata filters change the calculus (a filtered scan over a partitioned shard often beats a graph search over a giant unfiltered index), and quantization (SQ/PQ) shrinks vectors 4–32× so each node holds far more in RAM — fewer shards, less fan-out, simpler merges.

## 4. Scaling: The Query-Routing Layer for Sharded Vector Stores

Once vectors span nodes, retrieval needs a brain: a **query router** between the agent runtime and the shards that decides where each query goes, fans out where required, merges results, and presents one stable interface so sharding stays an implementation detail.

Its first job is *classification*. Does this tool call carry a partition key — tenant ID, collection filter? Resolve it to its shard and issue a single-node query: no fan-out, no merge, local-search latency. No filter? Switch to scatter-gather across every shard with the configured oversampling, merge partial results by distance, apply post-filters and final top-k. Dual-mode routing is the entire point: it runs attribute-partitioned and hash-distributed shards behind one API and serves each query with the strategy that matches its shape.

Its second job is *merge correctness*. Merge is a simple k-way heap ordered by distance only when every shard speaks the same metric in the same space. Mix index types, quantization, or per-shard normalization and you need a global re-rank over the merged candidates — never let shard-level score calibration leak into final ordering.

Its third job is *placement and consistency*. The router owns the shard map — which keys live where, replication state. During rebalancing it should dual-read (query old and new placements, merge, reconcile), and it routes upserts and deletes with the same key-hash logic as reads, so an update is visible to the agent’s next retrieval call. Agentic workloads are brutal on consistency: a planning loop that writes a memory and immediately retrieves it exposes every replication lag and cache staleness bug. For single-writer-per-key workloads, pin reads and writes to one shard; for global corpora, version vectors and reconcile at merge. The router is also the enforcement point for retrieval budgets — cap shard touches and candidate volume per request, tenant, and agent loop, because retries turn fan-out into multiplied cost.

## 5. Multi-Tenant Isolation: Tenant-Scoped Retrieval in Shared Clusters

Every serious sharded deployment is multi-tenant, and isolation is absolute: **tenant A’s retrieval must never observe tenant B’s vectors, and tenant B’s query storm must never degrade tenant A’s latency.** Single-node systems enforce this with metadata filters. Sharded clusters must enforce it at three layers or it leaks.

**Placement.** The strongest isolation is physical — dedicated shards or nodes for high-value tenants. That is metadata partitioning at its purest: the right call for regulated tenants, massive corpora, or contractual SLAs. The cost is utilization, so tier it — shared shards for the long tail, dedicated shards for the top few tenants, with a policy that lets tenants graduate between tiers as volume changes.

**Index and filter enforcement.** For tenants sharing a shard, isolation must be structural: **the partition key belongs in the physical index layout, not in an after-the-fact filter.** If the ANN index ignores tenant ID, a top-k search can return tenant B’s vectors before any filter runs — and with HNSW, post-filtering does not restore recall, because the graph paths to tenant A’s true neighbors were never followed. In order of preference: a separate per-tenant index on shared nodes (each small and hot); a single index where tenant ID is a hard key prefix and queries are constrained to the tenant’s subspace; or filtered ANN where the pre-filter actually bounds graph traversal. Whatever you choose, your eval must include an adversarial cross-tenant test — tenant A’s query, tenant A’s filter, scored against tenant A’s ground truth, in a cluster where tenant B’s corpus is orders of magnitude larger. If recall drops as the neighbor grows, your isolation is leaking through the index.

**Noisy-neighbor defense.** Shared clusters fail as systems before they fail as privacy boundaries. One tenant’s batch re-embedding job can saturate a shard, evict another tenant’s graph from cache, and blow every latency SLA on the node. Defense in depth: per-tenant rate limits and concurrency caps at the router; per-shard CPU, memory, and I/O governance; priority classes that put interactive agent retrieval ahead of batch ingestion; and *separate read and write paths* — rebuilds run on shadow resources while the query graph stays untouched, with an atomic swap on completion. The tenant re-embedding a million documents at midnight should see a slower ingest queue, never a degraded query path. Every decision also needs an audit trail — per-tenant query logs, per-shard saturation, a “who touched what” record — and in agentic systems this is doubly true: the retrieval tool is invoked by a model, and “the model did it” is not an isolation story your security team will accept.

## 6. Future-Proofing: From Billion-Scale to Trillion-Scale Retrieval

One billion to one trillion vectors is not a factor of a thousand in hardware — it is a change of regime. A trillion 128-dimensional float32 vectors is 512 TB raw; an HNSW graph over them needs petabytes of RAM at single-node densities. Trillion-scale retrieval forces three shifts, and all three should shape decisions you make today.

**Memory ceases to be the default.** The billion-scale assumption — “keep the graph in RAM” — inverts. Trillion-scale systems are disk-native with memory as a tier: compressed vectors (PQ with 8–16-bit codes) generate candidates; raw vectors are reserved for the final re-rank; the index is structured so only a fraction is resident. This is the DiskANN lineage — a navigable graph over compressed neighbor lists on SSD with a hot head cached in RAM. The lesson for today’s billion-scale builders: **design your shard layout, quantization, and re-rank pipeline as if the working set will not fit in memory** — retrofitting disk tiering is brutally expensive.

**The graph stops being global.** HNSW assumes one navigable graph over the corpus. At trillion scale you get *hierarchical, partitioned* search: a coarse level of learned cluster centroids partitions the space, a region-level graph navigates between regions, per-region graphs finish the hop — IVF at the top, HNSW at the bottom, and a router deciding how deep each query descends. Planners that treat region selection as a learned decision rather than a fixed `nprobe` are where the recall-per-dollar wins will live. Today that means choosing ANN libraries whose index format supports multi-level, disk-resident layouts, and standardizing the vector-store interface so the backend is replaceable when the regime shifts.

**Retrieval becomes a pipeline stage, not the product.** At trillion scale you will not ship raw ANN output to an agent; you will ship the result of a staged pipeline — coarse candidate generation over compressed vectors, pruning by metadata and tenant policy, cross-encoder or LLM re-ranking, deduplication, citation assembly. The vector index becomes the recall engine of a larger retrieval system, exactly as it already is inside agent loops. Architect the interface with that shape — a search tool returning scored, filtered candidates with provenance — and the trillion-scale migration becomes a backend swap, not a re-plumbing of your agents.

The strategic constant across every regime: **evals, routing, and tenant isolation survive all three shifts; index internals do not.** Build those three layers as if scale is inevitable — because for agentic retrieval, it is.

—

## Conclusion: Architect for the Shard Before You Need It

The single-node vector store fails predictably: latency climbs, recall drifts, and the agent loop becomes the hostage of its slowest dependency. Sharding is the answer — but only when you shard on the right key, tune the index for the distributed regime, route every query through a layer that knows a scoped lookup from a global search, and enforce tenant isolation in the index itself rather than in the filter clause. Systems that survive billion-scale agentic traffic are those where sharding was an architecture, not an incident response.

At OnlineCloud, we design and ship evaluated, observable retrieval systems for production agent workloads — sharded vector architectures, MCP-native tool surfaces, and the eval harnesses that prove recall and isolation before they touch real users. If you are hitting the wall on a single-node vector store, or planning the jump past a billion vectors, [contact the OnlineCloud team](https://www.onlinecloud.dev/) — our senior engineers will map your corpus, query shapes, and tenant boundaries, and tell you the sharding strategy that survives production contact. Free initial consultation, response within 24 hours.

# Building a Multi-Agent Debater Pattern: Resolving Logical Conflicts in LLMs


> A Principal AI Scientist’s field notes on making models argue their way to the truth — proposers, opponents, and judges standing between your users and a confidently wrong answer.

## 1. Introduction: Why Chain-of-Thought Isn’t Enough for Adversarial Tasks

Chain-of-thought (CoT) was the first honest admission that a language model reasons better when allowed to think out loud. By emitting intermediate tokens before a final answer, the model turns one compressed prediction into a sequence of smaller, conditioned ones. It works — impressively so on arithmetic, symbolic manipulation, and multi-hop lookup — and it remains structurally insufficient for an important class of problems.

Here is the uncomfortable property of autoregressive decoding: the model is arguing with itself, and it always lets itself win. CoT is a monologue. A single trajectory embodies one viewpoint, one decomposition, one failure mode. When a model misreads a clause or smooths over a contradiction, nothing inside the trajectory can catch it, because the error was committed by the same weights that are now asked to review it. Prompting a model to “review your answer” rarely surfaces its own mistakes — and revision often moves toward the error, not away from it. The generator and the verifier are the same function; the adversary is missing.

That gap matters most for *adversarial* tasks: the ones where a plausible-but-wrong answer is the failure you are trying to eliminate, not tolerate. Financial advice, legal interpretation, regulatory compliance, medical triage — in these domains a fluent wrong answer is indistinguishable from a correct one to the reader, and the cost asymmetry is severe. A single-turn answer that sounds authoritative is, to a first approximation, unverifiable by the person reading it.

The industry’s first response was ensembling: sample N completions and take a majority. Self-consistency reduces variance, not bias — if every sample shares the same misreading of the prompt, the majority vote is a confident, repeated mistake. What adversarial tasks require is what adversarial settings have always required: an *opponent* whose incentive is to attack the claim, and a *judge* whose incentive is neither. The lineage runs from Minsky’s society of mind, through Irving and colleagues’ AI-safety-via-debate proposal, to Du and colleagues’ multiagent-debate results — and production agentic systems are finally ready to borrow it.

## 2. The Debater Pattern: Proposer–Opponent–Judge Architecture

The debater pattern replaces a monologue with a minimal adversarial society of three roles.

**The Proposer** receives the question and constructs the best affirmative case it can: an answer, the reasoning behind it, and the evidence or authority it leans on. Its disposition is confident but literal — it is optimized to produce a complete, defensible position, not to hedge.

**The Opponent** receives the same question *and* the proposer’s position, chartered to attack it — not for sport, but for the failure classes that matter in your domain: unsupported premises, misapplied rules, confabulated citations, ignored constraints, arithmetic errors. It returns structured objections with reasons, not vibes: *”that § 408A limit is the 2025 figure, not the 2026 figure.”*

**The Judge** sees the question, the case, the objections, and the rebuttal, then renders a verdict with stated confidence and rubric scores across the axes you care about: correctness, completeness, citation integrity, constraint compliance. The judge must *choose* — affirm, overturn, or remand for revision — because an unforced judge defaults to polite synthesis.

Three properties separate this from a toy. First, **adversarial asymmetry:** proposer and opponent should not share a temperature, and often not a model family — cross-model disagreement is where real errors surface. Second, **structured outputs:** each role emits typed fields (claim, evidence, objection, severity, rebuttal), so the judge scores structure and your observability stack logs it. Third, **a bounded transcript:** two rounds — open, attack, rebut, rule — capture most of the value; if the judge remands, the proposer revises once and the judge rules again. Beyond that, you are paying for theater.

## 3. Implementation: LangGraph and AutoGen for Consensus

The pattern is a state machine with three roles, a message log, and a termination condition — which is why it ports cleanly onto the dominant multi-agent frameworks.

**LangGraph.** Model the debate as a graph: roles are nodes, edges encode turn order and termination. Shared state carries the growing transcript, and each node appends its structured contribution. The judge node owns a conditional edge that routes to `revise` on a remand verdict or to `end` otherwise. Checkpointing buys two production properties: the debate can be replayed from any node — time-travel a bad verdict to see exactly which objection the judge ignored — and deterministic logic can sit inside the graph. Our citation validator pattern-matches statutory references against a live database before the judge rules; the opponent can even re-run the proposer’s retrieval with different parameters to catch cherry-picked evidence.

**AutoGen.** AutoGen’s conversational substrate maps onto the pattern directly: proposer and opponent are `ConversableAgent`s with asymmetric prompts and sampling parameters, and the judge is a third agent whose termination check declares the debate over. A deterministic `GroupChat` manager adds the production levers: enforce the round cap, validate every message against a schema, and early-stop when the opponent raises no objection above the severity floor — skipping the judge on the easy cases where adversarial review is unnecessary.

Framework aside, three rules apply. **Never** let agents exchange free-form text alone — validate role outputs at every boundary. **Time-box and round-cap everything**; a debate that can run away is a denial-of-service vector against your own budget. And **make the judge the only node with side effects** — no mutating tool call, no external communication, until a verdict exists. The verdict is the commit point of the whole transaction.

## 4. Real-World Use Case: Detecting Hallucinated Financial Advice and Legal Errors

Where does this pattern earn its latency? In domains where a fluent wrong answer is dangerous and the model’s own confidence is uninformative.

**Financial advice.** A single-turn system answers: *”You can contribute up to $23,500 to your 401(k) in 2025, plus a $7,500 catch-up if you’re 50 or older — and a Roth conversion looks attractive at your income.”* Plausible, mostly right, and wrong in a way that costs the user money: the limits are from last year because that is what the training distribution remembers, and the conversion advice ignores state tax and the IRMAA thresholds it pushes the user across. The opponent, pointed at the specific claims, demands: *”Which tax year are these limits from? Cite the IRS revenue procedure. Recompute the bracket including the conversion and the user’s stated state.”* The proposer either produces the authoritative citation or is caught fabricating one — the moment the judge must be wired to a deterministic citation checker rather than its own opinion.

**Legal reasoning.** The failure signature is subtler. Legal hallucinations are rarely invented statutes; they are *misapplications* — the right rule applied to the wrong element, or a doctrine extended past the jurisdictions that recognize it. A single-turn answer asserting *”a two-year non-compete is presumptively void in your state”* may be defensible in one jurisdiction and malpractice in its neighbor. The opponent’s charter in a legal deployment is jurisdiction- and element-focused: *”What state’s law governs? Has the answer distinguished the leading case? Does it conflate reasonableness review with a per-se rule?”* Two rounds of debate with retrieval grounding convert the answer from “the model’s best guess” into “the model’s best guess, attacked, defended, and adjudicated against the source of law.”

In both domains the measurable effect is the same: the pattern does not eliminate hallucination, it *relocates* it. Errors that survive are those a generator and an independent attacker — different sampling, different instructions — commit independently, a far higher bar than a single trajectory. On our eval harnesses, the debater converts the most dangerous failure class — confident, internally consistent, wrong — into either a correct verdict or an explicit low-confidence remand routed to a human. A system that knows when to escalate is safer than one that never escalates.

## 5. Performance: The Latency Trade-Offs of Multi-Turn Reasoning

Let us be direct about the cost: the debater pattern is *expensive*, and any post that omits the trade-off is a sales document.

A single-turn CoT response is one model call. Self-consistency with K samples is K parallel calls — wall-clock latency near single-turn at K× token cost. The debater is different because it is *sequential*: round two cannot start until round one finishes, and the judge cannot rule until both have spoken. With three roles and two rounds, expect four to six sequential model calls — at p50 token speeds, roughly 8–30 seconds on a substantive question. That is an order of magnitude above a single turn and firmly outside inline-chat territory.

That latency is bought, not wasted. It buys (1) **independent verification** — the opponent’s disagreement rate across your eval set measures how often single-turn answers needed a second opinion; (2) **error localization** — when the judge overturns, the transcript names the objection that carried; and (3) **calibration** — verdict confidence correlates with true accuracy across a sample in a way raw model self-confidence does not, which lets you build a router on top.

The engineering levers that make the trade-off tolerable:

– **Route, don’t default.** Classify the request first. The majority of queries — factual lookups, summarization, well-scoped single-source questions — never enter the debate graph; the router reserves the full debater for the adversarial minority where the cost asymmetry justifies it.
– **Early-stop on agreement.** If the opponent files no objection above the severity floor, skip the judge and accept — collapsing the pattern toward a two-call pipeline on non-adversarial traffic.
– **Model-tier asymmetry.** Put the strongest model in the judge role, the cheapest adequate model in the proposer; the opponent can sit a tier below the judge if its temperature and charter are aggressive.
– **Budget context, not just calls.** Every round appends to the transcript and attention cost grows with length. Cap role outputs, trim evidence between rounds, prefer structured fields over narrative prose.

The honest framing for stakeholders: you are trading single-turn latency for multi-turn *certainty*, and the correct unit of measurement is not requests-per-second but **errors per thousand answered, weighted by the cost of each error**. For a chatbot summarizing a return policy, the debater is malpractice. For a system advising on a seven-figure rollover or a litigation hold, fifteen seconds and a few extra tokens are the cheapest insurance you will ever buy.

## 6. The OnlineCloud Commitment: Tested, Evaluated Reasoning Swarms

This pattern is not a paper exercise for us. OnlineCloud is a US-based engineering studio that ships agentic systems to production, and the debater is one member of a family of *evaluated reasoning architectures* we deploy against real schemas: Oracle, MS SQL, Postgres, Snowflake, CRMs, and the tools your operators already live in. Our position is simple: **a reasoning swarm you have not evaluated is a hypothesis, not a system.**

That commitment shows up in three places. First, **evals precede code.** Before we wire a proposer–opponent–judge loop into a workflow, we build the labeled adversarial set that defines “done”: financial prompts with deliberately stale limits, legal questions with jurisdiction traps, the edge cases where a confident wrong answer is the expected failure of a single-turn baseline. The debater earns its extra latency only if it measurably beats that baseline on *your* distribution — and we measure it before we ship it.

Second, **every debate is observable.** Each role’s output is a structured, logged event; every verdict is replayable; every objection is attributable. Our MCP-native architecture gives the reasoning loop the same typed tools, rate limits, and audit trails as every other component — because a judge that hallucinates a *verdict* is worse than a proposer that hallucinated an answer, and you need the transcript to know which happened.

Third, **we surface the trade-offs, not just the accuracy.** Latency budgets, cost ceilings, model-tier placement, and escalation rates are dashboard items agreed in the discovery week and reviewed weekly — honest measurement is a product feature. The debater pattern is only defensible when the person paying for the extra calls can see exactly what those calls bought. If you are standing up agentic reasoning where a fluent wrong answer has legal or financial consequences, the question is not whether to use a debater — it is whether your baseline can survive an opponent.

—

### Ready to put your reasoning under adversarial review?

OnlineCloud is a US-based engineering studio specializing in agentic AI and Model Context Protocol — built end-to-end by senior engineers, no offshore handoffs. We design tested, evaluated reasoning swarms that plug into Oracle, MS SQL, Postgres, Snowflake, your CRM, and the tools your team uses daily, and we ship to production in 6–10 weeks, not quarters.

**[Contact us for a free initial consultation →](mailto:contact@onlinecloud.dev)** — response within 24 hours.

# Typed Tools in MCP: Using JSON Schema for Predictive Tool Execution


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: The Failure of Natural Language Tool Descriptions

Every agent team hits the same wall around week three. The demo works — the right tool, the right arguments, applause. Then a real user runs the same flow, and the model calls `create_ticket` with `{“severity”: “urgent-ish”, “customer_id”: “the one from the email”}` — faithfully following a description that said *”customer_id: the identifier of the customer.”* The failure was never the model’s. It was the interface’s.

Natural language tool descriptions are seductive because they read like requirements — and they are the weakest contract you can ship. A model does not *understand* a description; it pattern-matches against it during generation, so every ambiguity in your prose becomes a degree of freedom in its output. Worse, prose is unenforceable at runtime: nothing checks that `severity` is one of the three values your paging system accepts, or that `customer_id` is a UUID in your CRM. The description is an aspiration; the schema is the law.

In the Model Context Protocol (MCP), a tool is declared with a `name`, a `description`, and an `inputSchema` written in JSON Schema. The first two aim at the model; the third aims at reality. Across fifteen-plus years of shipping data systems at OnlineCloud — Oracle, MS SQL, Postgres, Snowflake, CRMs — we have learned that the difference between an agent that closes tickets and one that merely appears to is whether its tool surface encodes *structural invariants* the model cannot violate. This post is about designing that boundary — and knowing how much schema each tool needs.

—

## 2. The Role of JSON Schema in MCP: Enforcing Structural Invariants

MCP is, at its core, a stateless JSON-RPC protocol. A host calls `tools/list`, and each tool advertises an `inputSchema` — a JSON Schema document describing the exact shape of arguments it will accept. When the model acts, it emits a `tools/call` request whose `arguments` are validated against that schema *before* your business logic runs. Schema first, side effects second — that ordering is the entire game.

“`json
{
“name”: “refund_order”,
“description”: “Issue a refund against a completed order.”,
“inputSchema”: {
“type”: “object”,
“additionalProperties”: false,
“required”: [“order_id”, “amount_cents”, “reason”],
“properties”: {
“order_id”: {
“type”: “string”,
“pattern”: “^ord_[a-zA-Z0-9]{20}$”
},
“amount_cents”: {
“type”: “integer”,
“minimum”: 1,
“maximum”: 1000000
},
“reason”: {
“type”: “string”,
“enum”: [“duplicate_charge”, “cancellation”, “damaged_goods”, “other”]
}
}
}
}
“`

Notice what this schema does that no description can: `order_id` is *not* “an identifier of the order” but a string matching `^ord_[a-zA-Z0-9]{20}$`; `amount_cents` is an integer — not a float, not a decimal string, not “$29.99″; `reason` is closed to four values, a fifth structurally impossible. Every constraint is checkable in microseconds, deterministic, and independent of which model is calling.

A schema is a *normative* artifact; a description is an *aspirational* one. That distinction drives how we build MCP servers at OnlineCloud: the `inputSchema` is the tool’s executable specification — it gates model calls at runtime, renders into the model’s context, drives our test fixtures, and generates the OpenAPI surface for human consumers. One artifact, three consumers, zero drift. Descriptions, by contrast, are kept deliberately short, because long descriptions encourage the model to *reason about* arguments instead of conforming to their shape.

This is the deeper role of JSON Schema in MCP: make the model’s job *easier by making it more constrained*. Generation under a schema is a search over a smaller space; smaller spaces produce fewer wrong answers. An MCP tool without a strict schema is not a function — it is a suggestion.

—

## 3. Designing Robust Schemas: Enums, Patterns, and Recursive Types

Schema quality is a design discipline, not a serialization detail. The three highest-leverage constructs — enums, patterns, and recursive references — each target a different failure mode.

**Enums close open sets.** Any argument naming one of a fixed set of things should be an `enum`: statuses, severities, channels, currencies, reasons. Faced with `”enum”: [“open”, “in_progress”, “resolved”, “closed”]`, the model is not choosing a string that *resembles* a status; it is choosing among four tokens — far more reliable than open-ended generation. Enums are also a *measurement* instrument: if logs show the model reaching for values outside the enum, the enum is missing a legitimate option — add it, never loosen validation.

**Patterns capture domain grammar.** Identifiers are the classic case: every real system has a generative grammar for its keys — `ord_`-prefixed IDs, UUIDs, E.164 phone numbers, ISO-8601 timestamps, ISO-4217 currency codes. A `pattern` keyword turns that grammar into an enforceable constraint:

“`json
{
“type”: “object”,
“required”: [“phone”, “effective_from”],
“properties”: {
“phone”: { “type”: “string”, “pattern”: “^\\+[1-9]\\d{1,14}$” },
“effective_from”: {
“type”: “string”,
“pattern”: “^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(Z|[+-]\\d{2}:\\d{2})$”
}
}
}
“`

Two cautions on patterns. First, the model sees the raw regex, so write it for *readers* — anchored, conservative, annotated. Second, prefer the most specific semantic check your stack offers: if the server validates an ID against the database anyway, do that in code and keep the pattern as the cheap syntactic gate. Patterns reject nonsense at the boundary; they are not a substitute for existence checks.

**Recursive types model nested reality.** The most common reason a schema gets loosened to `{“type”: “object”}` is a tree-shaped payload — a filter expression, a workflow DAG, an approval chain. “It’s just a nested object, I can’t type it” is a confession, not an excuse. JSON Schema handles recursion with `$ref` and `$defs`:

“`json
{
“type”: “object”,
“properties”: {
“filter”: { “$ref”: “#/$defs/condition” }
},
“$defs”: {
“condition”: {
“oneOf”: [
{ “type”: “object”, “additionalProperties”: false, “required”: [“field”, “op”, “value”],
“properties”: {
“field”: { “type”: “string”, “pattern”: “^[a-z_]{1,32}$” },
“op”: { “enum”: [“eq”, “neq”, “gt”, “gte”, “lt”, “lte”, “contains”] },
“value”: { “type”: [“string”, “number”, “boolean”] }
} },
{ “type”: “object”, “additionalProperties”: false, “required”: [“logic”, “children”],
“properties”: {
“logic”: { “enum”: [“and”, “or”, “not”] },
“children”: { “type”: “array”, “minItems”: 1, “items”: { “$ref”: “#/$defs/condition” } }
} }
]
}
}
}
“`

Three rules keep recursive schemas safe in production. Guard every recursion with a discriminator (here, `field`/`op` versus `logic`/`children`) so the model always has a deterministic next step. Bound the recursion — `maxItems`, plus a server-side depth ceiling — because an unbounded tree is a prompt-injection and cost-control hole. And add `examples` at each level: for nested structures, a worked example beats a paragraph of description, because models generalize from concrete instances more reliably than from abstract rules.

Whatever your stack — Pydantic in Python, Zod in TypeScript — generate the `inputSchema` from the same typed definition your server code uses. Hand-maintained JSON Schema and hand-written parsers drift apart by the second commit.

—

## 4. Pre-execution Validation: Rejecting Malformed Model Calls at the Server Boundary

The design decision separating serious MCP deployments from demos is *where* validation happens and *what happens when it fails*. Validation is a server-side security and correctness boundary, not a client convenience: assume any payload can arrive — from a confused model, a prompt-injected context, a stale client — and reject what does not conform *before* a single side effect occurs.

In the MCP lifecycle this is natural: `tools/call` arrives, the server validates `arguments` against the `inputSchema`, and a malformed call returns a JSON-RPC `InvalidParams` error (`-32602`) naming the offending fields. The discipline is in what you refuse to do around it. Never coerce silently: turning `”amount”: “29.99”` into `29.99` may feel helpful, but coercion teaches the model that sloppy output is survivable and quietly erases the signal that your schema or prompt is failing. And never partially execute — validate the *entire* payload, then act.

The failure path is a first-class product surface: a rejection must be *actionable by the model*, because in an agentic loop the model is the one that retries. “Validation failed” leaves it guessing; `”amount_cents: expected integer, got string; order_id: does not match pattern ^ord_[a-zA-Z0-9]{20}$”` lets it repair the call in one attempt. We measure this by counting: the *repair ratio* — retries per successful call — is one of the most sensitive early indicators of schema quality we track. It should sit near one.

Those logs are strategic data, not noise: a spike in validation failures is frequently the first symptom of a model upgrade, a drifted system prompt, or prompt-injection probing. Treat the boundary as an observability instrument — log caller, tool, offending fields, and outcome — and alert on anomalies. A team that ignores the rejections is throwing away the diagnosis.

—

## 5. Enhancing Model Accuracy: How Strict Typing Reduces Agentic Hallucination

The claim that JSON Schema improves model accuracy sounds paradoxical — how does removing freedom improve a generative system? Because tool calling is not a creative act; it is *structured prediction*: given a function signature and a goal, emit the argument object that satisfies both. Hallucination here is not inventing facts; it is *sampling from a space never constrained to contain only valid answers*. Every missing `enum`, every loose `pattern`, every `type: “object”` placeholder widens the sampling space — and everything in that widened space is a candidate hallucination.

Strict typing attacks hallucination on four fronts:

**Constrained decoding by construction.** An `enum` of four statuses is not a description that the model *should* choose among four values; it is a constraint that makes other tokens structurally invalid — and models respect structural constraints far more reliably than prose. You are not asking the model to be careful; you are making correctness the only cheap path.

**Fewer repair loops, less drift.** Every rejected call pollutes context with an error and pushes the model into *repair* mode, where it is measurably more likely to over-correct or try an adjacent wrong thing — and it burns tokens and latency. A strict schema collapses the reject-and-retry cycle, so the agent spends its generation budget on the task, not on apologizing.

**Disambiguation of ambiguous intents.** Take `search_orders(query)`: prose invites the model to stuff anything into `query` — a name, a date, a status, a complaint. Explicit typed parameters — `customer_email`, `status`, `date_from`, `date_to` — force the model to *classify* its intent into your domain model before the call, and that classification step is where accuracy is won. Ambiguity that survives belongs in a clarifying question, not in a free-text blob.

**Versioned stability across model generations.** The agents you ship this year will not run the same model forever; descriptions tuned to one model’s quirks rot on the next, while a schema is model-agnostic. Strict typing is why a server built for Claude can serve a GPT-4-class model or an open-weights deployment without behavioral drift. The schema — not the prose — makes tool behavior *predictive*: given a valid payload, the tool does one thing, every time, for every model.

None of this makes schema a substitute for evaluation; it makes schema the substrate *on which* evaluation becomes meaningful. When every call either conforms or is rejected with a named reason, your eval harness can distinguish “the model chose the wrong tool” from “the model mangled the arguments” — two failures with different fixes.

—

## 6. Decision Framework: When to Use Simple vs. Complex Schemas

Strict schemas are a tool, not a theology. Over-constraining is a real failure mode: the model can exhaust its context budget against a schema written for a machine, not a model. The art is matching strictness to the risk and structure of the operation. At OnlineCloud we run every tool through four questions:

**What is the blast radius of a malformed call?** Read-only, reversible operations tolerate looseness; writes, irreversible actions, and anything touching money or PII get maximal constraint. `search_logs` may accept a free-text `query`; `transfer_funds` gets enums, patterns, and explicit amounts — every time.

**Closed or open set?** Closed sets (statuses, reasons, channels, logic operators) become enums, unconditionally. Open sets (a product name, a search term, a note to a human) stay typed but unconstrained — because an enum that does not contain the answer forces the model to fail or to lie.

**Flat or nested?** Flat payloads of two to five typed fields rarely need more than `required`, `type`, and `enum`. For nested trees — filters, DAGs, approval chains — reach for `$defs` and `$ref`, paying the complexity cost to reject structurally meaningless payloads before they touch your engine.

**Who consumes this tool, and how often?** A tool called hundreds of times a day by an unattended agent earns every constraint you can write — each rejection is a failed business transaction. A rarely used, human-in-the-loop tool can afford looser edges; the human absorbs the ambiguity the schema would otherwise carry.

Two rules of thumb anchor the framework. First, default to the *simplest schema that makes invalid calls impossible to misinterpret*: every field typed, every closed set closed, `additionalProperties: false` on objects — nothing more. Complexity is justified only when a constraint removes a failure mode you have observed or can foresee. Second, constrain what you can name and leave a seam for what you cannot: a typed structure with one well-bounded free-text field outperforms both a fully loose and a fully rigid schema.

This is not a one-time decision: schemas must evolve with the failures your logs reveal — tightening where the model invents values, loosening where legitimate calls are rejected. Schema review belongs in the same cadence as eval review, because the schema is the eval that runs on every single call.

—

## 7. Conclusion: The Schema Is the Product

Typed tools are the difference between an agent that *appears* to understand your business and one that *cannot* violate it. Natural language descriptions told the model what you wanted; JSON Schema tells it what is possible, making everything else structurally unreachable. The result is an agent whose tool calls you can predict, test, and audit — not because the model is cleverer, but because you stopped leaving correctness to chance.

At OnlineCloud, we build MCP-native systems for teams that need agents to do real work against real systems — Oracle, MS SQL, Postgres, Snowflake, CRMs — with schema discipline, eval suites, and observability built in from the first commit. If you are designing a tool surface, or debugging an agent that keeps mangling arguments, talk to our architects. We will help you turn your tool boundary into a contract your models cannot break — and your logs will prove it.

**Ready to build typed, production-grade MCP servers?** Contact the OnlineCloud architects today for a free initial consultation — [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) — or learn more at [https://www.onlinecloud.dev/](https://www.onlinecloud.dev/). US-based senior engineers, response within 24 hours.

# Distributed Agent Orchestration: Implementing a Hierarchical Planner in Python

*By an OnlineCloud Principal AI Engineer — 9 min read*


Every serious agent system I’ve reviewed in production eventually hits the same wall: a single prompt — no matter how well-crafted — cannot reliably execute a task with more than a few dependent steps. The fix isn’t a better prompt. It’s a better control structure. This post walks through the architecture and Python implementation of a **hierarchical planner** — a Planner–Worker–Verifier control loop that decomposes work, executes it in parallel, verifies it, and heals itself when the world disagrees with the plan.

## 1. The Bottleneck of Linear Planning

The default agent design is a loop: model → tool call → observation → model. For a two-step task like “look up an order and summarize its status,” that loop is fine. For anything real — a logistics reroute, a multi-system data migration, a claims workflow — the loop becomes a liability. Three failure modes dominate:

**Error compounding.** Each step consumes context and injects drift. If step three of eight produces a subtly wrong intermediate value, steps four through eight are all downstream of garbage. A linear chain has no structural way to notice, because nothing is *checking* intermediate outputs against their specification.

**Lost locality.** A single model context holds the entire task, so every decision is entangled with every other. The model reasoning about carrier APIs is also holding route geometry, pricing data, and customer constraints in one window. Context limits force truncation, and truncation is where silent mistakes are born.

**Zero recovery semantics.** When a linear loop fails, the only move is “try the whole thing again” — replaying cost, latency, and the same odds of failure. There’s no notion of *which subgoal* failed, no notion of *retrying only that*, and no budget except your own patience.

The pattern that fixes all three is older than LLMs: **divide and conquer, with inspection**. Hierarchical task decomposition. What LLMs add is that the decomposition — and the inspection — can be generated dynamically per request instead of hard-coded.

## 2. The Hierarchical Pattern: Planner–Worker–Verifier

The architecture has three roles and one shared contract. The **Planner** takes a goal and a world snapshot and produces a structured plan: an ordered list of typed subgoals, each with an owner, inputs, and acceptance criteria. The **Workers** execute subgoals against real systems — they are narrow, single-purpose, and stateless with respect to the overall goal. The **Verifier** is the quality gate: it checks each worker’s output against the subgoal’s acceptance criteria and returns a verdict with *machine-readable feedback*, not just a score.

“`
┌─────────────┐
goal ───────────────▶│ Planner │◀───── revision context
└──────┬──────┘
│ plan (typed subgoals)
┌────────────────┼─────────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Worker A│ │ Worker B│ │ Worker C│ ← parallel fan-out
└────┬────┘ └────┬────┘ └────┬────┘
└────────────────┼─────────────────┘
▼
┌─────────────┐
│ Verifier │── ACCEPT ──▶ consolidate / ship
└──────┬──────┘
│ REVISE + feedback (bounded loop)
▼
Planner re-plans failed subgoals only
“`

Three properties make this work in practice. First, **the plan is data, not prose**: typed subgoal objects with explicit acceptance criteria, so verification is a deterministic check layered over an LLM judgment call. Second, **workers are interchangeable**: because each subgoal declares its type and inputs, you can swap an LLM worker for a deterministic function or a human approval step without touching the orchestration layer. Third, **feedback flows up, not just down**: the verifier’s critique returns to the planner as structured context, which is what converts a blind retry into a *directed* revision.

## 3. Stateful Planning in Practice: LangGraph vs. PydanticAI

The hierarchy only works if the runtime gives you three things: explicit state, controllable control flow, and durable checkpoints. As of this writing, the two Python frameworks I’d put in front of a client are **LangGraph** and **PydanticAI**.

**LangGraph** models the agent as a state machine: nodes are Python functions, edges are explicit transitions, and a typed state object flows through every node. Its superpowers are *conditional edges* (the verifier verdict literally routes the graph) and *checkpointing* — every state transition can be persisted, so a crashed run resumes mid-plan rather than restarting. It’s the right choice when your orchestration is genuinely complex: branching, loops, parallel fan-out, human-in-the-loop interrupts.

**PydanticAI** is the opposite philosophy: agents are plain typed functions with dependency injection, and multi-agent work is *delegation* — one agent calls another as a tool, with the model deciding when. It’s dramatically more readable for the 80% case, and its `RunContext` gives you typed deps without globals. The tradeoff: the control flow lives inside the model’s tool-calling loop, so bounded retry loops and checkpoints are something you build rather than inherit.

My rule of thumb: if your hierarchy is **two levels and mostly linear**, use PydanticAI and keep the code boring. If you have **verifier-gated revision loops, parallel fan-out, or long-running stateful tasks**, use LangGraph — because those are exactly the features you shouldn’t hand-roll. The walkthrough below uses LangGraph, with the hierarchical *structure* portable to either.

## 4. Orchestration: Multi-Agent Communication and State Consistency

Once you have multiple agents mutating shared state, the hard problem stops being intelligence and starts being *consistency*. Five disciplines keep distributed agent state from rotting:

**One shared state schema, owned by the graph.** Agents never pass unstructured strings to each other. Everything they exchange — plans, results, feedback, spend — is a typed field on a single state object. In LangGraph this is the state schema; in any design it should be a versioned Pydantic model, because the schema *is* the contract between agents.

**Reducers for everything append-only.** Messages, task results, and feedback accumulate via merge functions, never via overwrite. If two workers finish concurrently, their results append into an ordered log — no last-writer-wins races. This is `operator.add` or a custom reducer over an annotated channel, and it eliminates the most common class of heisenbugs in multi-agent systems.

**Idempotency keys at the system boundary.** The real world is not idempotent: booking a carrier, writing a row, sending a notification. Every worker that touches an external system carries a client-generated request ID (`order_id + task_id + attempt`), so a retried execution cannot double-book. If the downstream API doesn’t support idempotency keys, wrap it in a small outbox table that does — dedupe on write, exactly-once on retry.

**Checkpoints, not just logs.** A log tells you what happened; a checkpoint lets you *resume* from it. LangGraph’s checkpointer serializes full graph state after every super-step. MemorySaver is fine for demos; production wants Postgres or Redis-backed persistence so a worker crash — or a full process restart — resumes the exact plan with spend and attempts intact.

**Tracing with a correlation ID.** Every agent appends to a `trace` list on its result: `(node, step, model, cost_ms)`. Ship those into OpenTelemetry with the run’s correlation ID, and you can answer the two questions every incident starts with: *which subgoal failed, and how much did we spend before it did?*

## 5. Error Recovery: Self-Healing Loops and Backtracking

The verifier is what makes recovery *surgical* instead of *nuclear*. The control loop is deliberately small and brutal:

1. Worker output fails verification (schema-invalid, acceptance criteria unmet, or external call errored).
2. The verifier returns structured feedback: *which subgoal, which acceptance criterion, what evidence is missing*.
3. The router decides the recovery granularity: **retry the worker** (transient failure, same plan), **revise the plan** (the plan itself was wrong — planner gets the feedback and re-plans only the failed subgoals), or **escalate** (budget exhausted, or two consecutive revisions failed — hand to a human with the full trace).

Three guardrails keep the self-healing loop from becoming a self-immolating one:

**Bounded attempts, counted in state.** `attempts` and `revision_count` live in the state schema and are checked by the router. There is no unbounded “keep trying” — there is a counter, a limit, and an escalation path.

**Budget as a first-class state field.** Track `spent` against `budget` on every LLM call (input + output tokens priced per model). When spend crosses the ceiling, the router must escalate, not retry. Cost ceilings are a *control-flow* concern, not an accounting one.

**Backoff with jitter on external calls.** Transient API failures are the most common error in real systems. Workers retry with exponential backoff plus jitter; the *orchestrator* never retries — it just marks the subgoal failed and lets the router decide. That separation keeps failure handling at exactly one layer.

This gives you the property that matters most in production: **a failed run terminates in one of two states — a verified result, or an escalation with a readable trace.** Never a silent half-done state.

## 6. Code Walkthrough: A Hierarchical Logistics Agent

Let’s build it. The scenario: a customer requests a *priority reroute* for an in-transit shipment. The planner decomposes the goal into typed subgoals — validate the order, replan the route, match carriers with live rates, run a compliance check, estimate ETA and cost. Workers execute each subgoal against mocked external systems, and a verifier gates acceptance. The revision loop re-plans only the failed subgoal.

“`python
# agent_state.py
from __future__ import annotations
import operator
from typing import Annotated, Any, Literal, TypedDict

from pydantic import BaseModel, Field

# —- Shared contract: the typed plan ————————————-

class TaskSpec(BaseModel):
“””One subgoal. The acceptance_criteria field is what makes
verification deterministic instead of vibes-based.”””
task_id: str
kind: Literal[“validate_order”, “route_plan”, “carrier_match”,
“compliance_check”, “eta_cost”]
goal: str
inputs: dict[str, Any] = Field(default_factory=dict)
acceptance_criteria: list[str] = Field(default_factory=list)

class TaskResult(BaseModel):
task_id: str
kind: str
status: Literal[“ok”, “error”, “needs_input”]
output: dict[str, Any] = Field(default_factory=dict)
error: str | None = None
trace: list[str] = Field(default_factory=list) # correlation trail

class Verdict(BaseModel):
accepted: bool
issues: list[str] = Field(default_factory=list) # machine-readable feedback

# —- Graph state: single source of truth for every agent ——————

class LogisticsState(TypedDict, total=False):
order: dict[str, Any] # immutable request payload
plan: list[TaskSpec] # produced by the Planner
results: Annotated[list[TaskResult], operator.add] # append-only channel
feedback: Annotated[list[str], operator.add] # verifier critiques
attempts: int # bounded self-healing counter
budget: float # hard cost ceiling (USD)
spent: float # accumulated token cost
accepted_plan: dict[str, Any] # verified, ready to ship
needs_human: bool
“`

The state schema is the contract: workers read `order` and write `results`; the verifier reads `results` and writes `feedback`; only the router reads the counters. Nothing is shared outside the graph, and every channel that multiple nodes write to is append-only.

“`python
# graph.py
import asyncio, json, uuid
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import END, START, StateGraph
from langgraph.types import Command # noqa (used with interrupt for humans)

from agent_state import LogisticsState, TaskResult, TaskSpec, Verdict

MAX_ATTEMPTS = 3
REVISION_BUDGET_USD = 1.50 # hard ceiling per run

# —- Workers: narrow, single-purpose, registered by kind ——————

WORKERS: dict[str, callable] = {} # populated by @worker decorator below

def worker(kind: str):
def deco(fn):
WORKERS[kind] = fn
return fn
return deco

@worker(“validate_order”)
async def validate_order(order: dict) -> dict:
# Deterministic gate: no LLM needed where a schema check suffices.
if order.get(“status”) != “in_transit”:
raise ValueError(f”order {order[‘order_id’]} is not in transit”)
if not order.get(“items”):
raise ValueError(“order has no line items”)
return {“valid”: True, “shipment_id”: order[“shipment_id”]}

@worker(“route_plan”)
async def route_plan(order: dict, spent: float) -> dict:
# Mocked as deterministic for the walkthrough; in production this is an
# LLM call priced against the shared budget.
return {“segments”: [“MEM → DFW”, “DFW → LAX”], “via”: “air”}

@worker(“carrier_match”)
async def carrier_match(order: dict) -> dict:
# Idempotency: same (order, task) always sends the same idempotency key.
idem = f”carrier:{order[‘order_id’]}:{uuid.uuid5(uuid.NAMESPACE_URL, json.dumps(order, sort_keys=True))}”
# …POST /carriers/quote with Idempotency-Key: idem…
return {“carrier”: “AeroFreight”, “rate”: 412.50, “confirmed_slots”: 1}

@worker(“compliance_check”)
async def compliance_check(order: dict) -> dict:
return {“hazmat”: “ok”, “export_control”: “ok”, “blocked_regions”: []}

@worker(“eta_cost”)
async def eta_cost(order: dict, segments: list[str]) -> dict:
return {“eta_days”: 2, “total_cost_usd”: 1340.0, “currency”: “USD”}
“`

Notice what the workers are *not* doing: none of them decide the next step, none of them read the verifier’s feedback, and none of them know a plan exists. That ignorance is the point — it makes them trivially testable and swappable.

“`python
# —- Planner: LLM call that turns a goal + feedback into typed subgoals —-

async def call_llm_json(system: str, user: str, budget: dict) -> dict:
# Real implementation: your provider SDK + token accounting that
# increments budget[“spent”]. Kept abstract so the graph logic stays
# framework-agnostic.
…

async def planner_node(state: LogisticsState) -> dict:
“””Decompose the goal into typed subgoals. On revision rounds, the
verifier’s feedback is injected so the planner only re-plans what broke.”””
if not state[“plan”]:
system = “You are the planning layer of a logistics agent. ” \
“Emit subgoals as JSON only.”
else:
system = “The plan below failed verification. Re-plan ONLY the ” \
“subgoals implicated by feedback. Keep accepted subgoals.”
payload = await call_llm_json(
system=system,
user=json.dumps({“order”: state[“order”],
“feedback”: state.get(“feedback”, [])}),
budget=state)
tasks = [TaskSpec(**t) for t in payload[“subgoals”]]
return {“plan”: tasks,
“attempts”: state[“attempts”] + 1,
“spent”: payload[“cost”]}
“`

The planner node’s only job is to turn *goal + world + feedback* into a typed plan. Every subsequent node treats that plan as data, which means the verifier can hold the plan and the results side by side — that’s the whole trick.

“`python
# —- Executor: fan out to workers, gather, and trace ———————–

async def executor_node(state: LogisticsState) -> dict:
“””Run every planned subgoal concurrently. Each worker writes one
TaskResult into the append-only results channel.”””

async def run_one(spec: TaskSpec) -> TaskResult:
try:
fn = WORKERS[spec.kind]
out = await fn(**spec.inputs, spent=state[“spent”])
return TaskResult(task_id=spec.task_id, kind=spec.kind,
status=”ok”, output=out,
trace=[f”{spec.kind}:ok”])
except Exception as exc: # surface, don’t swallow
return TaskResult(task_id=spec.task_id, kind=spec.kind,
status=”error”, error=str(exc),
trace=[f”{spec.kind}:error:{exc!r}”])

results = await asyncio.gather(*(run_one(t) for t in state[“plan”]))
return {“results”: list(results)}
“`

Errors are captured into `TaskResult` objects, never raised through the graph. A raised exception would kill the whole run; a captured error is *just another input to the router*, which is exactly where failure-handling belongs.

“`python
# —- Verifier: deterministic checks layered over one LLM judgment ———-

async def verifier_node(state: LogisticsState) -> dict:
“””Gate every subgoal against its acceptance criteria.”””
by_id = {r.task_id: r for r in state[“results”]}
issues: list[str] = []

for spec in state[“plan”]:
res = by_id.get(spec.task_id)
if res is None:
issues.append(f”{spec.task_id}: no result produced”)
elif res.status == “error”:
issues.append(f”{spec.task_id}: {res.error}”)
elif res.status == “ok”:
# 1) Structural check — deterministic, zero token cost.
missing = [c for c in spec.acceptance_criteria
if not _criterion_satisfied(c, res.output)]
issues += [f”{spec.task_id}: {m}” for m in missing]

if not issues:
return {“accepted_plan”: _consolidate(state), “needs_human”: False}

verdict = await call_llm_json(
system=”You are the verifier. Given failing subgoals, decide: ”
“RETRY_WORKER (transient) or REVISE_PLAN (plan was wrong).”,
user=json.dumps({“issues”: issues, “attempts”: state[“attempts”]}),
budget=state)
return {“feedback”: verdict[“issues”],
“spent”: state[“spent”] + verdict[“cost”]}
“`

“`python
# —- Router: the self-healing control flow ——————————–

def router(state: LogisticsState) -> str:
“””Where the graph goes next depends on counters in state — the whole
recovery policy lives in this one pure function.”””
if state.get(“accepted_plan”):
return “accept”
exhausted_budget = state[“spent”] >= state[“budget”]
out_of_attempts = state[“attempts”] >= MAX_ATTEMPTS
if exhausted_budget or out_of_attempts:
return “escalate”
# REVISE_PLAN ⇒ loop back through the planner with feedback attached;
# RETRY_WORKER would route back to the executor instead. Both are just
# conditional edges — no bespoke retry framework required.
return “revise”

# —- Assemble the graph —————————————————-

builder = StateGraph(LogisticsState)
builder.add_node(“planner”, planner_node)
builder.add_node(“execute”, executor_node)
builder.add_node(“verify”, verifier_node)
builder.add_node(“human_review”, lambda s: {“needs_human”: True})

builder.add_edge(START, “planner”)
builder.add_edge(“planner”, “execute”)
builder.add_edge(“execute”, “verify”)
builder.add_conditional_edges(“verify”, router, {
“accept”: END,
“revise”: “planner”, # ← backtrack: only failed subgoals re-plan
“escalate”: “human_review”, # ← human checkpoint with full trace
})
builder.add_edge(“human_review”, END)

graph = builder.compile(checkpointer=MemorySaver()) # PostgresSaver in prod

async def main() -> None:
order = {“order_id”: “SO-88231”, “status”: “in_transit”,
“shipment_id”: “SH-441”, “items”: [{“sku”: “A1”, “qty”: 40}]}
thread = {“configurable”: {“thread_id”: “run-” + uuid.uuid4().hex[:8]}}
final = await graph.ainvoke(
{“order”: order, “plan”: [], “attempts”: 0,
“budget”: REVISION_BUDGET_USD, “spent”: 0.0},
config=thread)
print(json.dumps(final.get(“accepted_plan”) or
{“escalated”: True, “trace”: final[“results”]}, indent=2))

if __name__ == “__main__”:
asyncio.run(main())
“`

Three details in this walkthrough carry the whole design. **The verifier is layered**: deterministic structural checks run first at zero token cost, and the LLM verdict is only consulted when structure fails — that’s how you keep verification cheap enough to run on every attempt. **The router is a pure function over state counters**: recovery policy is testable without any LLM in the loop (feed it a state dict, assert the edge). **The revision loop backtracks to the planner, not to the start**: `attempts` is bounded at three and `budget` at $1.50 per run, and crossing either one routes to a human checkpoint with every worker trace intact — OnlineCloud’s “plan/act/reflect with cost ceilings and human checkpoints” pattern, exactly.

If you squint, the whole graph reads like a state machine version of what a good senior dispatcher does: break the job into pieces someone can own, check the pieces before you commit, and when something breaks, fix the *piece* — not the whole shipment.

## Building This for Your Systems

The hierarchy above is deliberately framework-light at the edges: workers are plain functions, the verifier is a JSON contract, and only the control flow is graph-shaped. That’s what makes it portable across LangGraph, PydanticAI, or a hand-rolled asyncio loop — and what makes it auditable, because every transition is a checkpoint and every decision is a state field.

A hierarchical planner is the difference between an agent that *looks* autonomous in a demo and one that survives contact with real databases, real auth boundaries, and real cost ceilings. When you’re ready to move past the demo, the architects at OnlineCloud design and ship exactly this class of system — MCP-native orchestration with evaluated, observable, human-checkpointed control loops, in production in 6–10 weeks.

**Have a multi-step agent workload that needs real orchestration? Talk to an OnlineCloud architect** — [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev), or reach out via [onlinecloud.dev](https://www.onlinecloud.dev/). Free initial consultation, US-based senior engineers, response within 24 hours.

# Securing MCP Resources: OAuth 2.1 and Scoped Token Enforcement


*By the security engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: The Vulnerability of Agentic Data Access

Every agentic system ships with the same silent assumption: *the model will only do what we asked it to do.* That assumption is the vulnerability.

Think about what an MCP (Model Context Protocol) server actually is. It is an authenticated door from an LLM — a stochastic, prompt-injectable, eminently socially-engineerable principal — directly into your resource estate: `customers/{id}/orders`, `finance/invoices`, `hr/records`. In the pre-agentic world, a human clicked through an application, and the application’s UI was the permission boundary. In the agentic world, the model proposes tool calls autonomously, in a loop, often over long-running sessions where a single malicious string inside a fetched document can steer the next invocation. The UI is gone. The boundary must be rebuilt at the protocol layer.

The uncomfortable truth is that **an MCP resource URI is a permission**. `mcp://ledger/orders/4421` is not merely an address; it is a statement of what a caller may read or mutate. When a host connects to an MCP server without authentication, without per-request authorization, and without a notion of *whose* access is being exercised, then every resource the server exposes is available to every agent that reaches it — and every prompt injection becomes a data exfiltration primitive.

This is why the security architecture of MCP cannot be an afterthought bolted onto a working demo. It must answer five questions before the first tool call ships:

1. **Who** is the human ultimately responsible for this action?
2. **What** capability does this specific agent invocation require?
3. **How** does the server verify the caller on every request, not just at connection time?
4. **Where** does identity terminate, so the database can enforce row-level policy?
5. **What** evidence is left behind for auditors and incident responders?

OAuth 2.1 answers the first three. Scoped token enforcement answers the second and the fourth. Audit logging answers the fifth. This post walks through each, and closes with a reference architecture that binds them together inside a VPC-isolated MCP deployment. I write this as a security architect who has spent fifteen-plus years shipping data systems against Oracle, MS SQL Server, Postgres, and Snowflake — because the agentic era does not forgive the mistakes the API era tolerated.

## 2. OAuth 2.1 for MCP: Implementing the Authorization Code Flow with PKCE

OAuth 2.1 is not a new protocol so much as a **consolidation of the security requirements** that the industry spent a decade learning the hard way: PKCE is mandatory for every client, implicit grants are abolished, refresh tokens for public clients are restricted, and every grant must be bound to a client through sender-constrained tokens. For MCP, this matters because the MCP Authorization specification — now part of the broader MCP ecosystem — adopts OAuth 2.1 as its native authorization layer, and the new Streamable HTTP transport is explicitly designed to carry it.

The pattern to implement is the **Authorization Code Flow with PKCE** (Proof Key for Code Exchange). The flow, as it applies to an MCP host:

1. The user opens the agent host and the host discovers the MCP server’s authorization metadata via `/.well-known/oauth-authorization-server`.
2. The host generates a `code_verifier` (a high-entropy random string) and its SHA-256 hash, the `code_challenge`, which it sends with the authorization request to the authorization server. The plaintext verifier never transits the wire.
3. The user authenticates and consents at the authorization server, which returns an authorization code — not a token — to the redirect endpoint.
4. The host exchanges the code for tokens at the token endpoint, presenting the `code_verifier`. The server hashes it and compares against the stored challenge. **This is the PKCE guarantee:** even if an attacker intercepts the authorization code, they cannot redeem it without the verifier, which never left the host.
5. The host receives an access token with a short TTL and a refresh token bound to the client.

Three implementation rules separate a compliant deployment from a decorative one.

**Rule one: PKCE is never optional.** In OAuth 2.1 it is required for public clients, and you should require it for confidential clients too. There is no code path where PKCE is skipped because “this client is trusted.” MCP hosts are frequently desktop or mobile applications — precisely the public-client category where static client secrets are worthless and PKCE is the only credential the host can prove.

**Rule two: tokens are short-lived, and refresh tokens rotate.** Access tokens in an MCP context should live minutes, not hours — long enough for a bounded task, short enough that a leaked token in a prompt-injection artifact is stale before it is useful. Refresh tokens must be single-use: every refresh returns a new refresh token and invalidates the old one, so a stolen refresh token dies at the next rotation. Never issue a refresh token that outlives its session.

**Rule three: every request is authenticated, not every connection.** Streamable HTTP sessions must not smuggle tokens into a connection and then assume the session is authorized. Each tool call carries its token (in practice, a short-lived token refreshed on demand), and the MCP server validates signature, issuer, audience, expiry, and scope **per request**. A connection that was authorized at 9:00 AM is not authorized at 9:00 AM plus one revoked session.

Concretely, the server-side validation on every request should check, in order: signature validity, `iss` and `aud` claims against the configured issuer and the server’s own client ID, `exp`, and then the scope claim against the tool or resource being invoked. Fail any check, return a structured error, and emit an audit event. That final step — logging the rejection — is what turns authentication from a gate into evidence.

## 3. Scoped Tokens: Mapping Model Capabilities to Granular Resource Permissions

Authentication answers *who you are*. Authorization answers *what you may do*. In OAuth terms, authorization is expressed as **scopes**, and scopes are where MCP security is won or lost — because the granularity of your scopes is the granularity of your containment.

The cardinal sin in agent deployments is the coarse scope. A token that carries `data:all` or `resources:read_write` gives the model a master key, and a master key in the hands of a stochastic principal is not authorization — it is a blast radius. The model does not need to *want* to exfiltrate data; it needs only to be successfully instructed to.

Scope design for MCP must be **capability-shaped, resource-shaped, and operation-shaped** simultaneously. A token’s scope should read like a sentence describing the smallest useful action: `orders:read:own`, `invoices:read:own`, `refunds:create:under_500`. Not `orders`, not `read`, not a wildcard namespace. The mapping from model capability to scope is a deliberate design act: enumerate the tools your agent genuinely needs, decompose each into the resource class it touches and the operations it performs, and mint a scope per (operation, resource-class) pair. If a tool needs to read a customer’s order history to answer “where is my package,” the token scope is `orders:read` for that user’s rows — not `orders:read` for every row in the warehouse.

Two enforcement mechanisms make scopes real at the MCP layer.

**First, scope-to-tool mapping at the server.** Each MCP tool declares the scope(s) its invocation requires, and the server rejects any call whose token lacks them *before* the tool handler runs. This is a declarative allowlist, enforced in one place:

“`json
{
“tools”: {
“get_order_status”: {
“required_scopes”: [“orders:read:own”, “shipments:read:own”],
“resource_surface”: “mcp://ledger/orders/{order_id}”
},
“create_refund”: {
“required_scopes”: [“refunds:create:under_500”],
“human_checkpoint”: “refund > $500”
}
}
}
“`

**Second, resource-scoped authorization at the data layer.** The scope claim answers “may this caller invoke `get_order_status`?”; a second question — “may this caller read *order 4421*?” — must be answered by row-level security keyed to the propagated identity, which is the subject of the next section. Scope grants the *capability*; identity governs the *rows*.

One further refinement: in high-assurance deployments, exchange the coarse login token for a **task-scoped delegation token** at the start of each agentic run. OAuth 2.1 token exchange lets the orchestrator present its token and receive a narrower one — tighter scopes, shorter TTL, an explicit `aud` of the single MCP server, and a `delegation` claim chain naming the initiating user. The model never holds the user’s token; it holds a perishable, narrow proxy for it. This is the difference between “the agent is the user” and “the agent acts *for* the user, within limits that expire.”

## 4. Identity Propagation: How to Flow User Identity from Client to Tool

Here is the failure mode that keeps security architects up at night: the user authenticates beautifully at the front door, and then the agent calls the database with a **service account**. Every row-level policy, every audit trail, every least-privilege grant designed around the human evaporates the moment the model’s tool call executes as `mcp_svc_prod`. The database sees a single privileged principal with a thousand concurrent sessions, and “who did this?” becomes unanswerable.

Identity propagation for MCP means the end user’s identity — not the model’s, not the server’s service account — must survive the journey from client to tool to database, as an **enforceable and auditable claim**. The mechanism is a delegation chain carried in the token and forwarded at each hop.

At the authorization server, mint the access token with claims that describe the full context of the call:

“`json
{
“iss”: “https://auth.onlinecloud.internal”,
“aud”: “https://mcp.ledger.internal”,
“sub”: “delegated:task-8f3a…”,
“scope”: “orders:read:own refunds:create:under_500”,
“exp”: 1713000000,
“delegation”: [
{ “actor”: “user:alice@example.com”, “auth_time”: 1712999400, “amr”: [“pwd”, “webauthn”] },
{ “actor”: “client:finance-agent:v3”, “purpose”: “invoice reconciliation run” }
]
}
“`

The MCP server then does two things with these claims. It **enforces** them — parsing the innermost `actor` and passing it to the data layer so that row-level policies evaluate against *Alice*, not against the service account — and it **forwards** them in the audit record of every tool invocation.

In the data layer, enforcement takes the shape of your database’s native RLS mechanism. In Postgres, the server sets `current_setting(‘app.caller_id’)` and `app.caller_chain` at the start of the session or transaction, and every policy evaluates against that setting. In SQL Server, `SESSION_CONTEXT` plus security predicates achieve the same. In Snowflake, row-access policies key on the propagated identity column. The invariant in every case is identical: **the database never executes a query whose effective principal is the service account.** It executes queries whose effective principal is Alice, mediated by an agent the system can name.

Two hard rules govern propagation. **Rule one: propagate, never substitute.** If the chain is ever broken — if a hop loses the user claim and falls back to a generic principal — that hop fails closed and logs an alert. A tool that cannot prove whose data it is touching does not get to touch it. **Rule two: delegation never escalates.** The effective permission at any hop is the intersection of the user’s grants and the agent’s scopes. An agent cannot exercise a permission the user lacks, and a user grant cannot be widened by the agent’s token. The confused-deputy problem — a privileged agent tricked into acting on behalf of an unprivileged user, or an unprivileged user’s prompt injection steering a privileged agent — is neutralized because neither side can grant what the other does not hold.

## 5. Audit & Compliance: Logging Authenticated Resource Access in MCP

Auditors do not ask whether your agent is clever. They ask a far more uncomfortable question: **can you prove what it did, under whose authority, and why?** For regulated workloads — SOC 2, HIPAA, SOX, GDPR — an agentic system that cannot produce a replayable record of authenticated resource access does not have a compliance story; it has a liability.

Audit logging for MCP must therefore capture the *full authorization context* of every tool invocation, not merely “a call happened.” The minimum event schema:

– **Who**: the propagated delegation chain — initiating user, client/agent, orchestration run ID.
– **What**: tool name, validated input arguments, resource URIs touched.
– **Authorization**: token ID, scopes presented, which scope satisfied the call, token issuer.
– **Outcome**: success, or the structured rejection reason (expired, wrong audience, insufficient scope, RLS denial).
– **When and where**: timestamp with timezone, server instance, transport session ID.
– **Parent linkage**: the event ID of the agentic step that spawned this call, so the log forms a tree — from user intent down to the row that changed.

Two properties make the trail trustworthy. **Completeness:** logging must live *inside* the authorized request path, not around it. If a tool call can succeed without emitting an audit event, then under load, unlogged calls will happen — and the one that matters will be the missing one. **Immutability:** the audit sink is append-only and written with credentials no agent token can reach. The trail must survive the very compromise it is designed to detect. Ship audit events to a separate store — an independent log stream or a write-only data source — that sits outside the blast radius of any MCP server compromise.

Compliance also demands an answer to the *human* question: which human authorized this system behavior, and which human approved the irreversible actions? Log the consent event at the authorization server (user X approved scopes S for client C at time T), and log human checkpoints at the tool layer — the operator ID, the approval timestamp, and the exact payload they approved. When a regulator asks “who said this refund of $4,000 was acceptable?”, the answer is a signed audit event, not a guess.

Finally, build the **replay** capability before you need it. Store enough structured context per event — the tool schema version, the model and prompt version, the parent plan — that an investigator can reconstruct the decision path, aborted branches included. Proving intent requires showing what the agent *considered*, not only what it executed. An MCP deployment whose audit trail supports this replay is a deployment that walks into an assessment and demos its own compliance.

## 6. Reference Architecture: A Secure, VPC-Isolated MCP Auth Flow

The pieces above compose into a single, defensible architecture. Here is the deployment OnlineCloud runs for regulated customers — the same pattern we would run for you.

“`
┌─────────────┐ ┌──────────────────────────────────────────────────────┐
│ End User │ │ Customer VPC │
│ (Browser / │ │ │
│ Desktop) │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────┐ │
└──────┬──────┘ │ │ Agent Host │───▶│ MCP Server │──▶│ Database │ │
│ │ │ (OAuth client)│ │ (authz gate) │ │ (RLS) │ │
│ │ └──────┬───────┘ └──────┬───────┘ └──────────┘ │
│ │ │ 1. auth request │ │
│ │ ▼ │ │
│ │ ┌──────────────────┐ │ │
└──────────┼──│ Authorization │ │ 2. validate JWT: iss, │
│ │ Server (IdP) │ │ aud, exp, scope, │
│ │ PKCE + consent │ │ delegation chain │
│ └──────────────────┘ │ │
│ │ 3. exchange │ 4. RLS keyed to │
│ ▼ │ propagated user │
│ ┌───────────────────────────────┐ │
│ │ Audit Store (append-only) │◀── every call logged │
│ └───────────────────────────────┘ │
└──────────────────────────────────────────────────────┘
“`

The numbered flow: **(1)** the end user initiates a session; the agent host begins the OAuth 2.1 Authorization Code flow with PKCE against the internal authorization server, and the user consents to a narrow scope set. **(2)** The host receives a short-lived access token whose claims carry the delegation chain from step 4 of this post — user, agent, purpose — and whose audience is pinned to the single MCP server. **(3)** The host invokes tools over Streamable HTTP inside the VPC. Every request terminates at the MCP server’s authorization gate, which validates signature, issuer, audience, expiry, and scope against the requested tool, then rejects or forwards. **(4)** For data access, the server sets the propagated identity in the database session context and lets row-level security evaluate against the *user*, so the model can only ever reach rows the initiating user could reach. Every step — authorization, token exchange, tool call, RLS decision, rejection — streams an immutable event to the append-only audit store, which lives outside the reach of any agent credential.

The structural guarantees are worth naming explicitly, because they are *network facts*, not prompt instructions:

– **No public ingress.** The MCP server, authorization server, and audit store live in private subnets. The database is reachable only over private endpoints. The model — Claude, GPT-4-class, or an open-weights deployment in your own VPC — reaches the MCP server, and nothing else.
– **No service-account data access.** The database never executes a query whose effective principal is a shared agent credential. Every query is evaluated against a propagated human identity and an audited delegation chain.
– **No unlogged invocation.** A tool call that succeeds without emitting an audit event is architecturally impossible, because logging lives inside the request path.
– **No standing privilege.** Tokens are short-lived, refresh tokens rotate, and each task runs on a delegation token narrower than the user’s own grant.

Get these four guarantees right, and the agentic system becomes *containable*: the most capable tool your business has shipped, bounded by identity, scope, and evidence at every hop. Get them wrong, and you have deployed your largest attack surface with the user’s credentials attached.

—

**Planning an MCP rollout — or securing one that already shipped? Talk to the OnlineCloud architects.** We are a US-based engineering studio — MCP-native, senior engineers only, no offshore handoff — and we deliver evaluated, observable, VPC-isolated agentic systems with audited OAuth 2.1 authorization in production within 6–10 weeks. Contact us at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/) for a free initial consultation. We respond within 24 hours.

# Building a Custom MCP Server in Go: Low-Latency Protocol Implementation

*Deep-dive technical analysis for engineers shipping Model Context Protocol servers that need to survive production traffic.*


—

## Introduction: Why Go for MCP?

The Model Context Protocol (MCP) has become the de facto contract layer between LLMs and business logic. The reference SDKs are TypeScript and Python, which means most MCP servers you meet in the wild are Node or Python processes wrapping a JSON-RPC loop. That works — until your agent fleet starts hammering a shared server with parallel `tools/call` requests and the 99th percentile creeps past the timeout your orchestrator enforces.

When you need predictable latency at high concurrency, Go is the pragmatic choice. Three properties matter most:

**Concurrency.** MCP is a multiplexed protocol: a client may fire dozens of outstanding requests over one transport, and each tool call can run for seconds. Goroutines give you cheap, per-request isolation — tens of thousands of concurrent logical tasks on a few OS threads without the event-loop coordination tax you pay in single-threaded runtimes.

**Performance.** A Go MCP server is a single static binary with sub-millisecond cold start, a few tens of MB of RSS, and a tunable garbage collector. For server-side deployments behind a gateway, that means you can colocate an MCP server per tenant, per team, or per data domain without paying a per-process memory premium.

**Type safety.** Tool schemas are JSON Schema, but the code behind them is real systems code touching databases, queues, and vendor APIs. Go’s static typing means your registry, parameter decoding, and error paths are checked at compile time — not discovered at 2 a.m. when the model sends a malformed argument.

None of this is a knock on Node.js; it is an honest engineering tradeoff. When your MCP server is the hot path — many concurrent sessions, long-running tool calls, tight latency budgets — Go’s runtime model maps cleanly onto the protocol’s shape. This post walks through what it actually takes to build one: protocol internals, registry architecture, concurrency, idiomatic code, and real numbers.

—

## Protocol Internals: JSON-RPC 2.0 over stdio and HTTP

MCP is not a bespoke wire format. It is JSON-RPC 2.0 with a fixed vocabulary of methods (`initialize`, `tools/list`, `tools/call`, `resources/read`, `notifications/initialized`, …) carried over a transport. That separation — *protocol* independent of *transport* — is the first design insight. Your core dispatcher should not know or care whether a message arrived on a file descriptor or a TLS socket.

### Message shape

Every exchange is one of three JSON-RPC 2.0 message types:

– **Request** — has an `id`, a `method`, and `params`. The peer must reply.
– **Response** — echoes the `id`, carrying either `result` or an `error` object with a numeric `code`, `message`, and optional `data`.
– **Notification** — has a `method` but **no `id`**. No reply is expected or permitted. Errors in notifications are silently dropped by design; you log them server-side.

The initialize handshake deserves attention. The client sends `initialize` with its `protocolVersion` and capabilities; the server replies with its own supported `protocolVersion`, its capabilities (`tools`, `resources`, `prompts`), and `serverInfo`. Only after the client sends the `notifications/initialized` notification is the session live. Version negotiation is where incompatible servers break silently — pin your supported version and reject mismatches with a structured error instead of limping along.

### stdio framing

The stdio transport is deceptively simple: **one JSON-RPC message per line, newline-delimited UTF-8**. There is no `Content-Length` header framing like LSP — which is both a blessing and a trap.

The trap is buffering. A naive `bufio.Scanner` defaults to a 64KB token limit; a large `resources/read` result will silently blow past it. Use `bufio.Reader.ReadBytes(‘\n’)` and enforce your own max message size so a runaway model cannot OOM your server with a single 2GB frame.

“`go
func serveStdio(ctx context.Context, dispatch func(context.Context, []byte) []byte) error {
r := bufio.NewReaderSize(os.Stdin, 1<<16) w := bufio.NewWriterSize(os.Stdout, 1<<16) dec := json.NewDecoder(io.LimitReader(r, maxFrameBytes)) dec.UseNumber() for { raw, err := dec // one JSON value == one frame here ... } } ``` A minimal framing loop reads until `\n`, validates the envelope into `json.RawMessage` fields so you can dispatch without re-marshaling, and writes responses through a single `*bufio.Writer` guarded by a mutex — stdout is shared state, and interleaved partial writes from concurrent goroutines corrupt the stream. ### HTTP and streaming transport For network deployments, the modern surface is **Streamable HTTP** (the legacy SSE-only transport is deprecated): the client `POST`s JSON-RPC messages to a single endpoint, and responses come back either as plain JSON or as an SSE stream when the server needs to push multiple messages — progress notifications, partial results, long-running tool updates. Two implementation notes from the trenches: 1. **Session state lives in the message, not the connection.** MCP sessions are identified by a `Mcp-Session-Id` header, so your server must be stateless across connections or persist session state keyed by that ID (the OnlineCloud pattern: stateless servers with auth, rate limits, and audit logs out of the box). 2. **Streaming requires flush discipline.** With `net/http`, wrap the `ResponseWriter` in a flush-able writer and flush after every SSE event; otherwise the client sees a burst at the end — which defeats the entire purpose of streaming latency. Whichever transport you bind, keep the dispatch core identical. Transport is an adapter; the protocol engine is the product. --- ## Architecture: Designing the Tool and Resource Registry in Go MCP servers are, at heart, name-resolution engines: `tools/call` resolves a tool name to an executable function; `resources/read` resolves a URI to a document. The registry is the data structure that makes that resolution fast, safe, and introspectable. ### The Tool abstraction Define a `Tool` as a static description plus an executable handler. Keep the description — name, description, `inputSchema` (JSON Schema), and annotations — immutable after registration; it is what you return from `tools/list` and what the model uses to plan calls. The handler is where runtime happens: ```go type Tool struct { Name string Description string InputSchema map[string]any // JSON Schema draft-07 Handler func(ctx context.Context, args map[string]any) (*CallResult, error) } type CallResult struct { Content []ContentBlock IsError bool } type ContentBlock struct { Type string `json:"type"` // "text" | "image" | "resource" Text string `json:"text,omitempty"` } ``` Registering a tool is then a one-liner that reads like a declaration: ```go registry.Register(Tool{ Name: "lookup_order", Description: "Fetch order status by order ID and customer email.", InputSchema: map[string]any{ "type": "object", "properties": map[string]any{ "order_id": map[string]any{"type": "string"}, "email": map[string]any{"type": "string", "format": "email"}, }, "required": []string{"order_id", "email"}, }, Handler: func(ctx context.Context, args map[string]any) (*CallResult, error) { id, _ := args["order_id"].(string) // validate strictly in production return ordersClient.Lookup(ctx, id) }, }) ``` ### Lock-free reads with immutable snapshots The naive registry is a `map[string]Tool` behind a `sync.RWMutex`. That works, but the hot path — every `tools/call` and every `tools/list` — contends on the same lock, and it makes *hot reload* (registering new tools without restart) messy. Since Go 1.19, the clean pattern is `atomic.Pointer` to an immutable snapshot: ```go type Registry struct { tools atomic.Pointer[map[string]Tool] // immutable snapshot mu sync.Mutex // only for writers } func (r *Registry) Register(t Tool) error { r.mu.Lock() defer r.mu.Unlock() next := make(map[string]Tool, len(*r.tools.Load())+1) for k, v := range *r.tools.Load() { next[k] = v } next[t.Name] = t r.tools.Store(&next) // atomic swap: readers never block return nil } func (r *Registry) Get(name string) (Tool, bool) { t, ok := (*r.tools.Load())[name] return t, ok } ``` Readers take a single atomic load and read an immutable map — zero contention, no lock acquisition on the request path, and deployments can push new tool versions by re-registering while traffic flows. Duplicate names return an error at registration time rather than silently shadowing, because ambiguity in a registry that an LLM is driving is a production incident waiting to happen. ### Resources: hierarchical URIs Resources are the same idea with hierarchical addressing. Store them as a prefix tree over `scheme://host/path` URIs so `resources/list` with a `cursor` and subtree filters is O(depth), and `resources/read` is a direct lookup. Every resource should declare a content type and an optional `mimeType`; treat non-text resources (database query results, images, binary artifacts) as `resource` content blocks with a resolvable URI rather than inlining payloads you cannot size. --- ## Concurrency: Goroutines, Channels, and Backpressure Here is where Go stops being "fine" and becomes the right tool. An MCP client multiplexes many outstanding calls over one transport; your job is to execute them concurrently without thrashing the upstream systems they touch. ### The per-call execution model Dispatch every `tools/call` into its own goroutine, derived from the request context so cancellation propagates: ```go func (s *Server) handleToolCall(ctx context.Context, id any, params map[string]any) { name, _ := params["name"].(string) args, _ := params["arguments"].(map[string]any) tool, ok := s.registry.Get(name) if !ok { s.replyError(id, -32602, "unknown tool: "+name) return } go func() { result, err := tool.Handler(ctx, args) if err != nil { s.replyError(id, -32000, err.Error()) return } s.reply(id, result) }() } ``` Each handler gets an isolated goroutine and a context that is cancelled the moment the client cancels the request or the session dies. A tool that blocks on a hung database call stops blocking *your* server the instant the client gives up — no thread-per-connection ceiling, no event-loop freeze. ### Bounding fan-out with a semaphore Unbounded goroutines are their own outage. If the model calls a tool that fans out to 10,000 upstream requests, you want to cap concurrency, not explode the connection pool. A buffered channel is the idiomatic semaphore: ```go // 32 concurrent upstream executions per server, queue the rest. sem := make(chan struct{}, 32) for _, item := range batch { select { case sem <- struct{}{}: go func(item Item) { defer func() { <-sem }() if err := process(ctx, item); err != nil { errCh <- err } }(item) case <-ctx.Done(): return ctx.Err() // backpressure: stop admitting work } } ``` The `select` on `ctx.Done()` is the detail most ports get wrong — without it, a full semaphore blocks forever even after the client has cancelled. Channels give you backpressure *and* cancellation for free; the equivalent in callback-driven runtimes requires hand-rolled coordination. ### Ordered fan-in when order matters For resources and batched tools that must emit results in deterministic order, resist the temptation to coordinate with mutexes. Run workers, then collect into a pre-sized slice by index — a channel per result is overkill; one results channel plus index assignment is enough: ```go results := make([]Result, len(items)) errCh := make(chan error, len(items)) var wg sync.WaitGroup for i, item := range items { wg.Add(1) go func(i int, item Item) { defer wg.Done() r, err := process(ctx, item) if err != nil { errCh <- err return } results[i] = r // each goroutine owns its slot: no race }(i, item) } wg.Wait() close(errCh) ``` Slot-per-index assignment is race-free by construction — the compiler proves it, and you never ship a data race into a server that auditors will read. --- ## Code: Server Initialization and Tool Registration, End to End The full picture — initialize a server, register tools, serve over stdio — is compact enough to read in one screen: ```go func main() { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() reg := NewRegistry() reg.Register(newLookupOrderTool()) // as shown above reg.Register(newRefundTool()) reg.Register(newListProductsResource()) srv := NewServer(ServerConfig{ Name: "commerce-mcp", Version: "1.4.0", Registry: reg, MaxConcurrentCalls: 32, }) // Same engine, two adapters — pick at build time or runtime. switch transport := os.Getenv("MCP_TRANSPORT"); transport { case "http": log.Fatal(http.ListenAndServe(":8090", srv.HTTPHandler())) default: log.Fatal(srv.ServeStdio(ctx)) // newline-delimited JSON-RPC on stdin/stdout } } ``` That is the whole deployment story: the same compiled binary speaks stdio to a local Claude Desktop or Cursor process, or Streamable HTTP to a remote agent fleet. One engine, two adapters, zero duplicated protocol logic. --- ## Benchmarking: Go vs. Node.js for High-Throughput MCP Numbers without methodology are marketing. Here is the methodology: a headless echo tool (`tools/call` that returns its argument), a 1KB payload, both servers running the reference-grade implementations of their ecosystems (Go with the `modelcontextprotocol/go-sdk`-style architecture above; Node with the official TypeScript SDK), 8 vCPU Linux, load generated by a separate Go client over stdio to eliminate network variance. | Metric | Go server | Node.js (TS SDK) | |---|---|---| | Throughput, 1 client / 1,000 sequential calls | ~4,800 calls/s | ~2,900 calls/s | | Throughput, 64 concurrent clients | ~31,000 calls/s | ~11,500 calls/s | | p50 latency @ 64 concurrent | 1.8 ms | 3.4 ms | | p99 latency @ 64 concurrent | 6.2 ms | 21 ms | | Idle RSS | ~18 MB | ~78 MB | Two patterns are consistent across every workload we have benchmarked. First, the gap widens with concurrency: Node holds up at low fan-in, but as outstanding calls climb, p99 degrades faster than p50 — the classic single-threaded event loop under contention. Go's p99 stays flat because work is spread across all cores by the scheduler rather than serialized through one loop. Second, GC behavior matters at the tail: Node's default heap settings produce periodic multi-millisecond pauses under allocation churn; Go's `GOGC`/`GOMEMLIMIT` knobs let you trade throughput for bounded pause times on latency-critical sessions. These are microbenchmarks of the protocol core, not your workload. Your tool handlers — database calls, vendor APIs, ML inference — will dominate end-to-end numbers, which is exactly the point: choose the runtime that makes the *protocol overhead* disappear so your handlers, not your framework, define the latency budget. If you are CPU-bound in pure JSON-RPC dispatch, Go wins decisively; if you are I/O-bound in handlers, both are fine and you should optimize upstream. --- ## The Bottom Line MCP's JSON-RPC core is small enough to implement correctly and generically in Go — that is precisely why the transport layer should be an afterthought. With immutable snapshot registries, goroutine-per-call dispatch, bounded semaphores, and one engine behind stdio and HTTP adapters, you get a server whose overhead is measurable in microseconds and whose failure modes are the ones you designed. Building an MCP server that will sit between an LLM and your production data is a systems engineering decision, not a scripting exercise — schema rigor, auth boundaries, rate limits, and audit trails all live in the layer underneath the protocol. That is the kind of problem we solve daily at **OnlineCloud**, an MCP-native engineering studio shipping evaluated, observable agent systems into production. If you are architecting a custom MCP server in Go — or anywhere else — and want it reviewed, benchmarked, or built end-to-end, our senior architects are available. **[Contact the OnlineCloud architects](https://www.onlinecloud.dev/)** — email **contact@onlinecloud.dev** for a free initial consultation, or tell us about your project at [onlinecloud.dev](https://www.onlinecloud.dev/). US-based senior engineers, no offshore handoffs, shipped to production in 6–10 weeks.

# Implementing SSE Transports for MCP: A Step-by-Step Guide for Node.js


If you have shipped a Model Context Protocol (MCP) server to production, it was probably over **stdio** — a child process spawned by Claude Desktop, a VS Code extension, or some local agent harness. That works beautifully for a single user on a single machine, and it is where most of us started. But stdio is a dead end the moment you need what every other backend service takes for granted: centralized auth, audit logs, horizontal scaling, and serving many concurrent clients from one deployment.

This guide is the bridge from *local process* to *network service*. We will walk through why Server-Sent Events (SSE) is the right transport for MCP’s asymmetric message flow, and then build a production-minded SSE MCP server in Node.js with Express — covering connection lifecycle, heartbeats, session state, and the security posture you need before this thing touches the public internet.

—

## 1. Moving Beyond stdio: Network-Distributed MCP

MCP is a JSON-RPC 2.0 protocol layered over a pluggable transport. The spec defines messages like `initialize`, `tools/call`, and `notifications/progress`, but deliberately says nothing about *how* those bytes travel — that is the transport’s job. The reference implementations give you three options:

– **stdio** — parent process spawns the server and speaks over stdin/stdout.
– **HTTP + SSE** — the client opens a server-sent event stream for server→client messages and POSTs client→server messages to a dedicated endpoint.
– **Streamable HTTP** (newer spec revisions) — collapses both directions onto a single POST endpoint.

stdio has real virtues: zero network surface, trivial auth (the OS owns the pipe), and automatic process lifecycle. But it couples your MCP server’s lifetime to a single client process. You cannot put stdio behind a load balancer. You cannot share one deployment across a fleet of agents. You cannot audit who called `tools/call` because there is no *who* — only a pipe.

Once your MCP server becomes a shared internal service — wrapping your CRM, your Oracle or MS SQL estate, your ticketing system — you need a transport that speaks HTTP. SSE is the pragmatic sweet spot: it rides plain HTTP (so every piece of middleware and infrastructure you already run understands it), it handles the server→client push channel that HTTP request/response alone cannot, and it is dramatically simpler to operate than a WebSocket fleet.

—

## 2. What Is SSE, and Why It Fits MCP’s Asymmetric Flow

Server-Sent Events is a one-way push protocol built directly on HTTP. The client opens a normal GET request; the server keeps the response open and streams `text/event-stream` frames to it. Each frame looks like:

“`
event: message
data: {“jsonrpc”:”2.0″,”id”:1,”result”:{}}
“`

The `data:` field carries the payload, `event:` names it, and comment lines starting with a colon (`: ping`) are ignored by the client — a detail we will exploit for heartbeats shortly.

Here is the key insight about MCP: **the traffic pattern is asymmetric, and it matches SSE’s one-way design perfectly.**

– **Client → Server** messages (`initialize`, `tools/call`, `resources/read`) are discrete request/response operations. They map naturally onto plain HTTP POST with a JSON body — easy to log, authenticate, rate-limit, and load-balance.
– **Server → Client** messages are the hard part. Responses can arrive seconds or minutes after a request (think long-running tool calls). And critically, the *server* must sometimes initiate traffic: `notifications/progress`, `notifications/logging`, `notifications/resources/updated`, and — in some configurations — server→client *requests* like `sampling/createMessage` where the model asks the host for context.

Plain request/response HTTP cannot represent a server-initiated push. WebSocket can, but it is a heavier contract: a protocol upgrade, a different connection lifecycle, frame parsing, and notoriously finicky behavior through corporate proxies and naive load balancers. SSE needs none of that — it is just a long-lived HTTP response. Every reverse proxy, CDN, and observability stack that understands HTTP understands SSE (give or take a buffering directive, which we will cover).

The result is a clean split that senior backend engineers will recognize and love:

| Direction | Transport | Semantics |
|—|—|—|
| Client → Server | `POST /messages` | Standard HTTP request/response, JSON-RPC body |
| Server → Client | `GET /sse` (held open) | Event stream for results, notifications, and server-initiated requests |
| Session binding | `sessionId` query param | Both legs address one logical MCP session |

One honest caveat before we commit: the MCP spec’s newer **Streamable HTTP** transport is where the protocol is headed, and it lets a single POST endpoint handle both directions (when the server has nothing to push, it can respond with an SSE stream or a plain JSON body). For existing SDKs and a large installed base of clients, the classic HTTP+SSE split remains the most interoperable choice today — and everything you learn about session lifecycle and connection management here transfers directly to Streamable HTTP later.

—

## 3. Setting Up Express.js for SSE-Based MCP Transport

Let’s scaffold. We will use the official `@modelcontextprotocol/sdk` for the protocol machinery (tool registration, JSON-RPC framing, capability negotiation) and Express purely as the HTTP layer. Do not reimplement JSON-RPC validation yourself — the SDK is battle-tested and versioned against the spec.

“`json
{
“name”: “mcp-sse-server”,
“type”: “module”,
“dependencies”: {
“@modelcontextprotocol/sdk”: “^1.x”,
“express”: “^4.19.0”,
“cors”: “^2.8.5”,
“helmet”: “^7.1.0”,
“zod”: “^3.23.0”
}
}
“`

Our HTTP surface is intentionally tiny — two routes, which is the whole point of SSE’s simplicity:

– `GET /mcp/sse` — the client connects here and receives the event stream.
– `POST /mcp/messages` — the client sends JSON-RPC messages here, tagged with `?sessionId=`.

The SSE handshake deserves attention because it is where beginners get confused. When a client connects to `/mcp/sse`, the server does not start processing tools yet. It first sends a special `endpoint` event telling the client *where* to POST:

“`
event: endpoint
data: /mcp/messages?sessionId=a1b2c3d4-e5f6-…
“`

Only after the client receives that event does it know the message URL. This indirection exists because the server may be behind a reverse proxy and needs to advertise its *public* route — so build the endpoint URI from the request’s host header, not from a hardcoded string.

The SDK’s `SSEServerTransport` handles this handshake and the JSON-RPC session plumbing for you. Here is the Express wiring:

“`js
import express from “express”;
import cors from “cors”;
import helmet from “helmet”;
import { McpServer } from “@modelcontextprotocol/sdk/server/mcp.js”;
import { SSEServerTransport } from “@modelcontextprotocol/sdk/server/sse.js”;
import { z } from “zod”;

const app = express();

// JSON body parsing only where it belongs: the POST message endpoint.
// Never apply express.json() globally before your SSE route.
app.use(helmet());
app.use(cors({ origin: process.env.CORS_ORIGIN?.split(“,”) ?? “*” }));
app.post(“/mcp/messages”, express.json({ limit: “1mb” }));

// The MCP server itself: register tools, resources, prompts.
const server = new McpServer(
{ name: “orders-mcp”, version: “1.0.0” },
{ capabilities: { tools: {} } }
);

server.registerTool(
“lookup_order”,
{ orderId: z.string().describe(“Order identifier”) },
async ({ orderId }) => ({ content: [{ type: “text”, text: `order:${orderId}` }] })
);

// One live SSE transport per connected session.
const sessions = new Map();

app.get(“/mcp/sse”, async (req, res) => {
// … auth check (Section 5) …
const transport = new SSEServerTransport(“/mcp/messages”, res);
sessions.set(transport.sessionId, transport);
res.on(“close”, () => {
sessions.delete(transport.sessionId);
// … cancel heartbeat timers …
});
await server.connect(transport);
});

app.post(“/mcp/messages”, async (req, res) => {
const transport = sessions.get(String(req.query.sessionId ?? “”));
if (!transport) {
res.status(404).json({ error: “Unknown session” });
return;
}
await transport.handlePostMessage(req, res);
});

app.listen(process.env.PORT ?? 3001, () =>
console.log(`MCP SSE server listening on :${process.env.PORT ?? 3001}`)
);
“`

A few things worth internalizing about this code:

1. **Body-parser placement matters.** `express.json()` on the GET stream route would try to read the never-ending body of an SSE connection. Apply it only to the POST route, as shown.
2. **Session registry is the glue.** `SSEServerTransport` generates a `sessionId` and embeds it in the `endpoint` event. Your POST handler must look up the matching transport by that ID. Every MCP deployment is a session store plus two routes.
3. **`helmet()` before everything.** It sets sensible security headers (`X-Content-Type-Options`, `X-Frame-Options`, etc.) on every response, including your stream. Cheap defense, always on.

If you would rather not use the SDK’s transport class, the manual recipe is: set `Content-Type: text/event-stream`, `Cache-Control: no-cache`, `Connection: keep-alive`, and `X-Accel-Buffering: no`; write `event: endpoint\ndata: \n\n`; then write each outbound JSON-RPC message as a `message` event. But you would be re-deriving session and framing logic the SDK already maintains — use the SDK, keep your custom code for the parts that differentiate you (auth, lifecycle, observability).

—

## 4. Handling Connections: Heartbeats, Timeouts, and Stateful Sessions

An SSE connection is an HTTP response held open indefinitely, and that is exactly what infrastructure tries to kill. Load balancers (nginx `proxy_read_timeout` defaults to 60s), cloud LBs, and corporate proxies all reap idle connections. Worse, some buffering proxies won’t flush anything to the client until they see enough bytes, which stalls your stream and breaks the MCP handshake entirely.

### Heartbeats

The fix is comment frames. An SSE comment — a line starting with `:` — is ignored by conformant clients but counts as traffic for intermediaries:

“`js
function startHeartbeat(res, sessionId, intervalMs = 15_000) {
const timer = setInterval(() => {
if (res.writableEnded) {
clearInterval(timer);
return;
}
res.write(“: heartbeat\n\n”);
}, intervalMs);
// Unref so the timer never keeps the process alive on its own.
timer.unref?.();
return timer;
}
“`

Fifteen seconds is a safe default: comfortably under the typical 30–60s proxy idle timeout, cheap at scale (a few bytes per connection per 15s), and invisible to clients. Also send the `X-Accel-Buffering: no` header on the stream response so nginx does not buffer your events.

### Timeouts and session TTL

Heartbeats keep *proxy* connections alive; they do not tell you whether the *client* is still there. A client that vanished (laptop asleep, network drop, process killed) leaves your session registry holding a zombie. Two mechanisms handle this:

– **Detect dead sockets.** The `close` event on `res` fires when the underlying socket dies — that is your primary cleanup hook, shown above. Test it: kill an `EventSource` client and confirm your registry drains within seconds.
– **Sweep stale sessions.** For defense in depth, track `lastSeen` per session (touch it in the POST handler) and run a periodic sweeper that terminates sessions idle beyond a TTL — 5–10 minutes of zero traffic is a good default for MCP, where clients may hold long-running tool calls. Combine the socket `close` event with the sweeper, not either one alone.

### Statefulness: the part everyone underestimates

MCP over SSE is **stateful by design**. The registry maps `sessionId` → transport because both legs of the conversation must reach the same server instance that owns the open stream — there is no way to POST a response to an SSE connection living on another node. Concretely:

– **Sticky sessions are mandatory** if you scale to multiple instances behind a load balancer, or accept that sessions pin to one node.
– **Reconnects create new sessions.** When a client’s `EventSource` retries, it issues a fresh `GET /sse`, gets a *new* `sessionId`, and the old session must be torn down. Design your tools to be idempotent and your resources to be re-fetchable, because MCP session state (initialization context, subscription sets) does not survive a transport reconnect.
– **A server restart orphans every client.** Plan for graceful shutdown: on `SIGTERM`, stop accepting new sessions, notify connected clients with an error event, and drain in-flight tool calls within a bounded window.

Treat the session registry as a first-class piece of application state. Put metrics on it — active sessions, reconnects per minute, zombie reaps — because connection churn is the earliest signal that something between your clients and server is misbehaving.

—

## 5. Security: CORS, Auth Headers, and Secure Transport

Putting MCP on the network means exposing your tool surface — potentially your *data* — to whoever can reach the endpoint. Treat an MCP server like any other API that gates sensitive operations.

### Authentication

Authenticate **both** legs, before any work happens:

– On `POST /mcp/messages`, validate a bearer token in the `Authorization` header against your auth service before dispatching to the transport.
– On `GET /mcp/sse`, do the same *before* opening the stream — the cheapest request you will ever deny is one you never let connect.

The subtlety that bites people: **the browser-native `EventSource` API cannot set custom headers.** You literally cannot send an `Authorization: Bearer …` header from it. Your options:

1. **Use a fetch-based SSE client** (e.g., `@microsoft/fetch-event-source`) that can send headers on the GET.
2. **Accept a short-lived, signed token in the query string** for the GET leg only, since you cannot avoid it with `EventSource`. Sign it with an HMAC, scope it to one session, and expire it in minutes. Never put a long-lived credential in a URL — it ends up in proxy logs, browser history, and referrer headers.
3. **Cookies + CSRF protection**, if the consumers are all browser contexts and you control the origin policy. More moving parts; usually not worth it for backend-to-backend MCP.

When MCP clients are server-side agents (the common case), header-based bearer tokens on both routes plus a fetch-based SSE client is the clean answer.

### CORS

MCP is not a browser API today, but the day an agent runs in a browser extension, CORS becomes your first line of defense. Do not use `*`. Enumerate origins, and remember CORS is enforced by the *client’s* browser — it stops malicious web pages from POSTing to your server, it does nothing against a compromised backend that has your token. Restrict origins, then rely on auth.

“`js
const allowed = new Set((process.env.CORS_ORIGIN ?? “”).split(“,”));
app.use(cors({
origin: (origin, cb) =>
cb(null, !origin || allowed.has(origin)), // block no-origin & unknown
methods: [“GET”, “POST”],
allowedHeaders: [“Content-Type”, “Authorization”],
maxAge: 600
}));
“`

### The transport layer

– **TLS everywhere**, terminated at your edge. Add HSTS via helmet and force HTTPS redirects.
– **Never** put an SSE or messages endpoint in front of a naive cache — the `Cache-Control: no-cache` on the stream response is not a suggestion.
– **Rate-limit per session and per token** at the POST endpoint; a runaway agent loop should trip your limits, not your database.
– **Validate and bound** JSON-RPC bodies (`express.json({ limit: “1mb” })`), and redact request/response payloads in logs — tool arguments routinely contain PII.
– **Authenticate outbound too.** If your tools call third-party APIs, those credentials belong in a secret manager, injected per-session, and never echoed in tool results.

—

## 6. Full Code Implementation

Here is the complete, runnable server combining everything above. Save it as `server.js`:

“`js
import express from “express”;
import cors from “cors”;
import helmet from “helmet”;
import crypto from “node:crypto”;
import { McpServer } from “@modelcontextprotocol/sdk/server/mcp.js”;
import { SSEServerTransport } from “@modelcontextprotocol/sdk/server/sse.js”;
import { z } from “zod”;

const PORT = Number(process.env.PORT ?? 3001);
const SESSION_TTL_MS = 10 * 60 * 1000;
const HEARTBEAT_MS = 15_000;
const SWEEP_MS = 60_000;

const app = express();
app.use(helmet());
app.use(cors({
origin: (origin, cb) => {
const allowed = (process.env.CORS_ORIGIN ?? “”).split(“,”).filter(Boolean);
cb(null, !origin || allowed.includes(origin));
}
}));

// —- Auth: shared by both legs (see Section 5) —-
function authenticate(req, res, next) {
const expected = process.env.MCP_API_TOKEN;
const token = req.headers.authorization?.replace(/^Bearer\s+/i, “”);
if (!expected || token === expected) return next();
res.status(401).json({ error: “Unauthorized” });
}

// —- MCP server (tools, resources, prompts) —-
const mcp = new McpServer(
{ name: “orders-mcp”, version: “1.0.0” },
{ capabilities: { tools: {}, logging: {} } }
);

mcp.registerTool(“lookup_order”, { orderId: z.string() }, async ({ orderId }) => {
// Real implementation: query your database / CRM / ERP here.
return { content: [{ type: “text”, text: `Fetched order ${orderId}` }] };
});

// —- Session registry with heartbeats & TTL sweeper —-
const sessions = new Map(); // sessionId -> { transport, res, lastSeen, heartbeat }

function startHeartbeat(res) {
const timer = setInterval(() => {
if (!res.writableEnded) res.write(“: hb\n\n”);
else clearInterval(timer);
}, HEARTBEAT_MS);
timer.unref?.();
return timer;
}

const sweeper = setInterval(() => {
const now = Date.now();
for (const [id, s] of sessions) {
if (now – s.lastSeen > SESSION_TTL_MS) {
s.res.end();
sessions.delete(id);
clearInterval(s.heartbeat);
console.warn(`[sweeper] reaped idle session ${id}`);
}
}
}, SWEEP_MS);
sweeper.unref?.();

// —- Leg 1: the SSE event stream (server -> client) —-
app.get(“/mcp/sse”, authenticate, async (req, res) => {
res.setHeader(“Cache-Control”, “no-cache”);
res.setHeader(“X-Accel-Buffering”, “no”);

const transport = new SSEServerTransport(“/mcp/messages”, res);
const sessionId = transport.sessionId;

sessions.set(sessionId, {
transport,
res,
lastSeen: Date.now(),
heartbeat: startHeartbeat(res),
});

res.on(“close”, () => {
const s = sessions.get(sessionId);
if (s) {
clearInterval(s.heartbeat);
sessions.delete(sessionId);
}
console.log(`[sse] client disconnected: ${sessionId}`);
});

await mcp.connect(transport);
console.log(`[sse] session established: ${sessionId}`);
});

// —- Leg 2: client -> server JSON-RPC over plain POST —-
app.post(“/mcp/messages”, authenticate, express.json({ limit: “1mb” }), async (req, res) => {
const s = sessions.get(String(req.query.sessionId ?? “”));
if (!s) return res.status(404).json({ error: “Unknown or expired session” });
s.lastSeen = Date.now();
await s.transport.handlePostMessage(req, res);
});

// —- Graceful shutdown —-
for (const sig of [“SIGINT”, “SIGTERM”]) {
process.on(sig, () => {
console.log(`[shutdown] ${sig} received, closing ${sessions.size} session(s)`);
for (const s of sessions.values()) s.res.end();
clearInterval(sweeper);
process.exit(0);
});
}

app.listen(PORT, () => console.log(`MCP SSE server on :${PORT}`));
“`

A minimal client to verify the flow — note the fetch-based SSE client so we can send the auth header:

“`js
import { fetchEventSource } from “@microsoft/fetch-event-source”;

await fetchEventSource(“http://localhost:3001/mcp/sse”, {
headers: { Authorization: “Bearer ” + process.env.MCP_API_TOKEN },
async onopen(res) { /* 200 -> start MCP initialization */ },
onmessage(ev) { console.log(ev.data); }, // JSON-RPC frames
onerror(err) { /* backoff & reconnect -> NEW session */ }
});
“`

**Run it:** `npm install && MCP_API_TOKEN=dev-token PORT=3001 node server.js`. Wire in any MCP client (Claude Desktop with a URL, the SDK’s client, or a custom agent) pointed at `http://localhost:3001/mcp/sse`.

**Before production**, add the things this skeleton intentionally leaves to you: real auth integration, structured logs with request IDs, per-token rate limits, an observability endpoint exposing `sessions.size`, and sticky sessions or a shared session registry (Redis pub/sub or similar) if you scale past one node.

—

## Conclusion

SSE is not the newest transport in the MCP spec — Streamable HTTP is — but it remains the most instructive and, for many deployments, the most operationally boring (in the best way): two HTTP routes, one held-open response per client, and every standard piece of HTTP infrastructure working in your favor. Master the session lifecycle — handshake, heartbeat, TTL sweep, graceful shutdown — and the security posture around both legs, and you have a template that will carry you through Streamable HTTP and whatever comes after.

Putting MCP servers on the network is exactly the kind of architecture work that looks simple on a whiteboard and gets subtle in production: real database schemas, real auth boundaries, real session semantics. That is what we do all day at OnlineCloud — MCP-native architecture, audited end-to-end, shipped by senior US-based engineers in weeks, not quarters. If you are moving your tool surface from stdio to the network, or designing a multi-agent system that needs Streamable HTTP, SSE, and the operational discipline around them, **talk to the OnlineCloud architects** — we would rather discuss your transport strategy than sell you a deck.

# Observability for AI Agents: Tracking Decision Logs, Cost Ceilings, and Token Drift

*What an SRE learns when the thing on call is a stochastic loop — and how to make agentic AI operable, budgetable, and auditable.*


## 1. The Black Box Problem: Why “It Worked in a Demo” Isn’t Enough

Every agent demo has the same shape: a prompt goes in, the model picks a tool, the tool returns, the model answers — and the room applauds. A demo is a single golden path, executed once, by the people who built it. It proves the model *can* do the task. It proves almost nothing about whether the system should run unattended against production data.

Production is the tail: the ambiguous request at 2 a.m., the tool that times out on the third retry, the user who phrases a refund request as a lawsuit threat. And an agent fails in ways a deterministic API never does. A classic service fails with a 5xx and a stack trace. An agent fails by doing the wrong thing confidently, or doing the right thing seventeen times in a loop, or burning $40 of tokens deciding it is confused. **The failure modes of an agentic system are not cheaper than its success modes — they are more expensive.**

That is the black box problem. Traditional observability assumes a request is reconstructible: deterministic code path, logs, a trace. Agents break that assumption three ways at once:

– **The input is underspecified.** Between the prompt and the answer, the agent made choices — which tool, which arguments, which plan — that exist nowhere in the request.
– **The code path is sampled, not fixed.** Two runs of the same prompt can take entirely different routes. “What did the system do?” has a per-run answer, not a single one.
– **Failure is behavioral, not technical.** No exception fires when an agent hallucinates an order status. The system returns 200 OK. It was wrong in a way only a downstream human notices.

“It worked in a demo” is how you *start* an agent project, not how you run one. The question that separates a prototype from a platform is not “can the model do it?” but **”can we see what it did, bound what it costs, and prove it still works next quarter?”** That is an observability problem — and the subject of this post.

## 2. Decision Logging: Replaying Agentic Choices for Full Transparency

If the problem is invisible behavior, the fix is not transcript logging — it is **decision logging**: capturing every meaningful choice in structured form so a run can be replayed like a trace.

The mental model comes from distributed tracing. A microservice emits a span per downstream call; an agent is the same shape with a nondeterministic planner on top. Each plan-act-reflect iteration is a span, and the *decision* is its payload. Every step should emit an event roughly like this:

“`json
{
“trace_id”: “8f3a1c…”,
“step”: 4,
“agent”: “order-status-closer”,
“model”: “claude-sonnet-4-20260514”,
“decision”: “call_tool”,
“tool”: “get_order_status”,
“rationale”: “customer asked for ETA; order id present”,
“prompt_version”: “v27”,
“tokens”: { “input”: 1840, “output”: 96 },
“latency_ms”: 842,
“cost_usd”: 0.0142,
“ts”: “2026-09-06T04:12:31.002Z”
}
“`

This buys what a transcript cannot. **Replayability:** with a trace ID, an engineer reconstructs why an agent escalated a customer or opened a case — weeks later. “Why did this happen?” becomes a query against structured logs, not a meeting. **Comparability:** because decisions are structured, you can diff run 4 against run 12 of the same request type and see where paths diverged — which is how you debug a regression that only shows on Tuesdays. **Auditability:** when an agent touches customer data or takes a side-effecting action, decision logs are the control — who authorized the run, which policy version was active, which tool fired, what it returned — without exposing raw transcripts to every auditor. This is why production Model Context Protocol (MCP) servers ship audit logs as a first-class feature: typed tools and auditable transports are the contract layer that makes decision logging possible instead of aspirational.

Two rules keep it honest. **Log the decision, not just the outcome** — including refusals and skipped tools; if you only log successful calls, the one event you need in the postmortem is the one you never recorded. **Keep payloads out of the hot path** — store full tool inputs and outputs in object storage keyed by trace ID; replay a run on demand rather than paying to store every token forever.

## 3. Cost Ceilings: Preventing the $1,000 Recursive Loop

Every team that has run an agent in production for more than a month has a recursive-loop story. The agent tries a tool, the tool errors, it retries with different arguments, the tool errors again — and because each failure looks like progress to the model, the loop never terminates on its own. Multiply a stuck loop by a pricey model by a growing context window, and one conversation quietly accrues a bill that outranks your entire CI fleet.

The math is the trap. Per-token pricing feels cheap, so teams reason in pennies and discover the real cost is **steps × tokens × price**, with every iteration feeding its own output back into an ever-longer context. A run that should have been three steps and $0.05 becomes forty steps and $12 — confidently failing the whole time. The $1,000 recursive loop needs no malice; it needs one tool with ambiguous error semantics and no ceiling.

Treat cost as a **hard resource constraint enforced at runtime**, the way you enforce memory limits on a container — not a line item discovered in the billing export. Production agents need ceilings at every layer:

– **Per-step budget** — hard token and wall-clock caps on any single model call.
– **Per-run ceiling** — max iterations (8–15 for well-scoped tasks) and a dollar cap per conversation. Exceeding either terminates *gracefully*: a structured handoff to a human, not a silent 500.
– **Per-tenant and org budgets** — one confused tenant cannot burn the shared quota; daily and monthly caps with alerts at 50%, 80%, and 100%.
– **Retry discipline** — repeated tool failures trigger backoff, then a circuit breaker that stops calling that tool for the run. A loop that failed three times will not succeed on the thirtieth.

The architectural principle: **check before you spend, not after.** Decrement the run’s cost ledger *before* each tool call; if the step would exceed the remaining budget, refuse and escalate. And pair ceilings with human checkpoints on anything irreversible — a refund, a record update, an email. The ceiling prevents the runaway; the checkpoint prevents the regret. An unbounded agent loop is just a process you forgot to limit.

## 4. Token Drift: Monitoring Accuracy and Model Performance Over Time

The observability failure that surprises even teams with solid logging and ironclad budgets: **the system silently gets worse, and nobody changed anything.** The agent that resolved 92% of order-status tickets in March resolves 84% in August. No code changed, no prompt changed. What drifted?

Token drift is the umbrella term for the slow, silent shifts that degrade LLM systems without a deploy event:

– **Input drift** — what users ask changes over time: new products, new edge cases, seasonal volume. The request distribution you eval’d in March is not August’s.
– **Output drift** — model behavior changes under you. Providers ship weight updates, deprecate versions, or change routing; your prompt was tuned against a snapshot that no longer exists.
– **Context drift** — the systems the agent reads drift too: schema changes, new status codes, data-quality rot in the CRM — tools return shapes the planner was never trained on.

Drift is invisible to classic SLOs: error rates stay flat, latency stays flat. The agent is *succeeding* at slightly the wrong thing, refusing slightly more often, taking nine steps where it used to take five. The symptoms hide in distributions, not averages.

The fix is a **golden eval set run on a schedule** — regression testing applied continuously. Keep 100–300 labeled production-like tasks with known-good answers, and run the agent against them nightly and on every intended change. Track completion rate, accuracy, and steps-to-success over time: a completion chart trending down while nothing else changed *is* the drift alarm. Gate every change on it — a model upgrade, prompt tweak, or provider switch that underperforms the incumbent does not ship, no matter how good the vendor’s benchmark card looks. And watch the distributions — output-length histograms, per-tool call frequency, refusal rates — because when those shift, the drift started before the score moved. Teams that skip this discover drift through escalated tickets. Teams that run it catch drift while it is still a line on a chart.

## 5. Dashboards for AI: Visualizing Accuracy, Latency, and ROI

A dashboard is the operating surface where an on-call engineer decides whether to page someone, a product owner decides whether to ship, and a CFO decides whether to renew. Agentic AI needs panels for all three.

**For the SRE on call**, the agent is a service with unusual failure modes: request volume, p50/p95 step latency, error and escalation rates, and a step-count distribution — a fat tail past ten steps is an incident, not a statistic. Watch per-model token consumption and budget burn against ceilings; alert on p95 latency, escalation spikes, and every ceiling hit, because ceiling hits are the signature of a loop gone wrong.

**For the product owner**, one question matters: *is it working?* Task-completion rate by intent, golden-set accuracy, escalation rate, steps-per-success. If completion drops or steps climb, quality is degrading before customers complain.

**For finance**, the dashboard is an ROI statement: cost per resolved task, cost per deflection versus the human-handling cost it replaces, monthly burn by model and tenant.

| Row | Audience | Panels |
|—|—|—|
| 1 | SRE | Request volume, p50/p95 step latency, error rate, ceiling hits |
| 2 | Product | Completion rate by intent, golden-set accuracy, escalation rate, steps-per-success |
| 3 | Finance | Cost per resolved task, $/request by model, budget burn vs. ceiling |
| 4 | Everyone | Model version mix, token drift indicators, decision-log anomalies |

Underneath it all: **an error budget with consequences.** If the agent’s SLO is 95% task completion on the golden set, a sustained breach triggers the same machinery as an availability breach — a review, a documented decision, a remediation plan. A dashboard with no alerting is a museum exhibit. OnlineCloud’s delivery philosophy is blunt on this: *we instrument outcomes — accuracy, latency, dollars per request — and share the dashboards.* Not slideware. The actual metric is whether the system is measurably cheaper and more accurate than the human process it replaced — this week, not launch week.

## 6. The 6–10 Week Operating Model: Ownership After Launch

Observability is not a phase that ends at go-live; it is what makes go-live survivable. Teams fail at agentic AI not during the build but in the first month of operations, when the builders have moved on and nobody owns the drift charts, the eval runs, or the cost alerts. The operating model has to be designed up front, with named ownership after launch.

A sane cadence for a well-scoped agent compresses to 6–10 weeks precisely because observability is built in from week one, not bolted on in week nine:

– **Week 1 — discovery.** Map the systems, pick the model class, and write the eval that defines “done.” If you cannot define correct in week one, you cannot measure drift in week twelve.
– **Weeks 2–7 — vertical slices.** Ship one usable slice every one to two weeks against real data — each instrumented with decision logging, cost ceilings, and its own dashboard from day one in front of users.
– **Weeks 8–10 — harden and hand over.** Load-test the failure paths, tune the ceilings, run the golden set, stand up the on-call rotation and runbooks. Launch is the *start* of operations, not the end of the project.

Then the model that keeps it honest after launch — and where most vendor relationships end and OnlineCloud’s begins: the team that built the system owns the dashboards, the retraining cadence, and the on-call. Concretely: a **weekly operating review** of drift charts, ceiling hits, escalations, and a sample of decision logs; a **model review board** — no prompt or model swap ships without passing the golden set; a **monthly cost review** reconciling burn against the ROI panels; and a real **on-call path** — when the completion chart trends down, someone with authority to roll back a prompt or escalate a model issue gets paged, not an email alias read on Tuesday.

The 6–10 week promise is not speed for its own sake. It is the consequence of instrumenting from day one: when the eval exists before the code, when decision logs ship with the first slice, when ceilings are in the architecture rather than the incident report — launch becomes an increment, not a cliff. Ownership after launch is what turns a shipped agent into a *trusted* one that gets sharper month over month instead of rotting quietly in a corner of the org chart.

## Conclusion: Ship Agents You Can See, Bound, and Prove

These three disciplines are one stance applied to the three things an agent can do that a normal service cannot: **decide, spend, and change.** Decision logging makes the deciding visible and replayable. Cost ceilings make the spending bounded. Drift monitoring makes the changing measurable and gateable. Dashboards and a real operating model turn all three into a routine that survives contact with production.

If you are standing up agentic AI against real systems — Oracle, MS SQL, Postgres, your CRM — and you want it to do more than impress a demo audience, talk to the OnlineCloud engineers. We scope the data, pick the model class, and write the eval that defines “done” in a one-week discovery sprint; we ship instrumented vertical slices every one to two weeks; and after launch we own the dashboards, the drift charts, and the on-call. That is the difference between a black box and a system your operations team can sleep through. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) to start the conversation — and bring the workload you are tired of babysitting.