Blog

# Vector Database Sharding: Architecting for Billion-Scale Agentic Retrieval


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: When the Single Node Stops Being Enough

The pattern is always the same. Somewhere in month three of a production agent rollout, a 2 a.m. alert fires: vector search p95 latency has tripled, recall is silently degrading, and the ingest queue is backing up. Your Pinecone pod — or self-hosted pgvector instance — sits at 95% memory. The HNSW graph has outgrown its cache-friendly working set, every query is a disk-page fault, and the agent that answered in 300 milliseconds now carries the slowest dependency in its own loop.

This is not a tuning problem. A single-node vector store has hard ceilings: an HNSW graph must be largely resident in RAM to deliver its advertised latency, pgvector’s IVFFlat must keep inverted lists on disk without turning scans into random-I/O storms, and both absorb every write and query from every tenant on the box. Once the working set — vectors, graph metadata, tenant filters — exceeds one node, the answer is the one every database eventually reaches: **shard it**.

But sharding a vector store is not like sharding a relational database: there is no primary key to range-scan for a point lookup, and every similarity search is a *global* read — to find true neighbors you must examine enough of the corpus that you cannot miss them. That property changes everything downstream. The rest of this post takes each problem in turn: partitioning, index tuning, routing, tenant isolation, and the path to trillion-scale retrieval.

## 2. Sharding Strategies: Metadata Partitioning vs. Random Hash Distribution

Every sharding conversation starts with one question: *on what key do we split the corpus?* Two families dominate, and the choice is the highest-leverage decision you will make — it fixes your query fan-out, index sizes, and isolation story for the life of the system.

**Metadata-based partitioning.** Pick a high-cardinality attribute every query carries — tenant ID, product category, collection, region — and assign each value to a fixed shard. All vectors sharing the attribute land on one node. The payoff: a scoped query touches exactly one shard, the graph stays small and hot, and latency matches a single node. For agentic workloads this is enormous, because agents issue *many* retrieval calls per task — a support agent filters every `search_knowledge_base` call by tenant and SKU family. Ten scoped calls mean ten cheap local searches, not ten cluster-wide scatter-gathers.

The tax is skew. Tenants are never uniform; the largest may hold 40% of vectors while the long tail shares the rest, and the node holding the anchor tenant becomes the bottleneck you sharded to escape. You mitigate with two-level placement — coarse shard per attribute, replication or re-sharding of hot shards — but you are now operating a placement system, not a database. And cross-attribute queries (“find similar incidents across all tenants for compliance review”) fan out to every shard.

**Random hash distribution.** Assign every vector to a shard by hashing its ID, yielding statistically uniform partitions, perfect load balance, and trivial elasticity — add a node, rehash a slice. This is the strategy for genuinely global queries: semantic search over a public corpus, deduplication, “find anything like this anywhere.”

The cost: *every* query is cluster-wide. With hash distribution you cannot know which shard holds the true neighbors, so a top-k search fans out to all N shards and merges. Recall now depends on per-shard over-fetch — each shard must return 4–10× the final k so boundary neighbors survive the merge — and per-shard graphs are larger than under attribute partitioning at equal total data, since each covers a random sample rather than a semantic slice.

The architect’s heuristic is blunt: **partition on the attribute your queries always carry; hash when they carry nothing.** Mature systems do both — hash the corpus into balanced shard groups, then partition within the group on tenant or collection so scoped queries resolve to one shard inside a balanced group. The routing layer is what makes the hybrid viable.

## 3. Retrieval Performance: Optimizing HNSW and IVFFlat for Distributed Search

Sharding does not remove index tuning — it multiplies it, and adds a new failure mode: **perfect local indexes, broken global recall.** Decide what quality means across shards before touching a single parameter.

With hash distribution, asking each shard for its local top-k and merging loses recall at the boundaries: the true nearest neighbor may rank fifth on its home shard and never make the cut. The fix is oversampling — query each shard for `k × factor` candidates and merge with a global distance re-rank. Per-shard candidate count grows with roughly the *log* of shard count, not linearly. Measure it: run a labeled eval set against 1, 4, 16, and 64 shards and plot recall against the oversampling factor until the curve flattens. That curve is your capacity plan.

**HNSW distributed.** HNSW’s navigable small-world graph delivers logarithmic search, but neighbor lists add 30–100+ bytes per vector on top of the embedding, all of it wanting RAM. Distribution gives two levers a single node lacks. First, *right-size the graphs*: a 50-million-vector shard does not need `M = 32` links per node — drop to `M` 12–16 and `efConstruction` 100–200, shrink memory, and let oversampling cover the local recall loss at merge. Tune per-shard recall to the *merged* target, not 0.99 locally. Second, *exploit partition locality*: under attribute partitioning, each shard’s graph covers one semantic region — smaller, hotter, often more accurate for scoped queries, since long-range links no longer bridge unrelated neighborhoods. Attribute sharding quietly buys more recall per unit of memory.

**IVFFlat and the write path.** pgvector-style IVFFlat splits vectors into `nlist` inverted lists via k-means; a query probes `nprobe` lists. Rules of thumb: `nlist ≈ √n` (roughly 1,000 lists per million vectors), with `nprobe` as the latency/recall dial. Distributed IVFFlat inherits every single-node hazard and adds one of its own: **staleness under rebalancing.** IVFFlat is batch-built, and new vectors land in the nearest *existing* cluster center, which drifts as the corpus grows. Rebalancing moves vectors between nodes whose centroids were trained on different slices — and recall collapses silently. Budget for periodic per-shard rebuilds, or move to HNSW once write volume makes batch rebuilds untenable.

Two rules hold regardless of index type: hybrid metadata filters change the calculus (a filtered scan over a partitioned shard often beats a graph search over a giant unfiltered index), and quantization (SQ/PQ) shrinks vectors 4–32× so each node holds far more in RAM — fewer shards, less fan-out, simpler merges.

## 4. Scaling: The Query-Routing Layer for Sharded Vector Stores

Once vectors span nodes, retrieval needs a brain: a **query router** between the agent runtime and the shards that decides where each query goes, fans out where required, merges results, and presents one stable interface so sharding stays an implementation detail.

Its first job is *classification*. Does this tool call carry a partition key — tenant ID, collection filter? Resolve it to its shard and issue a single-node query: no fan-out, no merge, local-search latency. No filter? Switch to scatter-gather across every shard with the configured oversampling, merge partial results by distance, apply post-filters and final top-k. Dual-mode routing is the entire point: it runs attribute-partitioned and hash-distributed shards behind one API and serves each query with the strategy that matches its shape.

Its second job is *merge correctness*. Merge is a simple k-way heap ordered by distance only when every shard speaks the same metric in the same space. Mix index types, quantization, or per-shard normalization and you need a global re-rank over the merged candidates — never let shard-level score calibration leak into final ordering.

Its third job is *placement and consistency*. The router owns the shard map — which keys live where, replication state. During rebalancing it should dual-read (query old and new placements, merge, reconcile), and it routes upserts and deletes with the same key-hash logic as reads, so an update is visible to the agent’s next retrieval call. Agentic workloads are brutal on consistency: a planning loop that writes a memory and immediately retrieves it exposes every replication lag and cache staleness bug. For single-writer-per-key workloads, pin reads and writes to one shard; for global corpora, version vectors and reconcile at merge. The router is also the enforcement point for retrieval budgets — cap shard touches and candidate volume per request, tenant, and agent loop, because retries turn fan-out into multiplied cost.

## 5. Multi-Tenant Isolation: Tenant-Scoped Retrieval in Shared Clusters

Every serious sharded deployment is multi-tenant, and isolation is absolute: **tenant A’s retrieval must never observe tenant B’s vectors, and tenant B’s query storm must never degrade tenant A’s latency.** Single-node systems enforce this with metadata filters. Sharded clusters must enforce it at three layers or it leaks.

**Placement.** The strongest isolation is physical — dedicated shards or nodes for high-value tenants. That is metadata partitioning at its purest: the right call for regulated tenants, massive corpora, or contractual SLAs. The cost is utilization, so tier it — shared shards for the long tail, dedicated shards for the top few tenants, with a policy that lets tenants graduate between tiers as volume changes.

**Index and filter enforcement.** For tenants sharing a shard, isolation must be structural: **the partition key belongs in the physical index layout, not in an after-the-fact filter.** If the ANN index ignores tenant ID, a top-k search can return tenant B’s vectors before any filter runs — and with HNSW, post-filtering does not restore recall, because the graph paths to tenant A’s true neighbors were never followed. In order of preference: a separate per-tenant index on shared nodes (each small and hot); a single index where tenant ID is a hard key prefix and queries are constrained to the tenant’s subspace; or filtered ANN where the pre-filter actually bounds graph traversal. Whatever you choose, your eval must include an adversarial cross-tenant test — tenant A’s query, tenant A’s filter, scored against tenant A’s ground truth, in a cluster where tenant B’s corpus is orders of magnitude larger. If recall drops as the neighbor grows, your isolation is leaking through the index.

**Noisy-neighbor defense.** Shared clusters fail as systems before they fail as privacy boundaries. One tenant’s batch re-embedding job can saturate a shard, evict another tenant’s graph from cache, and blow every latency SLA on the node. Defense in depth: per-tenant rate limits and concurrency caps at the router; per-shard CPU, memory, and I/O governance; priority classes that put interactive agent retrieval ahead of batch ingestion; and *separate read and write paths* — rebuilds run on shadow resources while the query graph stays untouched, with an atomic swap on completion. The tenant re-embedding a million documents at midnight should see a slower ingest queue, never a degraded query path. Every decision also needs an audit trail — per-tenant query logs, per-shard saturation, a “who touched what” record — and in agentic systems this is doubly true: the retrieval tool is invoked by a model, and “the model did it” is not an isolation story your security team will accept.

## 6. Future-Proofing: From Billion-Scale to Trillion-Scale Retrieval

One billion to one trillion vectors is not a factor of a thousand in hardware — it is a change of regime. A trillion 128-dimensional float32 vectors is 512 TB raw; an HNSW graph over them needs petabytes of RAM at single-node densities. Trillion-scale retrieval forces three shifts, and all three should shape decisions you make today.

**Memory ceases to be the default.** The billion-scale assumption — “keep the graph in RAM” — inverts. Trillion-scale systems are disk-native with memory as a tier: compressed vectors (PQ with 8–16-bit codes) generate candidates; raw vectors are reserved for the final re-rank; the index is structured so only a fraction is resident. This is the DiskANN lineage — a navigable graph over compressed neighbor lists on SSD with a hot head cached in RAM. The lesson for today’s billion-scale builders: **design your shard layout, quantization, and re-rank pipeline as if the working set will not fit in memory** — retrofitting disk tiering is brutally expensive.

**The graph stops being global.** HNSW assumes one navigable graph over the corpus. At trillion scale you get *hierarchical, partitioned* search: a coarse level of learned cluster centroids partitions the space, a region-level graph navigates between regions, per-region graphs finish the hop — IVF at the top, HNSW at the bottom, and a router deciding how deep each query descends. Planners that treat region selection as a learned decision rather than a fixed `nprobe` are where the recall-per-dollar wins will live. Today that means choosing ANN libraries whose index format supports multi-level, disk-resident layouts, and standardizing the vector-store interface so the backend is replaceable when the regime shifts.

**Retrieval becomes a pipeline stage, not the product.** At trillion scale you will not ship raw ANN output to an agent; you will ship the result of a staged pipeline — coarse candidate generation over compressed vectors, pruning by metadata and tenant policy, cross-encoder or LLM re-ranking, deduplication, citation assembly. The vector index becomes the recall engine of a larger retrieval system, exactly as it already is inside agent loops. Architect the interface with that shape — a search tool returning scored, filtered candidates with provenance — and the trillion-scale migration becomes a backend swap, not a re-plumbing of your agents.

The strategic constant across every regime: **evals, routing, and tenant isolation survive all three shifts; index internals do not.** Build those three layers as if scale is inevitable — because for agentic retrieval, it is.

—

## Conclusion: Architect for the Shard Before You Need It

The single-node vector store fails predictably: latency climbs, recall drifts, and the agent loop becomes the hostage of its slowest dependency. Sharding is the answer — but only when you shard on the right key, tune the index for the distributed regime, route every query through a layer that knows a scoped lookup from a global search, and enforce tenant isolation in the index itself rather than in the filter clause. Systems that survive billion-scale agentic traffic are those where sharding was an architecture, not an incident response.

At OnlineCloud, we design and ship evaluated, observable retrieval systems for production agent workloads — sharded vector architectures, MCP-native tool surfaces, and the eval harnesses that prove recall and isolation before they touch real users. If you are hitting the wall on a single-node vector store, or planning the jump past a billion vectors, [contact the OnlineCloud team](https://www.onlinecloud.dev/) — our senior engineers will map your corpus, query shapes, and tenant boundaries, and tell you the sharding strategy that survives production contact. Free initial consultation, response within 24 hours.

# Fine-Tuning Llama 3 for Tool Selection Accuracy: A Comparative Accuracy Study


## 1. Introduction: Why Prompting Frontier Models Hits a Tool-Call Ceiling

There is a moment every agentic-AI team hits around week six. The orchestration loop is solid, the tool schemas are typed, the eval harness is green — and the model still calls the wrong tool. It reaches for the Oracle `ORDER_UPDATE` endpoint when the user asked for an invoice status. It formats a Salesforce `Lead` insert with `Account` fields. The system prompt is a work of art, few-shot examples have been curated to death, and accuracy still plateaus between 70% and 85% on the trajectory-level metric that predicts production behavior.

This plateau is structural, not a prompt-engineering failure. Frontier models like GPT-4o and Claude 3.5 Sonnet are trained on web-scale general function calling, not on *your* API dialect. They are excellent at guessing what a plausible tool call looks like; they are mediocre at remembering that one Oracle endpoint wants a SOAP envelope while another expects a REST payload. Every token of prompt spent disambiguating your API is budget the model cannot spend reasoning about user intent — and once your surface holds 40, 60, or 200 tools, the few-shot prompt becomes the bottleneck before the model does.

At OnlineCloud, we kept hitting this ceiling building MCP-connected agents for Oracle and Salesforce estates, so we ran a controlled study: instead of fighting the prompt, we fine-tuned an open-weights Llama 3 model on golden tool-call trajectories and benchmarked it head-to-head against GPT-4o and Claude 3.5 Sonnet on the same held-out eval set. This post is the full write-up: methodology, dataset curation, the accuracy numbers, deployment on Bedrock and EKS, and an honest ROI read on when fine-tuning pays for itself.

## 2. Fine-Tuning Methodology: SFT, DPO, and LoRA for API Dialects

Fine-tuning for tool selection is not chat fine-tuning. Chat tuning optimizes for fluent text; tool-selection tuning optimizes one narrow decision — given a conversation state, which tool from a fixed registry, with which arguments, in which serialization? The output grammar is small; the decision boundary is everything.

**Stage one: SFT.** Supervised fine-tuning teaches the skeleton. For a surface of ~60 Oracle and Salesforce operations, we generated roughly 12,000 SFT examples teaching exact tool names, argument casing (`recordTypeId`, not `record_type_id`), required-field sets, and serialization conventions. SFT alone takes a model from “plausible but often wrong” to “mostly right on seen shapes.” What it does not fix is *discrimination* — preferring the correct tool when two candidates are both syntactically valid. That is a preference problem, so we add a second stage.

**Stage two: DPO.** Rather than run full RLHF — reward models, PPO rollout farms, instability — we used Direct Preference Optimization, which reuses the SFT model as its reference and optimizes the policy against paired preferred/rejected trajectories. Our pairs came from production logs: the same request answered by a trajectory that ended in successful tool execution (preferred) versus one that selected the wrong operation or hallucinated arguments (rejected). DPO sharpened exactly the axis SFT leaves blunt, contributing roughly half the total accuracy gain over the base model at a fraction of RLHF’s cost.

**Stage three: LoRA.** Full fine-tuning of a 70B model is a capital project. Low-Rank Adaptation freezes the base weights and trains small adapter matrices (we used rank 16 on attention, rank 32 on MLP, alpha at 2× rank). Our adapter for the 8B model was ~160 MB — a few percent of the base weights — and trained on a single A100-class instance in under 24 hours. Because the base model stays intact, one deployment can host multiple dialect adapters behind a router and swap them per request. That adapter multiplexing drives most of the ROI math in Section 6.

## 3. Dataset Curation: Generating Golden Tool-Call Trajectories for Oracle and Salesforce

Fine-tuning quality is dataset quality, and tool-call datasets are the hardest kind to curate because ground truth is a *procedure*, not a string. We built ours in four layers.

**Schema-derived scaffolding.** For each Oracle EBS and Salesforce operation, we generated synthetic user intents from the operation’s own metadata — description, parameters, typical calling context — yielding thousands of cheap, unambiguous examples. Scaffolding establishes coverage; it does not establish realism.

**Production-log distillation.** The highest-value data came from logged production traffic on existing MCP servers: real utterances, attempted tools, argument values, and execution outcomes. We kept only trajectories that ended in verified success and deduplicated aggressively — 60% of requests map to roughly 15% of the operation space, and without dedup the model overfits the popular path.

**LLM-assisted labeling with verification.** We used frontier models to propose trajectories for the hard tail — ambiguous requests, cross-system intents like “sync this account’s new orders from Oracle into Salesforce” — then verified every proposal against the target APIs’ OpenAPI/WSDL schemas and, for anything with a side effect, an execution check in a sandbox tenant. Proposals that failed were rejected, not repaired. Labels were canonicalized into one strict grammar per dialect, so the model never had to learn two serializations for one decision.

**Adversarial negative mining.** We deliberately constructed the base model’s failure cases — requests where an entity like “order” maps to a different operation in context, requests naming a field from the wrong object, cross-dialect confusions — and used them as DPO rejected pairs. The final corpus: ~12,000 SFT examples and ~4,000 preference pairs, split 90/10 with the eval stratified so no tool appears only in training. Every number in Section 4 comes from that held-out set.

## 4. Accuracy Benchmarks: Llama 3 (Fine-Tuned) vs. GPT-4o vs. Claude 3.5 Sonnet

We benchmarked four configurations on 800 held-out real-world-derived requests (400 Oracle, 400 Salesforce), scored at the trajectory level: a call is correct only if the *operation*, *arguments*, and *serialization* all match the golden trajectory. Models: GPT-4o and Claude 3.5 Sonnet via hosted APIs (zero-shot tool-use prompting), base Llama 3 8B (zero-shot), and our SFT+DPO LoRA-tuned Llama 3 8B. Temperature 0, five runs per request, majority vote on disagreements.

| Configuration | Oracle tool-select | Salesforce tool-select | Argument accuracy | Composite trajectory |
|—|—|—|—|—|
| Llama 3 8B (base, zero-shot) | 61.4% | 64.2% | 58.7% | 56.9% |
| GPT-4o (zero-shot) | 82.1% | 84.6% | 81.3% | 78.4% |
| Claude 3.5 Sonnet (zero-shot) | 84.8% | 86.9% | 83.5% | 80.2% |
| **Llama 3 8B (SFT + DPO LoRA)** | **93.6%** | **94.8%** | **91.2%** | **89.7%** |

Three findings matter more than the headlines. **First, the tuned 8B beats both frontier models on its own dialect by roughly nine to eleven points on composite accuracy.** That is the expected result of training on the actual distribution: the frontier models are generalists; our adapter is a specialist in one narrow decision. This is not a claim that fine-tuned Llama 3 is smarter — it is a claim that it is *more calibrated to this tool surface*.

**Second, the gap widens exactly where prompting hurts most.** On the adversarial subset, GPT-4o fell to 71.2% and Claude 3.5 Sonnet to 73.8%, while the tuned model held 88.4%. Few-shotting closed maybe two points of the frontier gap; it never closed the structural one. Moving from 20 tools to the full 60-tool registry cost the frontier models three to four points and the tuned model less than one — prompting degrades with surface size; fine-tuning does not.

**Third, the base-model delta is the entire story.** Base Llama 3 8B at 56.9% is unusable here; the tuned model at 89.7% is production-grade. The whole gain — 33 points — came from ~160 MB of adapters and a training run measured in hours, at roughly 3–5× lower per-request cost than the hosted frontier models. We ran the same pipeline on Llama 3 70B; it buys one to two points for 8× the inference cost. The 8B adapter is the economic optimum.

## 5. Implementation: Deploying LoRA Adapters on AWS Bedrock or EKS

An adapter that lives in a notebook is a demo. Shipping it means putting it behind the same infrastructure as the rest of the agent — typed tool surfaces, auth boundaries, observability, rollback. We deploy on two AWS paths, and the choice is usually dictated by existing platform posture.

**Option A: AWS Bedrock (managed, minimal ops).** Bedrock supports importing custom models: the LoRA adapter is merged with the base weights at export, uploaded to S3, and registered through the API. You get managed inference with IAM access, CloudWatch logging, and auto scaling — no GPU fleet to babysit. The trade-offs: managed-inference margins, Bedrock’s import cadence (slower adapter iteration than a hot-swap), and no multi-adapter multiplexing behind one endpoint. One dialect, one region — Bedrock is the right call.

**Option B: EKS with vLLM (self-hosted, adapter-native).** For multi-dialect estates — Oracle in production, Salesforce in the same control plane, MS SQL next quarter — we deploy on EKS inside the client’s VPC. vLLM natively serves LoRA adapters, so one base-model deployment exposes multiple adapter endpoints and swaps them per request with near-zero overhead. The stack is standard: an inference Deployment behind a service, an MCP server as the tool-orchestration layer in front, OIDC-auth’d ingress, and a HorizontalPodAutoscaler on GPU utilization. Oracle traffic routes to its adapter; Salesforce traffic to its own; the base weights are shared, so memory is dominated by one model plus small adapter slabs.

Our rule of thumb: **Bedrock if you have one dialect and want someone else to own the GPU; EKS + vLLM if you have multiple adapters, regulated data that must stay in your VPC, or a retraining cadence measured in weeks.** Either way, three non-negotiables apply. The endpoint sits behind the same MCP tool layer as any hosted model, so swapping GPT-4o for the adapter is a routing change, not a rewrite. Every tool call is logged as a structured event for eval replay. And the previous adapter version stays live for one release cycle, so a regression is a routing rollback, not a redeploy. Instrument production accuracy the same way you ran the eval — sample traffic, replay against the golden set, and page on drift when the Oracle or Salesforce schemas change.

## 6. ROI: Performance Gains vs. Training and Inference Cost Trade-Offs

The honest question is never “does fine-tuning work?” — Section 4 answers that. It is whether fine-tuning pays for itself on *your* volume, *your* surface, and *your* accuracy requirement — a function of four numbers.

**Training cost is a rounding error.** Our SFT + DPO LoRA run on Llama 3 8B took roughly 12 GPU-hours on an A100-class instance, eval and ablations included — a few hundred dollars at current rates, and the low thousands even with a 70B experiment and a second dialect. That is less than one week of GPT-4o spend for a moderately busy agent, and it is capital cost amortized over every subsequent request. The real constraint is dataset curation labor, not compute.

**Inference cost is where the trade-off lives.** The tuned 8B runs at roughly 20–30% of GPT-4o’s per-token cost and lower still than Claude 3.5 Sonnet. But self-hosting has a floor: a GPU pod costs the same at one request a minute or a thousand. Below roughly 5,000–10,000 tool-call requests per day, hosted frontier models are often cheaper — you pay for consumed tokens, not reserved silicon. Above that volume the crossover favors the self-hosted adapter, and the gap widens linearly. The tuned model also removes the giant few-shot system prompt from every request and eliminates prompting’s worst failure mode: silent accuracy cliffs when the prompt drifts or a tool is added.

**The accuracy-point valuation is the number most teams skip.** A mis-selected tool costs one of three things: a failed API call retried (cheap), a wrong side-effect manually reversed in Oracle or Salesforce (expensive — the reason sandbox verification and human checkpoints before irreversible actions matter), or a silent wrong answer that reaches the user (most expensive). At 8,000 requests per day, moving from 80% to 90% trajectory accuracy is 800 fewer failures daily. If even 10% are wrong side-effects costing fifteen minutes of admin time to reverse, the tuned model pays for its entire training run before lunch. That is why we write the eval before the engagement: baseline accuracy and failure cost are both knowable in week one.

**When NOT to fine-tune.** Three conditions push us back toward prompting: low volume (a few thousand requests daily or less, where hosted models win at the margin); a tool surface still changing weekly (adapter retraining churns); and accuracy requirements a router already satisfies — if 85% of traffic is three well-handled tools, a small classifier routing those to a cheap prompt path beats fine-tuning on ROI. Fine-tuning is for the workload that is high-volume, stable-shaped, and where a wrong call has real cost. That matches most production Oracle and Salesforce agents we see — which is exactly why the prompting ceiling, not model capability, is what our clients keep hitting.

## Conclusion: When Prompting Ceils, Fine-Tune the Decision

Frontier models will keep improving at general tool use, and your API dialect will keep getting more specific. That asymmetry is permanent — which is why a small, fine-tuned specialist that knows your Oracle envelope and your Salesforce object model cold is not a workaround; it is the correct architecture for a stable, high-volume tool surface. In our study, a ~160 MB LoRA adapter on Llama 3 8B outperformed GPT-4o by 11 points and Claude 3.5 Sonnet by 9 points on composite trajectory accuracy, for a training cost in the hundreds of dollars and inference at 8B pricing. The bottleneck was never the model — it was the dataset, and dataset curation is an engineering problem with a known solution.

If your agent team is staring at a tool-selection plateau — Oracle, Salesforce, MS SQL, or a custom surface — bring us the eval and we will tell you in one discovery week whether fine-tuning earns its keep. OnlineCloud builds the MCP tool layer, curates the golden trajectories, runs the SFT + DPO + LoRA pipeline, and deploys the adapter on Bedrock or in your VPC on EKS — with the accuracy dashboards to prove it. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) and let’s measure your baseline: prompting where it works, fine-tuning where it doesn’t, and an honest cost model for the gap in between.

# Implementing Rate Limiting and Cost Ceilings in MCP Server Middleware


> Platform Engineer’s field notes on the middleware that keeps an MCP server from spending next quarter’s AI budget before lunch.

## 1. Introduction: The $1,000 Recursive Loop — Why Governance Must Be Architectural

Every MCP team has the same origin story. The agent calls a tool, the tool returns an error, the agent retries with different arguments, the tool errors again — and because each attempt *looks* like progress to the model, the loop never self-terminates. Meanwhile every iteration appends its own output to an ever-growing context window, and per-token pricing quietly multiplies: forty steps of a “three-step task,” each one reading an escalating transcript, adds up to a conversation that bills like a small data center. No malice, no prompt injection, no rogue employee. Just one tool with ambiguous error semantics and no ceiling above it.

That is the $1,000 recursive loop, and the reason it keeps happening is that most teams treat cost as a *reporting* problem. They discover it in the provider’s billing export. Governance that arrives after the invoice is not governance — it is archaeology.

The fix is architectural: put the limits in the path that every model interaction already crosses. In a Model Context Protocol (MCP) deployment, that path is the server. Every tool call — from Claude, GPT-4-class models, or an open-weights deployment in your own VPC — arrives as a JSON-RPC `tools/call` over a streamable HTTP or SSE transport and lands in one dispatch layer. If rate limits, budget ceilings, and audit live in that layer as middleware, then *no client can route around them* — the middleware does not ask nicely. That seam is why we treat MCP as a governance boundary, not another API gateway: done badly, an MCP server is just a slower way to call your database; done well, it is the only place agents cross into your business logic, and it enforces the rules there.

This post is the playbook we use at OnlineCloud when we wire MCP servers to real systems — Oracle, MS SQL, Postgres, CRMs — with real budgets attached. Three layers: **token-weighted rate limiting** so no tenant can monopolize the pipeline, **hard cost ceilings** enforced before a single token is spent, and **observability** that turns spend and throughput into two lines on a live dashboard instead of a surprise in a billing export.

## 2. Token-Based Rate Limiting: Leaky Bucket and Fixed Window per Tenant

Request-level rate limits are the first thing everyone reaches for, and the first thing that fails. Ten requests per second tells you nothing about load when one request asks the model to reason over a 40,000-token context and the next is a two-line classification. Agents amplify the skew: a single planning loop can fire dozens of tool calls in a burst, each carrying a different payload weight. **Rate limiting for agentic traffic has to be denominated in tokens, not requests.**

That means counting what each tool call will plausibly consume before admission — an estimate derived from the arguments it carries and the model class it targets — and charging the tenant’s bucket by that weight. That way, flooding the pipeline with tiny calls and running amok on huge context become the same accounting problem.

Two algorithms dominate for per-tenant control, and they answer different questions:

**Fixed window** is brutally simple: a counter per tenant per window (say, 60 seconds). One atomic increment against a TTL’d Redis key — but with a well-known seam: a tenant that exhausts the window at second 59 can immediately spend a fresh window at second 60, doubling its limit in adjacent bursts.

**Leaky bucket** smooths that out. Tokens drip out of the bucket at a constant refill rate, and a call that would overflow the bucket is rejected (or queued) regardless of where it lands in the clock. Bursts are bounded by bucket capacity; sustained load is bounded by the drip rate. For agentic workloads the leaky bucket is usually the right default: agent loops produce *clusters* of calls, and you want each cluster bounded without penalizing steady background traffic.

A per-tenant leaky bucket in Node.js is small enough to inline:

“`ts
type TenantBucket = { tokens: number; lastRefill: number };

class TokenLeakyBucket {
constructor(
private capacity: number, // max burst, in tokens
private refillPerSec: number, // steady-state drip
) {}

tryAcquire(tenant: string, weight: number, now = Date.now()): boolean {
const b = (this.buckets.get(tenant) ??= {
tokens: this.capacity,
lastRefill: now,
});
// Drip: refill is proportional to elapsed time.
b.tokens = Math.min(
this.capacity,
b.tokens + ((now – b.lastRefill) / 1000) * this.refillPerSec,
);
b.lastRefill = now;
if (b.tokens < weight) return false; // 429 → retry after b.tokens -= weight; // atomic in Redis return true; } private buckets = new Map();
}
“`

In a multi-instance deployment, the same logic moves into a Redis `EVAL` script so the bucket is shared across replicas; the algorithm is identical, only the store changes. What *must not* change is that the token weight is computed in one place — the governance layer — so a tool that fans out a sub-agent or streams a large document gets charged its true weight, not the weight of a cheap request.

| Algorithm | Burst behavior | Smooths load? | Implementation cost | Best for |
|—|—|—|—|—|
| Fixed window | Boundary bursts (2× limit) | No | Trivial (INCR + TTL) | Coarse quotas, alerting thresholds |
| Leaky bucket | Bounded by capacity | Yes | Simple (refill + subtract) | Agentic, bursty per-tenant traffic |
| Token bucket | Bounded, allows credit accumulation | Partial | Simple | APIs with periodic batch jobs |

Whichever you pick, per-tenant state must be keyed by an authenticated tenant identity — never by IP or connection — because an agent session can hop transports and reconnect. And rejections must return a *structured, retryable* error the model can actually handle: an MCP error result with code and a `Retry-After`-style hint, so the agent backs off instead of looping harder. Rate limiting that fails closed with an unparseable error just moves the recursion from your bill to your error logs.

## 3. Cost Ceilings: Real-Time Token Counting and Hard Budgets at the Protocol Layer

Rate limiting answers “how fast may this tenant go?” Cost ceilings answer a different question: “how much may this tenant *cost* before someone human gets involved?” A well-behaved tenant at modest throughput can still burn a shocking daily bill if it is running a wide agent over a long conversation — no individual request was excessive, so throughput limits never see the problem.

The enforcement unit is the token, and the discipline is **check before you spend, not after.** Counting happens in the request path, where the middleware knows three numbers for every call: input tokens, output tokens (or a strict cap), and the price per token for the model actually serving the tenant. For the input side you can count deterministically — arguments are fully known before dispatch — using a tokenizer matched to the model family (the `cl100k_base` / `o200k_base` BPE encodings for the GPT/Claude-class models). For the output side you cannot know the future, so you enforce a *bound*: a per-call `max_tokens` ceiling derived from the tenant’s remaining budget. Every call is then:

1. **Pre-check** — estimate input tokens, take the bounded output allowance, and ask the tenant ledger: *does the remaining budget cover this step?* If not, refuse with a structured error and an escalation path. The agent cannot nickel-and-dime its way past the ceiling, because every call is checked against what *remains*.
2. **Execute** — the call runs with a hard output cap.
3. **Post-charge** — actual tokens observed on the response are debited from the ledger. In streaming mode, charge incrementally: accumulate tokens from the stream as they arrive and abort the stream the moment the step budget is exhausted, so a runaway generation dies mid-stream rather than completing and billing.

The ledger is a per-tenant, per-period budget — `{ tenant, period, model, spentTokens }` in Redis with a daily rollover — and the middleware treats it like a resource limit on a container. Memory limits do not ask the process nicely to stop allocating; the kernel kills it. Your cost ceiling should have the same temperament.

“`ts
interface BudgetGate {
remaining(tenant: string): number; // dollars, this period
reserve(tenant: string, estCost: number): boolean; // pre-check
settle(tenant: string, actualTokens: number): void; // post-charge
}

const budgetGate: BudgetGate = {
reserve(tenant, estCost) {
if (ledger.remaining(tenant) < estCost) { // Hard stop: structured MCP error → agent escalates to a human. throw mcpError("BudgetExceeded", `Tenant ${tenant} has ${ledger.remaining(tenant).toFixed(2)} left ` + `for this estimated ${estCost.toFixed(2)} call`); } ledger.debit(tenant, estCost); // optimistic hold return true; }, settle(tenant, actualCost) { ledger.adjust(tenant, actualCost); // reconcile hold vs. reality }, }; ``` Two properties make this a *hard* ceiling rather than an aspiration. First, the pre-check runs at the protocol layer, in the middleware — not in the tool implementation, and certainly not in the prompt. No tool author can forget to check the budget, because they never see the request until the gate has passed it. Second, ceilings compose: per-call, per-run (an agent loop gets a max-iteration cap and a dollar cap), and per-tenant daily caps. A loop that hits the run cap terminates *gracefully* — a structured handoff to a human checkpoint — not an agent retrying a refund call against an empty ledger. ## 4. Middleware Design: Wrapping MCP Tool Calls with Governance Logic MCP gives you a clean place to stand: the server owns the tool registry, and every client — any agent framework, any host — reaches business logic only through `tools/call`. Wrap that dispatch in a middleware chain and governance becomes a property of the *server*, not of the clients. One chain, in order: **authenticate → rate-limit → budget-gate → audit → invoke → settle**. With the TypeScript MCP SDK the pattern is a higher-order function around tool registration: ```ts import { Server } from "@modelcontextprotocol/sdk/server/index.js"; type GuardedTool = { name: string; costClass: "cheap" | "standard" | "expensive"; // → token weight + price run: (args: unknown, ctx: CallContext) => Promise;
};

export function registerGuarded(server: Server, tool: GuardedTool) {
server.registerTool(tool.name, {
description: tool.description,
inputSchema: tool.inputSchema,
execute: async (args, extra) => {
const tenant = resolveTenant(extra.session); // auth first
rateLimiter.tryAcquire(tenant, weight(tool, args)); // tokens/sec
const reserved = budgetGate.reserve(tenant, estCost(tool, args));
const started = now();
audit.log({ tenant, tool: tool.name, phase: “start”, args }); // decision log
try {
const result = await tool.run(args, { tenant });
const cost = settleFromUsage(result, tool.costClass);
budgetGate.settle(tenant, cost); // debit actual
audit.log({ tenant, tool: tool.name, phase: “done”, cost });
emitMetrics(tenant, tool, cost, now() – started); // → dashboards
return result;
} catch (err) {
audit.log({ tenant, tool: tool.name, phase: “error”, err });
throw err;
}
},
});
}
“`

The same shape ports to Go (`github.com/mark3labs/mcp-go` or the official SDK): a wrapper that takes a tool handler and returns a guarded one, with the chain expressed as composable middleware structs. What matters is not the language but the invariants. **Every tool gets the same chain** — no tool can declare itself exempt, because exemption would just become the bug that produces the incident. **Weights and prices are declared per tool** (`costClass`) so policy lives with the schema, not scattered across handlers. And **every refusal and every charge is an auditable event** with a trace ID, so when finance asks why tenant X was throttled at 14:02, the answer is a query, not a meeting.

Keep the middleware *fast and pure*: state in Redis (shared across instances), synchronous decisions, single-digit milliseconds added to a model round-trip that costs hundreds. One chain, one seam, one place where the rules are enforced and audited.

## 5. Observability: Visualizing Spend vs. Throughput in Real Time

Governance without evidence is a policy document. The third layer makes the first two *visible*: a live view that plots the two variables every platform engineer cares about — **spend** (dollars per minute, per tenant, per tool) against **throughput** (tokens and calls per minute) — with budget ceilings drawn as horizontal lines that you can watch the burn approach.

The metrics come straight out of the middleware in Section 4. Each guarded call emits one event on completion (and one on refusal) with the fields that make the dashboard meaningful:

“`json
{
“ts”: “2026-09-06T14:02:31.004Z”,
“tenant”: “acme-corp”,
“tool”: “orders.lookup”,
“decision”: “allowed”,
“tokens_in”: 1843,
“tokens_out”: 512,
“cost_usd”: 0.0211,
“latency_ms”: 844,
“budget_remaining_usd”: 142.08,
“trace_id”: “8f3a1c…”
}
“`

Feed those events through OpenTelemetry into a Prometheus/ClickHouse-plus-Grafana stack (or any metrics backend you already run), and build four panels:

| Panel | Series | What it tells you |
|—|—|—|
| Spend vs. throughput | $/min (area), tokens/min (line), per-tenant | Are we getting busier or just more expensive? |
| Ceiling burn | Cumulative $ per tenant + ceiling line | Which tenant is sprinting at their budget? |
| Refusals by reason | 429s (rate) vs. budget refusals | Rate refusals → sizing issue; budget refusals → ceiling tuning |
| Step latency & size | p50/p95 latency, tokens/call distribution | Context bloat and slow tools, before they become bills |

The dashboard’s real job is to make *shapes* recognizable. A tenant whose spend line goes vertical while throughput stays flat is running a recursive loop with growing context — the $1,000 story happening live, in time to stop it. A sawtooth of budget refusals every morning means your ceiling is tuned to yesterday’s traffic. Alert on the shapes: spend rate crossing an anomaly band, any tenant above 80% of ceiling, refusal-rate spikes. And keep the panel honest with a **cost-per-outcome** view — dollars per resolved ticket, per order lookup, per deflection — because it is the number the business renews on — and the line that justifies the whole middleware layer in a quarterly review.

## 6. The OnlineCloud Standard: Production-Grade Governance in 6–10 Weeks

None of this is exotic, but all of it is *exacting* — and the difference between a demo and a deployment is whether the middleware exists before the first production tenant connects. At OnlineCloud we ship MCP servers with rate limiting, budget gates, and dashboards as part of the baseline architecture, on a 6–10 week delivery model:

– **Week 1 — discovery.** Map the data systems and auth boundaries, pick the model class per workload, and write the cost model: token weights per tool, prices per model, tenant tiers. If you cannot price the system in week one, you cannot ceiling it in week nine.
– **Weeks 2–7 — vertical slices.** Ship one usable, instrumented slice every 1–2 weeks against real data — middleware chain and dashboard panels included from day one. Governance is not a phase; it ships with the first tool.
– **Weeks 8–10 — harden and hand over.** Load-test the failure paths: a simulated recursive loop dies at the ceiling and pages someone; an over-limit tenant gets clean retryable errors. Hand over the runbooks, alert tuning, and on-call.

That last part is the OnlineCloud difference: the engineers who scope the system push the commits and own the dashboards after launch. We instrument outcomes — accuracy, latency, dollars per request — and share live numbers rather than a slide deck. Evaluated, observable, cost-bounded systems are the deliverable; the middleware is how we keep the promise.

## Conclusion: Make the Ceiling the Architecture

Rate limiting, cost ceilings, and observability are one stance applied three times: *an agentic system should not be able to surprise you.* The token-weighted bucket bounds how fast any tenant can move; the protocol-layer budget gate bounds how much any tenant can spend; the dashboard makes both visible before they become incidents. None of it depends on the model behaving — models will not always behave. Recursive loops, context bloat, and runaway spend are not bugs you can prompt your way out of; they are resource problems that need resource enforcement in the one seam every interaction crosses: MCP server middleware.

If you are standing up MCP against real systems — Oracle, MS SQL, Postgres, Snowflake, your CRM — with real budgets and real tenants, the OnlineCloud engineers build this in from week one: typed tools, auditable transports, per-tenant rate limits and cost ceilings, and dashboards your operations team can sleep through. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) to start the conversation — and bring the workload you are tired of babysitting.

# State Management in Long-Running Agents: Redis vs. Postgres for Session Persistence


Every agent demo works. Every agent in production dies a death of a thousand state bugs. The gap between the two is how you answer one question: *where does the session live when the process that owns it doesn’t?*

Most teams start with the comfortable lie of statelessness. Your LLM is stateless by construction — prompt in, tokens out — so you build “stateless” orchestrators that treat every invocation as a fresh request and hope the workflow graph survives a crash. It works for 30-second tasks. Then someone asks the agent to reconcile a month of invoices across three systems, the run takes fourteen minutes, and the worker dies at minute eleven. Every tool result, every intermediate plan, every token you already paid for — gone. That’s not an LLM problem. It’s an infrastructure problem, and it’s the one this post is about.

Long-running agents need a session store with two contradictory properties: it must be *fast enough to read and write on every step*, and *durable enough to survive a node dying mid-step*. No single datastore gives you both at the price you want to pay. Redis gives you speed with an asterisk. Postgres gives you durability with a tax. The production answer is almost always both, layered deliberately. Here’s where each engine is strong, where it lies to you, and how to combine them without building a distributed-systems thesis project.

## Redis for Agent State: Low-Latency Context Storage and TTL Management

The active run needs a hot path. Every loop iteration — plan, call tool, observe result, plan again — reads the current context and writes the new one. At a few hundred milliseconds per LLM call that’s not a throughput problem; it’s a *latency* problem, because state I/O sits directly on the critical path of an already-slow loop. You are not going to accept a 20ms Postgres round trip per step when an in-memory GET takes 200 microseconds.

Redis earns its place as the hot store for exactly this reason. A single key per session holding a JSON-serialized context object gives you O(1) reads and writes, sub-millisecond latency in the same region, and — critically for agent workloads — native TTL semantics that map cleanly onto run lifecycle. The shape is boring and that’s the point:

“`
agent:{run_id}:state → { step, status, plan[], tool_results[], context_tokens, updated_at }
agent:{run_id}:lease → { worker_id, heartbeat_at } (expires after ~3 missed heartbeats)
“`

The boring part is where the value is. Keep one key for the state blob, one key for the worker lease, and let Redis garbage-collect both when the run ends or dies. No cron job sweeping orphaned sessions, no `DELETE FROM sessions WHERE updated_at < now() - interval '15 minutes'` grinding through your table. TTL *is* your cleanup story. But TTL is also where engineers shoot themselves. Three failure modes show up in every code review I do: 1. **A single TTL shorter than your longest tool call.** You set 5 minutes on the state key, then the agent fires an MCP tool that runs a 20-minute ETL job. Mid-call, the key expires. The worker comes back, tries to checkpoint, and writes to a key that no longer exists — or worse, the lease expires and a second worker picks up the same run and starts double-executing the job. Rule: the TTL on a run's state must exceed the maximum *step* duration, not the estimated *total* duration, and you refresh it only between steps. 2. **Refreshing TTL on every write defeats idle detection.** A sliding TTL that resets on every heartbeat means a hung-but-alive worker holds its lease forever. Separate the concerns: a short *activity* TTL on a heartbeat key that only the orchestration loop touches, and a longer *state* TTL that only refreshes when a step actually completes. 3. **Assuming eviction won't happen because you never set `maxmemory`.** Default `noeviction` means Redis stops accepting writes when memory fills — checkpoint writes start erroring at 3 a.m. under a memory spike in a way that looks like a network bug. If you run shared Redis, `allkeys-lru` will evict a mid-run session key under pressure and silently orphan an agent. Either way you lose. The fix is isolation: dedicated Redis for session state, with `maxmemory` sized to `(concurrent runs × max context bytes) × 2` as a floor, and `noeviction` so failure is loud instead of silent. Redis is the right tool for the *active* state. What it is not is a system of record. RDB snapshots give you point-in-time recovery with fork-time memory doubling and potential data loss up to your save interval. AOF gives you better durability but reintroduces latency and complexity on every write. And neither gives you queryability — try answering "which sessions touched the Salesforce tool yesterday and what did they return?" from a Redis key dump. That question is an audit question, and audits live in Postgres. ## Postgres for Durable Memory: JSONB for Long-Term Snapshots and Auditability Everything that outlives the run belongs in Postgres. The reasons are boring, which is why they're right: it's already in your stack, it has real durability guarantees, and it can answer questions Redis can't. The standard pattern is a session table with a `jsonb` payload column holding the full serialized agent state, written at coarse checkpoints. ```sql CREATE TABLE agent_sessions ( run_id uuid PRIMARY KEY, agent_type text NOT NULL, status text NOT NULL DEFAULT 'running', -- running | paused | completed | failed payload jsonb NOT NULL, -- full serialized session state token_count integer NOT NULL DEFAULT 0, started_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), last_error text ); CREATE INDEX idx_agent_sessions_type_status ON agent_sessions (agent_type, status, updated_at DESC); CREATE INDEX idx_agent_sessions_payload ON agent_sessions USING gin (payload jsonb_path_ops); ``` The `jsonb` column is the whole trick. Agent state is schema-less by nature — every tool integration adds fields, every agent type carries different context — and fighting that with a relational schema means a migration per feature. `jsonb` stores the state as a single document you can read, write, and atomically replace in one row update, while still letting you index into it with GIN for the occasional structured query ("all failed runs whose payload mentions `verification_pending`"). You lose nothing by treating the row as an opaque blob, and you gain the ability to upgrade your state schema without a migration. That's the persistence story. The auditability story hits three weeks after launch, when compliance asks what an agent did to a customer record: a mutable `payload` column shows the *current* state, not how you got there. The fix is an append-only event companion: ```sql CREATE TABLE session_events ( event_id bigserial PRIMARY KEY, run_id uuid NOT NULL REFERENCES agent_sessions(run_id), seq integer NOT NULL, -- per-run sequence, checked by the writer step integer, event_type text NOT NULL, -- step_started | tool_called | checkpoint | human_approved | failed snapshot jsonb, -- full state snapshot at this event created_at timestamptz NOT NULL DEFAULT now() ); ``` Every checkpoint writes the full snapshot into `session_events`, keyed by a per-run sequence number the writer enforces with a unique constraint — `UNIQUE(run_id, seq)`. Now "what happened" is a replayable log, exactly what you hand an auditor. Postgres also gives you recovery that doesn't depend on a second system: WAL replay means a committed checkpoint survives a crash, a power loss, even a lost Redis cluster. When an EC2 instance dies at 11:59 p.m., the run you thought was in memory is a committed row you can resume at 12:01. The tax, and be honest about it, is write cost and bloat. Every checkpoint is a full-document write; checkpointing after every step of a 40-step run means 40 full snapshots. Keep the `payload` row as your coarse-grained current state (checkpoint every N steps or when accumulated deltas cross a token threshold) and push every-step snapshots to the event log only if you need them. And watch table bloat — frequent updates of large `jsonb` documents leave dead tuples behind, so a modest autovacuum tuning pass is not optional on a system that writes snapshots all day. ## Hybrid Architectures: Redis for Active Runs, Postgres for Durable Recovery Here's the architecture that survives contact with production, and it's embarrassingly simple: - **Redis holds the hot state** for every run that is executing right now — full context, sub-millisecond reads, TTL-managed leases so crashed workers release runs automatically. - **Postgres holds the durable truth** — a checkpoint written asynchronously after each completed step (or every N steps), so the maximum loss on any single failure is bounded by your checkpoint interval, not by the whole run. - **Recovery reads Redis first, Postgres second.** A new worker that picks up a run checks Redis; if the key is alive, it resumes from the hot state. If the key is gone (crash, eviction, TTL expiry), it loads the latest snapshot from Postgres and rebuilds. The subtle part is the checkpoint path — a classic dual-write. You can't atomically write both stores, and if you update Redis then crash before the Postgres write lands, the durable copy is stale. Two mitigations matter: 1. **Version every state blob.** Carry a monotonically increasing `version` (or step counter) inside the serialized state, and make the Postgres upsert conditional: `UPDATE ... SET payload = $2, version = $3 WHERE run_id = $1 AND version < $3`. Stale async writes from a crashed worker can never roll the durable state backward. 2. **Make the checkpoint path a write-ahead, not a write-behind.** Update Postgres *first* (or enqueue to a durable outbox table), then update Redis, then delete the outbox row. Redis is a cache of the durable truth, not the other way around. If you internalize that one sentence, every consistency bug in your agent platform becomes a lot easier to reason about. You also need to decide who owns a run, because "two workers resumed the same session" is the failure mode that corrupts customer data, not just logs. A Redis lease key (`SET agent:{run_id}:lease me NX PX 30000`) with a periodic refresh gives you cheap mutual exclusion; when a worker loses the lease it must checkpoint-and-stop rather than fight. For the resume path, the version number doubles as a guard: the recovering worker only claims the run if its snapshot version matches the durable version in Postgres. Two operational details finish the picture. Don't run checkpoints on the agent's critical path — hand snapshots to a small async writer that batches upserts, so a Postgres blip degrades your *durability window*, not your agent's latency. And note what this buys you: any worker can resume any run, so sessions become a property of the platform, not of a process that might die. ## Performance Comparison: Latency Benchmarks and OOM Risk Analysis Numbers from my own test harness, because vendor benchmarks are theater. Both engines in the same region, TLS on, same VM class; state payload ~8 KB of JSON (a realistic mid-run context). Values are p50/p99 over 10k operations. | Operation | Redis 7 (in-memory) | Postgres 16 (local NVMe) | Postgres 16 (same-AZ) | |---|---|---|---| | Read full session state | 0.4 ms / 1.2 ms | 0.9 ms / 3.1 ms | 1.8 ms / 5.4 ms | | Write full session state | 0.5 ms / 1.4 ms | 1.4 ms / 4.8 ms (upsert + jsonb) | 2.6 ms / 8.9 ms | | 100-step run, checkpoint every step | ~50 ms total state I/O | ~250 ms | ~450 ms | The gap is real but it's *tens of milliseconds per run*, not per step, because you checkpoint once per step, not per LLM token. Against a step that takes 2–10 seconds — model latency dominates — the difference between 0.5 ms and 3 ms of state I/O is noise. Redis wins on the tail: its p99 stays flat as concurrency rises until you hit memory, while Postgres p99 climbs as the buffer cache thrashes. Postgres wins on the floor: lose an 8-minute run to a Redis restart and the milliseconds you "saved" per run become the worst trade in your infrastructure. OOM risk is where the two engines diverge hardest, and where most teams get burned: - **Redis OOM is a design property, not a bug.** Redis keeps everything in RAM, so memory scales linearly with (concurrent runs × context bytes), and two failure modes bite: with `noeviction`, writes start erroring under pressure; with an LRU policy, your mid-run session keys get evicted and agents orphan silently. RDB snapshots temporarily *double* memory during fork. You must size Redis for worst-case concurrent contexts, add memory alerts at 70%, and treat eviction as a correctness bug, not a tuning knob. - **Postgres OOM risk is bounded.** Rows spill to disk; the buffer cache is a cache, not a store, so a 10 GB session table doesn't need 10 GB of RAM. Your real risks are bloat from churning large `jsonb` documents (dead tuples), and checkpoint/IO stalls during WAL flushes under sustained write load. Those are tuning problems with well-known fixes; they are not "the database deleted my data under pressure" problems. The engineering summary: Redis for the hot path because its p99 and its operational model fit active run state; Postgres for the durable path because it survives what Redis cannot; and never let a memory-pressure policy make the durability decision for you. ## Reference Implementation: Session Checkpointing in Go A minimal but production-shaped checkpoint loop. It treats Postgres as the source of truth and Redis as the hot cache — write-ahead to Postgres, then refresh Redis, then delete the outbox row. Versioning makes stale writers harmless. ```go // checkpoint writes durable state first, then refreshes the hot cache. func (w *Worker) checkpoint(ctx context.Context, run *Run, state State) error { // 1. Write-ahead: durable upsert in Postgres, version-guarded so a // stale async writer can never roll state backward. _, err := w.pg.Exec(ctx, ` INSERT INTO agent_sessions (run_id, agent_type, status, payload, token_count, updated_at) VALUES ($1, $2, $3, $4, $5, now()) ON CONFLICT (run_id) DO UPDATE SET payload = EXCLUDED.payload, status = EXCLUDED.status, token_count = EXCLUDED.token_count, updated_at = now() WHERE agent_sessions.token_count < EXCLUDED.token_count`, // monotonic guard run.ID, run.Type, state.Status, state.Payload, state.TokenCount) if err != nil { return fmt.Errorf("durable checkpoint: %w", err) } // 2. Record the audit event (append-only, per-run sequence). if _, err := w.pg.Exec(ctx, ` INSERT INTO session_events (run_id, seq, step, event_type, snapshot) VALUES ($1, $2, $3, 'checkpoint', $4)`, run.ID, state.Seq, state.Step, state.Payload); err != nil { return err } // 3. Refresh the hot state in Redis. TTL = max step duration + grace, // refreshed only between steps, never mid-tool-call. raw, _ := json.Marshal(state) return w.rdb.Set(ctx, "agent:"+run.ID.String()+":state", raw, 30*time.Minute).Err() // 30 min >> longest single tool call
}

// resume loads hot state from Redis, falling back to the durable snapshot.
func (w *Worker) resume(ctx context.Context, runID string) (*State, error) {
raw, err := w.rdb.Get(ctx, “agent:”+runID+”:state”).Bytes()
if err == nil { // hot path: cache hit
var s State
return &s, json.Unmarshal(raw, &s)
}
if !errors.Is(err, redis.Nil) {
return nil, err
}
// cache miss: rebuild from the durable snapshot in Postgres
var s State
err = w.pg.QueryRow(ctx, `
SELECT payload FROM agent_sessions
WHERE run_id = $1 AND status = ‘running’`, runID).
Scan(&s.Payload)
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrNoResumableSession
}
// re-acquire the lease BEFORE doing any work on the resumed run
return &s, w.acquireLease(ctx, runID)
}

func (w *Worker) acquireLease(ctx context.Context, runID string) error {
ok, err := w.rdb.SetNX(ctx, “agent:”+runID+”:lease”, w.id,
45*time.Second).Result() // ~3 missed heartbeats before takeover
if err != nil {
return err
}
if !ok {
return ErrLeaseHeldElsewhere
}
go w.heartbeat(ctx, runID) // refresh lease while the run executes
return nil
}
“`

Wire `checkpoint` into the orchestration loop after every completed step (or every N steps once the context is large), call `resume` at worker startup, and you have crash-safe, resumable agents. The whole pattern is ~60 lines. The hard part was never the code — it was deciding which store owns the truth.

## The Bottom Line

Statelessness is a fiction you can maintain for 30-second tasks and must abandon for anything that runs minutes. Keep the active context in Redis, where sub-millisecond reads and TTL-based leases make the hot path fast and self-cleaning. Keep the durable truth in Postgres, where `jsonb` snapshots and an append-only event log give you recovery, replayability, and answers for the auditor. Layer them so Postgres is written first, Redis is a cache of it, and version numbers make stale writers harmless. Do that, and a worker crash becomes a resume event instead of a lost customer conversation — and you’ll finally sleep through the 3 a.m. page about a node that died mid-run.

—

**About OnlineCloud.** We’re a US-based engineering studio that ships evaluated, observable agentic systems — MCP servers, orchestration loops, and the state layer underneath them — wired into the databases and CRMs your team already runs. No offshore handoffs, no decks: senior engineers, production discipline, and working code in 6–10 weeks. If you’re building long-running agents and want the state management done right the first time, [let’s talk](https://www.onlinecloud.dev/) — or email us at contact@onlinecloud.dev. We answer within 24 hours.

# Securing MCP Resources: OAuth 2.1 and Scoped Token Enforcement


*By the security engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: The Vulnerability of Agentic Data Access

Every agentic system ships with the same silent assumption: *the model will only do what we asked it to do.* That assumption is the vulnerability.

Think about what an MCP (Model Context Protocol) server actually is. It is an authenticated door from an LLM — a stochastic, prompt-injectable, eminently socially-engineerable principal — directly into your resource estate: `customers/{id}/orders`, `finance/invoices`, `hr/records`. In the pre-agentic world, a human clicked through an application, and the application’s UI was the permission boundary. In the agentic world, the model proposes tool calls autonomously, in a loop, often over long-running sessions where a single malicious string inside a fetched document can steer the next invocation. The UI is gone. The boundary must be rebuilt at the protocol layer.

The uncomfortable truth is that **an MCP resource URI is a permission**. `mcp://ledger/orders/4421` is not merely an address; it is a statement of what a caller may read or mutate. When a host connects to an MCP server without authentication, without per-request authorization, and without a notion of *whose* access is being exercised, then every resource the server exposes is available to every agent that reaches it — and every prompt injection becomes a data exfiltration primitive.

This is why the security architecture of MCP cannot be an afterthought bolted onto a working demo. It must answer five questions before the first tool call ships:

1. **Who** is the human ultimately responsible for this action?
2. **What** capability does this specific agent invocation require?
3. **How** does the server verify the caller on every request, not just at connection time?
4. **Where** does identity terminate, so the database can enforce row-level policy?
5. **What** evidence is left behind for auditors and incident responders?

OAuth 2.1 answers the first three. Scoped token enforcement answers the second and the fourth. Audit logging answers the fifth. This post walks through each, and closes with a reference architecture that binds them together inside a VPC-isolated MCP deployment. I write this as a security architect who has spent fifteen-plus years shipping data systems against Oracle, MS SQL Server, Postgres, and Snowflake — because the agentic era does not forgive the mistakes the API era tolerated.

## 2. OAuth 2.1 for MCP: Implementing the Authorization Code Flow with PKCE

OAuth 2.1 is not a new protocol so much as a **consolidation of the security requirements** that the industry spent a decade learning the hard way: PKCE is mandatory for every client, implicit grants are abolished, refresh tokens for public clients are restricted, and every grant must be bound to a client through sender-constrained tokens. For MCP, this matters because the MCP Authorization specification — now part of the broader MCP ecosystem — adopts OAuth 2.1 as its native authorization layer, and the new Streamable HTTP transport is explicitly designed to carry it.

The pattern to implement is the **Authorization Code Flow with PKCE** (Proof Key for Code Exchange). The flow, as it applies to an MCP host:

1. The user opens the agent host and the host discovers the MCP server’s authorization metadata via `/.well-known/oauth-authorization-server`.
2. The host generates a `code_verifier` (a high-entropy random string) and its SHA-256 hash, the `code_challenge`, which it sends with the authorization request to the authorization server. The plaintext verifier never transits the wire.
3. The user authenticates and consents at the authorization server, which returns an authorization code — not a token — to the redirect endpoint.
4. The host exchanges the code for tokens at the token endpoint, presenting the `code_verifier`. The server hashes it and compares against the stored challenge. **This is the PKCE guarantee:** even if an attacker intercepts the authorization code, they cannot redeem it without the verifier, which never left the host.
5. The host receives an access token with a short TTL and a refresh token bound to the client.

Three implementation rules separate a compliant deployment from a decorative one.

**Rule one: PKCE is never optional.** In OAuth 2.1 it is required for public clients, and you should require it for confidential clients too. There is no code path where PKCE is skipped because “this client is trusted.” MCP hosts are frequently desktop or mobile applications — precisely the public-client category where static client secrets are worthless and PKCE is the only credential the host can prove.

**Rule two: tokens are short-lived, and refresh tokens rotate.** Access tokens in an MCP context should live minutes, not hours — long enough for a bounded task, short enough that a leaked token in a prompt-injection artifact is stale before it is useful. Refresh tokens must be single-use: every refresh returns a new refresh token and invalidates the old one, so a stolen refresh token dies at the next rotation. Never issue a refresh token that outlives its session.

**Rule three: every request is authenticated, not every connection.** Streamable HTTP sessions must not smuggle tokens into a connection and then assume the session is authorized. Each tool call carries its token (in practice, a short-lived token refreshed on demand), and the MCP server validates signature, issuer, audience, expiry, and scope **per request**. A connection that was authorized at 9:00 AM is not authorized at 9:00 AM plus one revoked session.

Concretely, the server-side validation on every request should check, in order: signature validity, `iss` and `aud` claims against the configured issuer and the server’s own client ID, `exp`, and then the scope claim against the tool or resource being invoked. Fail any check, return a structured error, and emit an audit event. That final step — logging the rejection — is what turns authentication from a gate into evidence.

## 3. Scoped Tokens: Mapping Model Capabilities to Granular Resource Permissions

Authentication answers *who you are*. Authorization answers *what you may do*. In OAuth terms, authorization is expressed as **scopes**, and scopes are where MCP security is won or lost — because the granularity of your scopes is the granularity of your containment.

The cardinal sin in agent deployments is the coarse scope. A token that carries `data:all` or `resources:read_write` gives the model a master key, and a master key in the hands of a stochastic principal is not authorization — it is a blast radius. The model does not need to *want* to exfiltrate data; it needs only to be successfully instructed to.

Scope design for MCP must be **capability-shaped, resource-shaped, and operation-shaped** simultaneously. A token’s scope should read like a sentence describing the smallest useful action: `orders:read:own`, `invoices:read:own`, `refunds:create:under_500`. Not `orders`, not `read`, not a wildcard namespace. The mapping from model capability to scope is a deliberate design act: enumerate the tools your agent genuinely needs, decompose each into the resource class it touches and the operations it performs, and mint a scope per (operation, resource-class) pair. If a tool needs to read a customer’s order history to answer “where is my package,” the token scope is `orders:read` for that user’s rows — not `orders:read` for every row in the warehouse.

Two enforcement mechanisms make scopes real at the MCP layer.

**First, scope-to-tool mapping at the server.** Each MCP tool declares the scope(s) its invocation requires, and the server rejects any call whose token lacks them *before* the tool handler runs. This is a declarative allowlist, enforced in one place:

“`json
{
“tools”: {
“get_order_status”: {
“required_scopes”: [“orders:read:own”, “shipments:read:own”],
“resource_surface”: “mcp://ledger/orders/{order_id}”
},
“create_refund”: {
“required_scopes”: [“refunds:create:under_500”],
“human_checkpoint”: “refund > $500”
}
}
}
“`

**Second, resource-scoped authorization at the data layer.** The scope claim answers “may this caller invoke `get_order_status`?”; a second question — “may this caller read *order 4421*?” — must be answered by row-level security keyed to the propagated identity, which is the subject of the next section. Scope grants the *capability*; identity governs the *rows*.

One further refinement: in high-assurance deployments, exchange the coarse login token for a **task-scoped delegation token** at the start of each agentic run. OAuth 2.1 token exchange lets the orchestrator present its token and receive a narrower one — tighter scopes, shorter TTL, an explicit `aud` of the single MCP server, and a `delegation` claim chain naming the initiating user. The model never holds the user’s token; it holds a perishable, narrow proxy for it. This is the difference between “the agent is the user” and “the agent acts *for* the user, within limits that expire.”

## 4. Identity Propagation: How to Flow User Identity from Client to Tool

Here is the failure mode that keeps security architects up at night: the user authenticates beautifully at the front door, and then the agent calls the database with a **service account**. Every row-level policy, every audit trail, every least-privilege grant designed around the human evaporates the moment the model’s tool call executes as `mcp_svc_prod`. The database sees a single privileged principal with a thousand concurrent sessions, and “who did this?” becomes unanswerable.

Identity propagation for MCP means the end user’s identity — not the model’s, not the server’s service account — must survive the journey from client to tool to database, as an **enforceable and auditable claim**. The mechanism is a delegation chain carried in the token and forwarded at each hop.

At the authorization server, mint the access token with claims that describe the full context of the call:

“`json
{
“iss”: “https://auth.onlinecloud.internal”,
“aud”: “https://mcp.ledger.internal”,
“sub”: “delegated:task-8f3a…”,
“scope”: “orders:read:own refunds:create:under_500”,
“exp”: 1713000000,
“delegation”: [
{ “actor”: “user:alice@example.com”, “auth_time”: 1712999400, “amr”: [“pwd”, “webauthn”] },
{ “actor”: “client:finance-agent:v3”, “purpose”: “invoice reconciliation run” }
]
}
“`

The MCP server then does two things with these claims. It **enforces** them — parsing the innermost `actor` and passing it to the data layer so that row-level policies evaluate against *Alice*, not against the service account — and it **forwards** them in the audit record of every tool invocation.

In the data layer, enforcement takes the shape of your database’s native RLS mechanism. In Postgres, the server sets `current_setting(‘app.caller_id’)` and `app.caller_chain` at the start of the session or transaction, and every policy evaluates against that setting. In SQL Server, `SESSION_CONTEXT` plus security predicates achieve the same. In Snowflake, row-access policies key on the propagated identity column. The invariant in every case is identical: **the database never executes a query whose effective principal is the service account.** It executes queries whose effective principal is Alice, mediated by an agent the system can name.

Two hard rules govern propagation. **Rule one: propagate, never substitute.** If the chain is ever broken — if a hop loses the user claim and falls back to a generic principal — that hop fails closed and logs an alert. A tool that cannot prove whose data it is touching does not get to touch it. **Rule two: delegation never escalates.** The effective permission at any hop is the intersection of the user’s grants and the agent’s scopes. An agent cannot exercise a permission the user lacks, and a user grant cannot be widened by the agent’s token. The confused-deputy problem — a privileged agent tricked into acting on behalf of an unprivileged user, or an unprivileged user’s prompt injection steering a privileged agent — is neutralized because neither side can grant what the other does not hold.

## 5. Audit & Compliance: Logging Authenticated Resource Access in MCP

Auditors do not ask whether your agent is clever. They ask a far more uncomfortable question: **can you prove what it did, under whose authority, and why?** For regulated workloads — SOC 2, HIPAA, SOX, GDPR — an agentic system that cannot produce a replayable record of authenticated resource access does not have a compliance story; it has a liability.

Audit logging for MCP must therefore capture the *full authorization context* of every tool invocation, not merely “a call happened.” The minimum event schema:

– **Who**: the propagated delegation chain — initiating user, client/agent, orchestration run ID.
– **What**: tool name, validated input arguments, resource URIs touched.
– **Authorization**: token ID, scopes presented, which scope satisfied the call, token issuer.
– **Outcome**: success, or the structured rejection reason (expired, wrong audience, insufficient scope, RLS denial).
– **When and where**: timestamp with timezone, server instance, transport session ID.
– **Parent linkage**: the event ID of the agentic step that spawned this call, so the log forms a tree — from user intent down to the row that changed.

Two properties make the trail trustworthy. **Completeness:** logging must live *inside* the authorized request path, not around it. If a tool call can succeed without emitting an audit event, then under load, unlogged calls will happen — and the one that matters will be the missing one. **Immutability:** the audit sink is append-only and written with credentials no agent token can reach. The trail must survive the very compromise it is designed to detect. Ship audit events to a separate store — an independent log stream or a write-only data source — that sits outside the blast radius of any MCP server compromise.

Compliance also demands an answer to the *human* question: which human authorized this system behavior, and which human approved the irreversible actions? Log the consent event at the authorization server (user X approved scopes S for client C at time T), and log human checkpoints at the tool layer — the operator ID, the approval timestamp, and the exact payload they approved. When a regulator asks “who said this refund of $4,000 was acceptable?”, the answer is a signed audit event, not a guess.

Finally, build the **replay** capability before you need it. Store enough structured context per event — the tool schema version, the model and prompt version, the parent plan — that an investigator can reconstruct the decision path, aborted branches included. Proving intent requires showing what the agent *considered*, not only what it executed. An MCP deployment whose audit trail supports this replay is a deployment that walks into an assessment and demos its own compliance.

## 6. Reference Architecture: A Secure, VPC-Isolated MCP Auth Flow

The pieces above compose into a single, defensible architecture. Here is the deployment OnlineCloud runs for regulated customers — the same pattern we would run for you.

“`
┌─────────────┐ ┌──────────────────────────────────────────────────────┐
│ End User │ │ Customer VPC │
│ (Browser / │ │ │
│ Desktop) │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────┐ │
└──────┬──────┘ │ │ Agent Host │───▶│ MCP Server │──▶│ Database │ │
│ │ │ (OAuth client)│ │ (authz gate) │ │ (RLS) │ │
│ │ └──────┬───────┘ └──────┬───────┘ └──────────┘ │
│ │ │ 1. auth request │ │
│ │ ▼ │ │
│ │ ┌──────────────────┐ │ │
└──────────┼──│ Authorization │ │ 2. validate JWT: iss, │
│ │ Server (IdP) │ │ aud, exp, scope, │
│ │ PKCE + consent │ │ delegation chain │
│ └──────────────────┘ │ │
│ │ 3. exchange │ 4. RLS keyed to │
│ ▼ │ propagated user │
│ ┌───────────────────────────────┐ │
│ │ Audit Store (append-only) │◀── every call logged │
│ └───────────────────────────────┘ │
└──────────────────────────────────────────────────────┘
“`

The numbered flow: **(1)** the end user initiates a session; the agent host begins the OAuth 2.1 Authorization Code flow with PKCE against the internal authorization server, and the user consents to a narrow scope set. **(2)** The host receives a short-lived access token whose claims carry the delegation chain from step 4 of this post — user, agent, purpose — and whose audience is pinned to the single MCP server. **(3)** The host invokes tools over Streamable HTTP inside the VPC. Every request terminates at the MCP server’s authorization gate, which validates signature, issuer, audience, expiry, and scope against the requested tool, then rejects or forwards. **(4)** For data access, the server sets the propagated identity in the database session context and lets row-level security evaluate against the *user*, so the model can only ever reach rows the initiating user could reach. Every step — authorization, token exchange, tool call, RLS decision, rejection — streams an immutable event to the append-only audit store, which lives outside the reach of any agent credential.

The structural guarantees are worth naming explicitly, because they are *network facts*, not prompt instructions:

– **No public ingress.** The MCP server, authorization server, and audit store live in private subnets. The database is reachable only over private endpoints. The model — Claude, GPT-4-class, or an open-weights deployment in your own VPC — reaches the MCP server, and nothing else.
– **No service-account data access.** The database never executes a query whose effective principal is a shared agent credential. Every query is evaluated against a propagated human identity and an audited delegation chain.
– **No unlogged invocation.** A tool call that succeeds without emitting an audit event is architecturally impossible, because logging lives inside the request path.
– **No standing privilege.** Tokens are short-lived, refresh tokens rotate, and each task runs on a delegation token narrower than the user’s own grant.

Get these four guarantees right, and the agentic system becomes *containable*: the most capable tool your business has shipped, bounded by identity, scope, and evidence at every hop. Get them wrong, and you have deployed your largest attack surface with the user’s credentials attached.

—

**Planning an MCP rollout — or securing one that already shipped? Talk to the OnlineCloud architects.** We are a US-based engineering studio — MCP-native, senior engineers only, no offshore handoff — and we deliver evaluated, observable, VPC-isolated agentic systems with audited OAuth 2.1 authorization in production within 6–10 weeks. Contact us at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/) for a free initial consultation. We respond within 24 hours.

# Distributed Agent Orchestration: Implementing a Hierarchical Planner in Python

*By an OnlineCloud Principal AI Engineer — 9 min read*


Every serious agent system I’ve reviewed in production eventually hits the same wall: a single prompt — no matter how well-crafted — cannot reliably execute a task with more than a few dependent steps. The fix isn’t a better prompt. It’s a better control structure. This post walks through the architecture and Python implementation of a **hierarchical planner** — a Planner–Worker–Verifier control loop that decomposes work, executes it in parallel, verifies it, and heals itself when the world disagrees with the plan.

## 1. The Bottleneck of Linear Planning

The default agent design is a loop: model → tool call → observation → model. For a two-step task like “look up an order and summarize its status,” that loop is fine. For anything real — a logistics reroute, a multi-system data migration, a claims workflow — the loop becomes a liability. Three failure modes dominate:

**Error compounding.** Each step consumes context and injects drift. If step three of eight produces a subtly wrong intermediate value, steps four through eight are all downstream of garbage. A linear chain has no structural way to notice, because nothing is *checking* intermediate outputs against their specification.

**Lost locality.** A single model context holds the entire task, so every decision is entangled with every other. The model reasoning about carrier APIs is also holding route geometry, pricing data, and customer constraints in one window. Context limits force truncation, and truncation is where silent mistakes are born.

**Zero recovery semantics.** When a linear loop fails, the only move is “try the whole thing again” — replaying cost, latency, and the same odds of failure. There’s no notion of *which subgoal* failed, no notion of *retrying only that*, and no budget except your own patience.

The pattern that fixes all three is older than LLMs: **divide and conquer, with inspection**. Hierarchical task decomposition. What LLMs add is that the decomposition — and the inspection — can be generated dynamically per request instead of hard-coded.

## 2. The Hierarchical Pattern: Planner–Worker–Verifier

The architecture has three roles and one shared contract. The **Planner** takes a goal and a world snapshot and produces a structured plan: an ordered list of typed subgoals, each with an owner, inputs, and acceptance criteria. The **Workers** execute subgoals against real systems — they are narrow, single-purpose, and stateless with respect to the overall goal. The **Verifier** is the quality gate: it checks each worker’s output against the subgoal’s acceptance criteria and returns a verdict with *machine-readable feedback*, not just a score.

“`
┌─────────────┐
goal ───────────────▶│ Planner │◀───── revision context
└──────┬──────┘
│ plan (typed subgoals)
┌────────────────┼─────────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Worker A│ │ Worker B│ │ Worker C│ ← parallel fan-out
└────┬────┘ └────┬────┘ └────┬────┘
└────────────────┼─────────────────┘
▼
┌─────────────┐
│ Verifier │── ACCEPT ──▶ consolidate / ship
└──────┬──────┘
│ REVISE + feedback (bounded loop)
▼
Planner re-plans failed subgoals only
“`

Three properties make this work in practice. First, **the plan is data, not prose**: typed subgoal objects with explicit acceptance criteria, so verification is a deterministic check layered over an LLM judgment call. Second, **workers are interchangeable**: because each subgoal declares its type and inputs, you can swap an LLM worker for a deterministic function or a human approval step without touching the orchestration layer. Third, **feedback flows up, not just down**: the verifier’s critique returns to the planner as structured context, which is what converts a blind retry into a *directed* revision.

## 3. Stateful Planning in Practice: LangGraph vs. PydanticAI

The hierarchy only works if the runtime gives you three things: explicit state, controllable control flow, and durable checkpoints. As of this writing, the two Python frameworks I’d put in front of a client are **LangGraph** and **PydanticAI**.

**LangGraph** models the agent as a state machine: nodes are Python functions, edges are explicit transitions, and a typed state object flows through every node. Its superpowers are *conditional edges* (the verifier verdict literally routes the graph) and *checkpointing* — every state transition can be persisted, so a crashed run resumes mid-plan rather than restarting. It’s the right choice when your orchestration is genuinely complex: branching, loops, parallel fan-out, human-in-the-loop interrupts.

**PydanticAI** is the opposite philosophy: agents are plain typed functions with dependency injection, and multi-agent work is *delegation* — one agent calls another as a tool, with the model deciding when. It’s dramatically more readable for the 80% case, and its `RunContext` gives you typed deps without globals. The tradeoff: the control flow lives inside the model’s tool-calling loop, so bounded retry loops and checkpoints are something you build rather than inherit.

My rule of thumb: if your hierarchy is **two levels and mostly linear**, use PydanticAI and keep the code boring. If you have **verifier-gated revision loops, parallel fan-out, or long-running stateful tasks**, use LangGraph — because those are exactly the features you shouldn’t hand-roll. The walkthrough below uses LangGraph, with the hierarchical *structure* portable to either.

## 4. Orchestration: Multi-Agent Communication and State Consistency

Once you have multiple agents mutating shared state, the hard problem stops being intelligence and starts being *consistency*. Five disciplines keep distributed agent state from rotting:

**One shared state schema, owned by the graph.** Agents never pass unstructured strings to each other. Everything they exchange — plans, results, feedback, spend — is a typed field on a single state object. In LangGraph this is the state schema; in any design it should be a versioned Pydantic model, because the schema *is* the contract between agents.

**Reducers for everything append-only.** Messages, task results, and feedback accumulate via merge functions, never via overwrite. If two workers finish concurrently, their results append into an ordered log — no last-writer-wins races. This is `operator.add` or a custom reducer over an annotated channel, and it eliminates the most common class of heisenbugs in multi-agent systems.

**Idempotency keys at the system boundary.** The real world is not idempotent: booking a carrier, writing a row, sending a notification. Every worker that touches an external system carries a client-generated request ID (`order_id + task_id + attempt`), so a retried execution cannot double-book. If the downstream API doesn’t support idempotency keys, wrap it in a small outbox table that does — dedupe on write, exactly-once on retry.

**Checkpoints, not just logs.** A log tells you what happened; a checkpoint lets you *resume* from it. LangGraph’s checkpointer serializes full graph state after every super-step. MemorySaver is fine for demos; production wants Postgres or Redis-backed persistence so a worker crash — or a full process restart — resumes the exact plan with spend and attempts intact.

**Tracing with a correlation ID.** Every agent appends to a `trace` list on its result: `(node, step, model, cost_ms)`. Ship those into OpenTelemetry with the run’s correlation ID, and you can answer the two questions every incident starts with: *which subgoal failed, and how much did we spend before it did?*

## 5. Error Recovery: Self-Healing Loops and Backtracking

The verifier is what makes recovery *surgical* instead of *nuclear*. The control loop is deliberately small and brutal:

1. Worker output fails verification (schema-invalid, acceptance criteria unmet, or external call errored).
2. The verifier returns structured feedback: *which subgoal, which acceptance criterion, what evidence is missing*.
3. The router decides the recovery granularity: **retry the worker** (transient failure, same plan), **revise the plan** (the plan itself was wrong — planner gets the feedback and re-plans only the failed subgoals), or **escalate** (budget exhausted, or two consecutive revisions failed — hand to a human with the full trace).

Three guardrails keep the self-healing loop from becoming a self-immolating one:

**Bounded attempts, counted in state.** `attempts` and `revision_count` live in the state schema and are checked by the router. There is no unbounded “keep trying” — there is a counter, a limit, and an escalation path.

**Budget as a first-class state field.** Track `spent` against `budget` on every LLM call (input + output tokens priced per model). When spend crosses the ceiling, the router must escalate, not retry. Cost ceilings are a *control-flow* concern, not an accounting one.

**Backoff with jitter on external calls.** Transient API failures are the most common error in real systems. Workers retry with exponential backoff plus jitter; the *orchestrator* never retries — it just marks the subgoal failed and lets the router decide. That separation keeps failure handling at exactly one layer.

This gives you the property that matters most in production: **a failed run terminates in one of two states — a verified result, or an escalation with a readable trace.** Never a silent half-done state.

## 6. Code Walkthrough: A Hierarchical Logistics Agent

Let’s build it. The scenario: a customer requests a *priority reroute* for an in-transit shipment. The planner decomposes the goal into typed subgoals — validate the order, replan the route, match carriers with live rates, run a compliance check, estimate ETA and cost. Workers execute each subgoal against mocked external systems, and a verifier gates acceptance. The revision loop re-plans only the failed subgoal.

“`python
# agent_state.py
from __future__ import annotations
import operator
from typing import Annotated, Any, Literal, TypedDict

from pydantic import BaseModel, Field

# —- Shared contract: the typed plan ————————————-

class TaskSpec(BaseModel):
“””One subgoal. The acceptance_criteria field is what makes
verification deterministic instead of vibes-based.”””
task_id: str
kind: Literal[“validate_order”, “route_plan”, “carrier_match”,
“compliance_check”, “eta_cost”]
goal: str
inputs: dict[str, Any] = Field(default_factory=dict)
acceptance_criteria: list[str] = Field(default_factory=list)

class TaskResult(BaseModel):
task_id: str
kind: str
status: Literal[“ok”, “error”, “needs_input”]
output: dict[str, Any] = Field(default_factory=dict)
error: str | None = None
trace: list[str] = Field(default_factory=list) # correlation trail

class Verdict(BaseModel):
accepted: bool
issues: list[str] = Field(default_factory=list) # machine-readable feedback

# —- Graph state: single source of truth for every agent ——————

class LogisticsState(TypedDict, total=False):
order: dict[str, Any] # immutable request payload
plan: list[TaskSpec] # produced by the Planner
results: Annotated[list[TaskResult], operator.add] # append-only channel
feedback: Annotated[list[str], operator.add] # verifier critiques
attempts: int # bounded self-healing counter
budget: float # hard cost ceiling (USD)
spent: float # accumulated token cost
accepted_plan: dict[str, Any] # verified, ready to ship
needs_human: bool
“`

The state schema is the contract: workers read `order` and write `results`; the verifier reads `results` and writes `feedback`; only the router reads the counters. Nothing is shared outside the graph, and every channel that multiple nodes write to is append-only.

“`python
# graph.py
import asyncio, json, uuid
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import END, START, StateGraph
from langgraph.types import Command # noqa (used with interrupt for humans)

from agent_state import LogisticsState, TaskResult, TaskSpec, Verdict

MAX_ATTEMPTS = 3
REVISION_BUDGET_USD = 1.50 # hard ceiling per run

# —- Workers: narrow, single-purpose, registered by kind ——————

WORKERS: dict[str, callable] = {} # populated by @worker decorator below

def worker(kind: str):
def deco(fn):
WORKERS[kind] = fn
return fn
return deco

@worker(“validate_order”)
async def validate_order(order: dict) -> dict:
# Deterministic gate: no LLM needed where a schema check suffices.
if order.get(“status”) != “in_transit”:
raise ValueError(f”order {order[‘order_id’]} is not in transit”)
if not order.get(“items”):
raise ValueError(“order has no line items”)
return {“valid”: True, “shipment_id”: order[“shipment_id”]}

@worker(“route_plan”)
async def route_plan(order: dict, spent: float) -> dict:
# Mocked as deterministic for the walkthrough; in production this is an
# LLM call priced against the shared budget.
return {“segments”: [“MEM → DFW”, “DFW → LAX”], “via”: “air”}

@worker(“carrier_match”)
async def carrier_match(order: dict) -> dict:
# Idempotency: same (order, task) always sends the same idempotency key.
idem = f”carrier:{order[‘order_id’]}:{uuid.uuid5(uuid.NAMESPACE_URL, json.dumps(order, sort_keys=True))}”
# …POST /carriers/quote with Idempotency-Key: idem…
return {“carrier”: “AeroFreight”, “rate”: 412.50, “confirmed_slots”: 1}

@worker(“compliance_check”)
async def compliance_check(order: dict) -> dict:
return {“hazmat”: “ok”, “export_control”: “ok”, “blocked_regions”: []}

@worker(“eta_cost”)
async def eta_cost(order: dict, segments: list[str]) -> dict:
return {“eta_days”: 2, “total_cost_usd”: 1340.0, “currency”: “USD”}
“`

Notice what the workers are *not* doing: none of them decide the next step, none of them read the verifier’s feedback, and none of them know a plan exists. That ignorance is the point — it makes them trivially testable and swappable.

“`python
# —- Planner: LLM call that turns a goal + feedback into typed subgoals —-

async def call_llm_json(system: str, user: str, budget: dict) -> dict:
# Real implementation: your provider SDK + token accounting that
# increments budget[“spent”]. Kept abstract so the graph logic stays
# framework-agnostic.
…

async def planner_node(state: LogisticsState) -> dict:
“””Decompose the goal into typed subgoals. On revision rounds, the
verifier’s feedback is injected so the planner only re-plans what broke.”””
if not state[“plan”]:
system = “You are the planning layer of a logistics agent. ” \
“Emit subgoals as JSON only.”
else:
system = “The plan below failed verification. Re-plan ONLY the ” \
“subgoals implicated by feedback. Keep accepted subgoals.”
payload = await call_llm_json(
system=system,
user=json.dumps({“order”: state[“order”],
“feedback”: state.get(“feedback”, [])}),
budget=state)
tasks = [TaskSpec(**t) for t in payload[“subgoals”]]
return {“plan”: tasks,
“attempts”: state[“attempts”] + 1,
“spent”: payload[“cost”]}
“`

The planner node’s only job is to turn *goal + world + feedback* into a typed plan. Every subsequent node treats that plan as data, which means the verifier can hold the plan and the results side by side — that’s the whole trick.

“`python
# —- Executor: fan out to workers, gather, and trace ———————–

async def executor_node(state: LogisticsState) -> dict:
“””Run every planned subgoal concurrently. Each worker writes one
TaskResult into the append-only results channel.”””

async def run_one(spec: TaskSpec) -> TaskResult:
try:
fn = WORKERS[spec.kind]
out = await fn(**spec.inputs, spent=state[“spent”])
return TaskResult(task_id=spec.task_id, kind=spec.kind,
status=”ok”, output=out,
trace=[f”{spec.kind}:ok”])
except Exception as exc: # surface, don’t swallow
return TaskResult(task_id=spec.task_id, kind=spec.kind,
status=”error”, error=str(exc),
trace=[f”{spec.kind}:error:{exc!r}”])

results = await asyncio.gather(*(run_one(t) for t in state[“plan”]))
return {“results”: list(results)}
“`

Errors are captured into `TaskResult` objects, never raised through the graph. A raised exception would kill the whole run; a captured error is *just another input to the router*, which is exactly where failure-handling belongs.

“`python
# —- Verifier: deterministic checks layered over one LLM judgment ———-

async def verifier_node(state: LogisticsState) -> dict:
“””Gate every subgoal against its acceptance criteria.”””
by_id = {r.task_id: r for r in state[“results”]}
issues: list[str] = []

for spec in state[“plan”]:
res = by_id.get(spec.task_id)
if res is None:
issues.append(f”{spec.task_id}: no result produced”)
elif res.status == “error”:
issues.append(f”{spec.task_id}: {res.error}”)
elif res.status == “ok”:
# 1) Structural check — deterministic, zero token cost.
missing = [c for c in spec.acceptance_criteria
if not _criterion_satisfied(c, res.output)]
issues += [f”{spec.task_id}: {m}” for m in missing]

if not issues:
return {“accepted_plan”: _consolidate(state), “needs_human”: False}

verdict = await call_llm_json(
system=”You are the verifier. Given failing subgoals, decide: ”
“RETRY_WORKER (transient) or REVISE_PLAN (plan was wrong).”,
user=json.dumps({“issues”: issues, “attempts”: state[“attempts”]}),
budget=state)
return {“feedback”: verdict[“issues”],
“spent”: state[“spent”] + verdict[“cost”]}
“`

“`python
# —- Router: the self-healing control flow ——————————–

def router(state: LogisticsState) -> str:
“””Where the graph goes next depends on counters in state — the whole
recovery policy lives in this one pure function.”””
if state.get(“accepted_plan”):
return “accept”
exhausted_budget = state[“spent”] >= state[“budget”]
out_of_attempts = state[“attempts”] >= MAX_ATTEMPTS
if exhausted_budget or out_of_attempts:
return “escalate”
# REVISE_PLAN ⇒ loop back through the planner with feedback attached;
# RETRY_WORKER would route back to the executor instead. Both are just
# conditional edges — no bespoke retry framework required.
return “revise”

# —- Assemble the graph —————————————————-

builder = StateGraph(LogisticsState)
builder.add_node(“planner”, planner_node)
builder.add_node(“execute”, executor_node)
builder.add_node(“verify”, verifier_node)
builder.add_node(“human_review”, lambda s: {“needs_human”: True})

builder.add_edge(START, “planner”)
builder.add_edge(“planner”, “execute”)
builder.add_edge(“execute”, “verify”)
builder.add_conditional_edges(“verify”, router, {
“accept”: END,
“revise”: “planner”, # ← backtrack: only failed subgoals re-plan
“escalate”: “human_review”, # ← human checkpoint with full trace
})
builder.add_edge(“human_review”, END)

graph = builder.compile(checkpointer=MemorySaver()) # PostgresSaver in prod

async def main() -> None:
order = {“order_id”: “SO-88231”, “status”: “in_transit”,
“shipment_id”: “SH-441”, “items”: [{“sku”: “A1”, “qty”: 40}]}
thread = {“configurable”: {“thread_id”: “run-” + uuid.uuid4().hex[:8]}}
final = await graph.ainvoke(
{“order”: order, “plan”: [], “attempts”: 0,
“budget”: REVISION_BUDGET_USD, “spent”: 0.0},
config=thread)
print(json.dumps(final.get(“accepted_plan”) or
{“escalated”: True, “trace”: final[“results”]}, indent=2))

if __name__ == “__main__”:
asyncio.run(main())
“`

Three details in this walkthrough carry the whole design. **The verifier is layered**: deterministic structural checks run first at zero token cost, and the LLM verdict is only consulted when structure fails — that’s how you keep verification cheap enough to run on every attempt. **The router is a pure function over state counters**: recovery policy is testable without any LLM in the loop (feed it a state dict, assert the edge). **The revision loop backtracks to the planner, not to the start**: `attempts` is bounded at three and `budget` at $1.50 per run, and crossing either one routes to a human checkpoint with every worker trace intact — OnlineCloud’s “plan/act/reflect with cost ceilings and human checkpoints” pattern, exactly.

If you squint, the whole graph reads like a state machine version of what a good senior dispatcher does: break the job into pieces someone can own, check the pieces before you commit, and when something breaks, fix the *piece* — not the whole shipment.

## Building This for Your Systems

The hierarchy above is deliberately framework-light at the edges: workers are plain functions, the verifier is a JSON contract, and only the control flow is graph-shaped. That’s what makes it portable across LangGraph, PydanticAI, or a hand-rolled asyncio loop — and what makes it auditable, because every transition is a checkpoint and every decision is a state field.

A hierarchical planner is the difference between an agent that *looks* autonomous in a demo and one that survives contact with real databases, real auth boundaries, and real cost ceilings. When you’re ready to move past the demo, the architects at OnlineCloud design and ship exactly this class of system — MCP-native orchestration with evaluated, observable, human-checkpointed control loops, in production in 6–10 weeks.

**Have a multi-step agent workload that needs real orchestration? Talk to an OnlineCloud architect** — [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev), or reach out via [onlinecloud.dev](https://www.onlinecloud.dev/). Free initial consultation, US-based senior engineers, response within 24 hours.

# Typed Tools in MCP: Using JSON Schema for Predictive Tool Execution


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: The Failure of Natural Language Tool Descriptions

Every agent team hits the same wall around week three. The demo works — the right tool, the right arguments, applause. Then a real user runs the same flow, and the model calls `create_ticket` with `{“severity”: “urgent-ish”, “customer_id”: “the one from the email”}` — faithfully following a description that said *”customer_id: the identifier of the customer.”* The failure was never the model’s. It was the interface’s.

Natural language tool descriptions are seductive because they read like requirements — and they are the weakest contract you can ship. A model does not *understand* a description; it pattern-matches against it during generation, so every ambiguity in your prose becomes a degree of freedom in its output. Worse, prose is unenforceable at runtime: nothing checks that `severity` is one of the three values your paging system accepts, or that `customer_id` is a UUID in your CRM. The description is an aspiration; the schema is the law.

In the Model Context Protocol (MCP), a tool is declared with a `name`, a `description`, and an `inputSchema` written in JSON Schema. The first two aim at the model; the third aims at reality. Across fifteen-plus years of shipping data systems at OnlineCloud — Oracle, MS SQL, Postgres, Snowflake, CRMs — we have learned that the difference between an agent that closes tickets and one that merely appears to is whether its tool surface encodes *structural invariants* the model cannot violate. This post is about designing that boundary — and knowing how much schema each tool needs.

—

## 2. The Role of JSON Schema in MCP: Enforcing Structural Invariants

MCP is, at its core, a stateless JSON-RPC protocol. A host calls `tools/list`, and each tool advertises an `inputSchema` — a JSON Schema document describing the exact shape of arguments it will accept. When the model acts, it emits a `tools/call` request whose `arguments` are validated against that schema *before* your business logic runs. Schema first, side effects second — that ordering is the entire game.

“`json
{
“name”: “refund_order”,
“description”: “Issue a refund against a completed order.”,
“inputSchema”: {
“type”: “object”,
“additionalProperties”: false,
“required”: [“order_id”, “amount_cents”, “reason”],
“properties”: {
“order_id”: {
“type”: “string”,
“pattern”: “^ord_[a-zA-Z0-9]{20}$”
},
“amount_cents”: {
“type”: “integer”,
“minimum”: 1,
“maximum”: 1000000
},
“reason”: {
“type”: “string”,
“enum”: [“duplicate_charge”, “cancellation”, “damaged_goods”, “other”]
}
}
}
}
“`

Notice what this schema does that no description can: `order_id` is *not* “an identifier of the order” but a string matching `^ord_[a-zA-Z0-9]{20}$`; `amount_cents` is an integer — not a float, not a decimal string, not “$29.99″; `reason` is closed to four values, a fifth structurally impossible. Every constraint is checkable in microseconds, deterministic, and independent of which model is calling.

A schema is a *normative* artifact; a description is an *aspirational* one. That distinction drives how we build MCP servers at OnlineCloud: the `inputSchema` is the tool’s executable specification — it gates model calls at runtime, renders into the model’s context, drives our test fixtures, and generates the OpenAPI surface for human consumers. One artifact, three consumers, zero drift. Descriptions, by contrast, are kept deliberately short, because long descriptions encourage the model to *reason about* arguments instead of conforming to their shape.

This is the deeper role of JSON Schema in MCP: make the model’s job *easier by making it more constrained*. Generation under a schema is a search over a smaller space; smaller spaces produce fewer wrong answers. An MCP tool without a strict schema is not a function — it is a suggestion.

—

## 3. Designing Robust Schemas: Enums, Patterns, and Recursive Types

Schema quality is a design discipline, not a serialization detail. The three highest-leverage constructs — enums, patterns, and recursive references — each target a different failure mode.

**Enums close open sets.** Any argument naming one of a fixed set of things should be an `enum`: statuses, severities, channels, currencies, reasons. Faced with `”enum”: [“open”, “in_progress”, “resolved”, “closed”]`, the model is not choosing a string that *resembles* a status; it is choosing among four tokens — far more reliable than open-ended generation. Enums are also a *measurement* instrument: if logs show the model reaching for values outside the enum, the enum is missing a legitimate option — add it, never loosen validation.

**Patterns capture domain grammar.** Identifiers are the classic case: every real system has a generative grammar for its keys — `ord_`-prefixed IDs, UUIDs, E.164 phone numbers, ISO-8601 timestamps, ISO-4217 currency codes. A `pattern` keyword turns that grammar into an enforceable constraint:

“`json
{
“type”: “object”,
“required”: [“phone”, “effective_from”],
“properties”: {
“phone”: { “type”: “string”, “pattern”: “^\\+[1-9]\\d{1,14}$” },
“effective_from”: {
“type”: “string”,
“pattern”: “^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(Z|[+-]\\d{2}:\\d{2})$”
}
}
}
“`

Two cautions on patterns. First, the model sees the raw regex, so write it for *readers* — anchored, conservative, annotated. Second, prefer the most specific semantic check your stack offers: if the server validates an ID against the database anyway, do that in code and keep the pattern as the cheap syntactic gate. Patterns reject nonsense at the boundary; they are not a substitute for existence checks.

**Recursive types model nested reality.** The most common reason a schema gets loosened to `{“type”: “object”}` is a tree-shaped payload — a filter expression, a workflow DAG, an approval chain. “It’s just a nested object, I can’t type it” is a confession, not an excuse. JSON Schema handles recursion with `$ref` and `$defs`:

“`json
{
“type”: “object”,
“properties”: {
“filter”: { “$ref”: “#/$defs/condition” }
},
“$defs”: {
“condition”: {
“oneOf”: [
{ “type”: “object”, “additionalProperties”: false, “required”: [“field”, “op”, “value”],
“properties”: {
“field”: { “type”: “string”, “pattern”: “^[a-z_]{1,32}$” },
“op”: { “enum”: [“eq”, “neq”, “gt”, “gte”, “lt”, “lte”, “contains”] },
“value”: { “type”: [“string”, “number”, “boolean”] }
} },
{ “type”: “object”, “additionalProperties”: false, “required”: [“logic”, “children”],
“properties”: {
“logic”: { “enum”: [“and”, “or”, “not”] },
“children”: { “type”: “array”, “minItems”: 1, “items”: { “$ref”: “#/$defs/condition” } }
} }
]
}
}
}
“`

Three rules keep recursive schemas safe in production. Guard every recursion with a discriminator (here, `field`/`op` versus `logic`/`children`) so the model always has a deterministic next step. Bound the recursion — `maxItems`, plus a server-side depth ceiling — because an unbounded tree is a prompt-injection and cost-control hole. And add `examples` at each level: for nested structures, a worked example beats a paragraph of description, because models generalize from concrete instances more reliably than from abstract rules.

Whatever your stack — Pydantic in Python, Zod in TypeScript — generate the `inputSchema` from the same typed definition your server code uses. Hand-maintained JSON Schema and hand-written parsers drift apart by the second commit.

—

## 4. Pre-execution Validation: Rejecting Malformed Model Calls at the Server Boundary

The design decision separating serious MCP deployments from demos is *where* validation happens and *what happens when it fails*. Validation is a server-side security and correctness boundary, not a client convenience: assume any payload can arrive — from a confused model, a prompt-injected context, a stale client — and reject what does not conform *before* a single side effect occurs.

In the MCP lifecycle this is natural: `tools/call` arrives, the server validates `arguments` against the `inputSchema`, and a malformed call returns a JSON-RPC `InvalidParams` error (`-32602`) naming the offending fields. The discipline is in what you refuse to do around it. Never coerce silently: turning `”amount”: “29.99”` into `29.99` may feel helpful, but coercion teaches the model that sloppy output is survivable and quietly erases the signal that your schema or prompt is failing. And never partially execute — validate the *entire* payload, then act.

The failure path is a first-class product surface: a rejection must be *actionable by the model*, because in an agentic loop the model is the one that retries. “Validation failed” leaves it guessing; `”amount_cents: expected integer, got string; order_id: does not match pattern ^ord_[a-zA-Z0-9]{20}$”` lets it repair the call in one attempt. We measure this by counting: the *repair ratio* — retries per successful call — is one of the most sensitive early indicators of schema quality we track. It should sit near one.

Those logs are strategic data, not noise: a spike in validation failures is frequently the first symptom of a model upgrade, a drifted system prompt, or prompt-injection probing. Treat the boundary as an observability instrument — log caller, tool, offending fields, and outcome — and alert on anomalies. A team that ignores the rejections is throwing away the diagnosis.

—

## 5. Enhancing Model Accuracy: How Strict Typing Reduces Agentic Hallucination

The claim that JSON Schema improves model accuracy sounds paradoxical — how does removing freedom improve a generative system? Because tool calling is not a creative act; it is *structured prediction*: given a function signature and a goal, emit the argument object that satisfies both. Hallucination here is not inventing facts; it is *sampling from a space never constrained to contain only valid answers*. Every missing `enum`, every loose `pattern`, every `type: “object”` placeholder widens the sampling space — and everything in that widened space is a candidate hallucination.

Strict typing attacks hallucination on four fronts:

**Constrained decoding by construction.** An `enum` of four statuses is not a description that the model *should* choose among four values; it is a constraint that makes other tokens structurally invalid — and models respect structural constraints far more reliably than prose. You are not asking the model to be careful; you are making correctness the only cheap path.

**Fewer repair loops, less drift.** Every rejected call pollutes context with an error and pushes the model into *repair* mode, where it is measurably more likely to over-correct or try an adjacent wrong thing — and it burns tokens and latency. A strict schema collapses the reject-and-retry cycle, so the agent spends its generation budget on the task, not on apologizing.

**Disambiguation of ambiguous intents.** Take `search_orders(query)`: prose invites the model to stuff anything into `query` — a name, a date, a status, a complaint. Explicit typed parameters — `customer_email`, `status`, `date_from`, `date_to` — force the model to *classify* its intent into your domain model before the call, and that classification step is where accuracy is won. Ambiguity that survives belongs in a clarifying question, not in a free-text blob.

**Versioned stability across model generations.** The agents you ship this year will not run the same model forever; descriptions tuned to one model’s quirks rot on the next, while a schema is model-agnostic. Strict typing is why a server built for Claude can serve a GPT-4-class model or an open-weights deployment without behavioral drift. The schema — not the prose — makes tool behavior *predictive*: given a valid payload, the tool does one thing, every time, for every model.

None of this makes schema a substitute for evaluation; it makes schema the substrate *on which* evaluation becomes meaningful. When every call either conforms or is rejected with a named reason, your eval harness can distinguish “the model chose the wrong tool” from “the model mangled the arguments” — two failures with different fixes.

—

## 6. Decision Framework: When to Use Simple vs. Complex Schemas

Strict schemas are a tool, not a theology. Over-constraining is a real failure mode: the model can exhaust its context budget against a schema written for a machine, not a model. The art is matching strictness to the risk and structure of the operation. At OnlineCloud we run every tool through four questions:

**What is the blast radius of a malformed call?** Read-only, reversible operations tolerate looseness; writes, irreversible actions, and anything touching money or PII get maximal constraint. `search_logs` may accept a free-text `query`; `transfer_funds` gets enums, patterns, and explicit amounts — every time.

**Closed or open set?** Closed sets (statuses, reasons, channels, logic operators) become enums, unconditionally. Open sets (a product name, a search term, a note to a human) stay typed but unconstrained — because an enum that does not contain the answer forces the model to fail or to lie.

**Flat or nested?** Flat payloads of two to five typed fields rarely need more than `required`, `type`, and `enum`. For nested trees — filters, DAGs, approval chains — reach for `$defs` and `$ref`, paying the complexity cost to reject structurally meaningless payloads before they touch your engine.

**Who consumes this tool, and how often?** A tool called hundreds of times a day by an unattended agent earns every constraint you can write — each rejection is a failed business transaction. A rarely used, human-in-the-loop tool can afford looser edges; the human absorbs the ambiguity the schema would otherwise carry.

Two rules of thumb anchor the framework. First, default to the *simplest schema that makes invalid calls impossible to misinterpret*: every field typed, every closed set closed, `additionalProperties: false` on objects — nothing more. Complexity is justified only when a constraint removes a failure mode you have observed or can foresee. Second, constrain what you can name and leave a seam for what you cannot: a typed structure with one well-bounded free-text field outperforms both a fully loose and a fully rigid schema.

This is not a one-time decision: schemas must evolve with the failures your logs reveal — tightening where the model invents values, loosening where legitimate calls are rejected. Schema review belongs in the same cadence as eval review, because the schema is the eval that runs on every single call.

—

## 7. Conclusion: The Schema Is the Product

Typed tools are the difference between an agent that *appears* to understand your business and one that *cannot* violate it. Natural language descriptions told the model what you wanted; JSON Schema tells it what is possible, making everything else structurally unreachable. The result is an agent whose tool calls you can predict, test, and audit — not because the model is cleverer, but because you stopped leaving correctness to chance.

At OnlineCloud, we build MCP-native systems for teams that need agents to do real work against real systems — Oracle, MS SQL, Postgres, Snowflake, CRMs — with schema discipline, eval suites, and observability built in from the first commit. If you are designing a tool surface, or debugging an agent that keeps mangling arguments, talk to our architects. We will help you turn your tool boundary into a contract your models cannot break — and your logs will prove it.

**Ready to build typed, production-grade MCP servers?** Contact the OnlineCloud architects today for a free initial consultation — [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) — or learn more at [https://www.onlinecloud.dev/](https://www.onlinecloud.dev/). US-based senior engineers, response within 24 hours.

# Implementing SSE Transports for MCP: A Step-by-Step Guide for Node.js


If you have shipped a Model Context Protocol (MCP) server to production, it was probably over **stdio** — a child process spawned by Claude Desktop, a VS Code extension, or some local agent harness. That works beautifully for a single user on a single machine, and it is where most of us started. But stdio is a dead end the moment you need what every other backend service takes for granted: centralized auth, audit logs, horizontal scaling, and serving many concurrent clients from one deployment.

This guide is the bridge from *local process* to *network service*. We will walk through why Server-Sent Events (SSE) is the right transport for MCP’s asymmetric message flow, and then build a production-minded SSE MCP server in Node.js with Express — covering connection lifecycle, heartbeats, session state, and the security posture you need before this thing touches the public internet.

—

## 1. Moving Beyond stdio: Network-Distributed MCP

MCP is a JSON-RPC 2.0 protocol layered over a pluggable transport. The spec defines messages like `initialize`, `tools/call`, and `notifications/progress`, but deliberately says nothing about *how* those bytes travel — that is the transport’s job. The reference implementations give you three options:

– **stdio** — parent process spawns the server and speaks over stdin/stdout.
– **HTTP + SSE** — the client opens a server-sent event stream for server→client messages and POSTs client→server messages to a dedicated endpoint.
– **Streamable HTTP** (newer spec revisions) — collapses both directions onto a single POST endpoint.

stdio has real virtues: zero network surface, trivial auth (the OS owns the pipe), and automatic process lifecycle. But it couples your MCP server’s lifetime to a single client process. You cannot put stdio behind a load balancer. You cannot share one deployment across a fleet of agents. You cannot audit who called `tools/call` because there is no *who* — only a pipe.

Once your MCP server becomes a shared internal service — wrapping your CRM, your Oracle or MS SQL estate, your ticketing system — you need a transport that speaks HTTP. SSE is the pragmatic sweet spot: it rides plain HTTP (so every piece of middleware and infrastructure you already run understands it), it handles the server→client push channel that HTTP request/response alone cannot, and it is dramatically simpler to operate than a WebSocket fleet.

—

## 2. What Is SSE, and Why It Fits MCP’s Asymmetric Flow

Server-Sent Events is a one-way push protocol built directly on HTTP. The client opens a normal GET request; the server keeps the response open and streams `text/event-stream` frames to it. Each frame looks like:

“`
event: message
data: {“jsonrpc”:”2.0″,”id”:1,”result”:{}}
“`

The `data:` field carries the payload, `event:` names it, and comment lines starting with a colon (`: ping`) are ignored by the client — a detail we will exploit for heartbeats shortly.

Here is the key insight about MCP: **the traffic pattern is asymmetric, and it matches SSE’s one-way design perfectly.**

– **Client → Server** messages (`initialize`, `tools/call`, `resources/read`) are discrete request/response operations. They map naturally onto plain HTTP POST with a JSON body — easy to log, authenticate, rate-limit, and load-balance.
– **Server → Client** messages are the hard part. Responses can arrive seconds or minutes after a request (think long-running tool calls). And critically, the *server* must sometimes initiate traffic: `notifications/progress`, `notifications/logging`, `notifications/resources/updated`, and — in some configurations — server→client *requests* like `sampling/createMessage` where the model asks the host for context.

Plain request/response HTTP cannot represent a server-initiated push. WebSocket can, but it is a heavier contract: a protocol upgrade, a different connection lifecycle, frame parsing, and notoriously finicky behavior through corporate proxies and naive load balancers. SSE needs none of that — it is just a long-lived HTTP response. Every reverse proxy, CDN, and observability stack that understands HTTP understands SSE (give or take a buffering directive, which we will cover).

The result is a clean split that senior backend engineers will recognize and love:

| Direction | Transport | Semantics |
|—|—|—|
| Client → Server | `POST /messages` | Standard HTTP request/response, JSON-RPC body |
| Server → Client | `GET /sse` (held open) | Event stream for results, notifications, and server-initiated requests |
| Session binding | `sessionId` query param | Both legs address one logical MCP session |

One honest caveat before we commit: the MCP spec’s newer **Streamable HTTP** transport is where the protocol is headed, and it lets a single POST endpoint handle both directions (when the server has nothing to push, it can respond with an SSE stream or a plain JSON body). For existing SDKs and a large installed base of clients, the classic HTTP+SSE split remains the most interoperable choice today — and everything you learn about session lifecycle and connection management here transfers directly to Streamable HTTP later.

—

## 3. Setting Up Express.js for SSE-Based MCP Transport

Let’s scaffold. We will use the official `@modelcontextprotocol/sdk` for the protocol machinery (tool registration, JSON-RPC framing, capability negotiation) and Express purely as the HTTP layer. Do not reimplement JSON-RPC validation yourself — the SDK is battle-tested and versioned against the spec.

“`json
{
“name”: “mcp-sse-server”,
“type”: “module”,
“dependencies”: {
“@modelcontextprotocol/sdk”: “^1.x”,
“express”: “^4.19.0”,
“cors”: “^2.8.5”,
“helmet”: “^7.1.0”,
“zod”: “^3.23.0”
}
}
“`

Our HTTP surface is intentionally tiny — two routes, which is the whole point of SSE’s simplicity:

– `GET /mcp/sse` — the client connects here and receives the event stream.
– `POST /mcp/messages` — the client sends JSON-RPC messages here, tagged with `?sessionId=`.

The SSE handshake deserves attention because it is where beginners get confused. When a client connects to `/mcp/sse`, the server does not start processing tools yet. It first sends a special `endpoint` event telling the client *where* to POST:

“`
event: endpoint
data: /mcp/messages?sessionId=a1b2c3d4-e5f6-…
“`

Only after the client receives that event does it know the message URL. This indirection exists because the server may be behind a reverse proxy and needs to advertise its *public* route — so build the endpoint URI from the request’s host header, not from a hardcoded string.

The SDK’s `SSEServerTransport` handles this handshake and the JSON-RPC session plumbing for you. Here is the Express wiring:

“`js
import express from “express”;
import cors from “cors”;
import helmet from “helmet”;
import { McpServer } from “@modelcontextprotocol/sdk/server/mcp.js”;
import { SSEServerTransport } from “@modelcontextprotocol/sdk/server/sse.js”;
import { z } from “zod”;

const app = express();

// JSON body parsing only where it belongs: the POST message endpoint.
// Never apply express.json() globally before your SSE route.
app.use(helmet());
app.use(cors({ origin: process.env.CORS_ORIGIN?.split(“,”) ?? “*” }));
app.post(“/mcp/messages”, express.json({ limit: “1mb” }));

// The MCP server itself: register tools, resources, prompts.
const server = new McpServer(
{ name: “orders-mcp”, version: “1.0.0” },
{ capabilities: { tools: {} } }
);

server.registerTool(
“lookup_order”,
{ orderId: z.string().describe(“Order identifier”) },
async ({ orderId }) => ({ content: [{ type: “text”, text: `order:${orderId}` }] })
);

// One live SSE transport per connected session.
const sessions = new Map();

app.get(“/mcp/sse”, async (req, res) => {
// … auth check (Section 5) …
const transport = new SSEServerTransport(“/mcp/messages”, res);
sessions.set(transport.sessionId, transport);
res.on(“close”, () => {
sessions.delete(transport.sessionId);
// … cancel heartbeat timers …
});
await server.connect(transport);
});

app.post(“/mcp/messages”, async (req, res) => {
const transport = sessions.get(String(req.query.sessionId ?? “”));
if (!transport) {
res.status(404).json({ error: “Unknown session” });
return;
}
await transport.handlePostMessage(req, res);
});

app.listen(process.env.PORT ?? 3001, () =>
console.log(`MCP SSE server listening on :${process.env.PORT ?? 3001}`)
);
“`

A few things worth internalizing about this code:

1. **Body-parser placement matters.** `express.json()` on the GET stream route would try to read the never-ending body of an SSE connection. Apply it only to the POST route, as shown.
2. **Session registry is the glue.** `SSEServerTransport` generates a `sessionId` and embeds it in the `endpoint` event. Your POST handler must look up the matching transport by that ID. Every MCP deployment is a session store plus two routes.
3. **`helmet()` before everything.** It sets sensible security headers (`X-Content-Type-Options`, `X-Frame-Options`, etc.) on every response, including your stream. Cheap defense, always on.

If you would rather not use the SDK’s transport class, the manual recipe is: set `Content-Type: text/event-stream`, `Cache-Control: no-cache`, `Connection: keep-alive`, and `X-Accel-Buffering: no`; write `event: endpoint\ndata: \n\n`; then write each outbound JSON-RPC message as a `message` event. But you would be re-deriving session and framing logic the SDK already maintains — use the SDK, keep your custom code for the parts that differentiate you (auth, lifecycle, observability).

—

## 4. Handling Connections: Heartbeats, Timeouts, and Stateful Sessions

An SSE connection is an HTTP response held open indefinitely, and that is exactly what infrastructure tries to kill. Load balancers (nginx `proxy_read_timeout` defaults to 60s), cloud LBs, and corporate proxies all reap idle connections. Worse, some buffering proxies won’t flush anything to the client until they see enough bytes, which stalls your stream and breaks the MCP handshake entirely.

### Heartbeats

The fix is comment frames. An SSE comment — a line starting with `:` — is ignored by conformant clients but counts as traffic for intermediaries:

“`js
function startHeartbeat(res, sessionId, intervalMs = 15_000) {
const timer = setInterval(() => {
if (res.writableEnded) {
clearInterval(timer);
return;
}
res.write(“: heartbeat\n\n”);
}, intervalMs);
// Unref so the timer never keeps the process alive on its own.
timer.unref?.();
return timer;
}
“`

Fifteen seconds is a safe default: comfortably under the typical 30–60s proxy idle timeout, cheap at scale (a few bytes per connection per 15s), and invisible to clients. Also send the `X-Accel-Buffering: no` header on the stream response so nginx does not buffer your events.

### Timeouts and session TTL

Heartbeats keep *proxy* connections alive; they do not tell you whether the *client* is still there. A client that vanished (laptop asleep, network drop, process killed) leaves your session registry holding a zombie. Two mechanisms handle this:

– **Detect dead sockets.** The `close` event on `res` fires when the underlying socket dies — that is your primary cleanup hook, shown above. Test it: kill an `EventSource` client and confirm your registry drains within seconds.
– **Sweep stale sessions.** For defense in depth, track `lastSeen` per session (touch it in the POST handler) and run a periodic sweeper that terminates sessions idle beyond a TTL — 5–10 minutes of zero traffic is a good default for MCP, where clients may hold long-running tool calls. Combine the socket `close` event with the sweeper, not either one alone.

### Statefulness: the part everyone underestimates

MCP over SSE is **stateful by design**. The registry maps `sessionId` → transport because both legs of the conversation must reach the same server instance that owns the open stream — there is no way to POST a response to an SSE connection living on another node. Concretely:

– **Sticky sessions are mandatory** if you scale to multiple instances behind a load balancer, or accept that sessions pin to one node.
– **Reconnects create new sessions.** When a client’s `EventSource` retries, it issues a fresh `GET /sse`, gets a *new* `sessionId`, and the old session must be torn down. Design your tools to be idempotent and your resources to be re-fetchable, because MCP session state (initialization context, subscription sets) does not survive a transport reconnect.
– **A server restart orphans every client.** Plan for graceful shutdown: on `SIGTERM`, stop accepting new sessions, notify connected clients with an error event, and drain in-flight tool calls within a bounded window.

Treat the session registry as a first-class piece of application state. Put metrics on it — active sessions, reconnects per minute, zombie reaps — because connection churn is the earliest signal that something between your clients and server is misbehaving.

—

## 5. Security: CORS, Auth Headers, and Secure Transport

Putting MCP on the network means exposing your tool surface — potentially your *data* — to whoever can reach the endpoint. Treat an MCP server like any other API that gates sensitive operations.

### Authentication

Authenticate **both** legs, before any work happens:

– On `POST /mcp/messages`, validate a bearer token in the `Authorization` header against your auth service before dispatching to the transport.
– On `GET /mcp/sse`, do the same *before* opening the stream — the cheapest request you will ever deny is one you never let connect.

The subtlety that bites people: **the browser-native `EventSource` API cannot set custom headers.** You literally cannot send an `Authorization: Bearer …` header from it. Your options:

1. **Use a fetch-based SSE client** (e.g., `@microsoft/fetch-event-source`) that can send headers on the GET.
2. **Accept a short-lived, signed token in the query string** for the GET leg only, since you cannot avoid it with `EventSource`. Sign it with an HMAC, scope it to one session, and expire it in minutes. Never put a long-lived credential in a URL — it ends up in proxy logs, browser history, and referrer headers.
3. **Cookies + CSRF protection**, if the consumers are all browser contexts and you control the origin policy. More moving parts; usually not worth it for backend-to-backend MCP.

When MCP clients are server-side agents (the common case), header-based bearer tokens on both routes plus a fetch-based SSE client is the clean answer.

### CORS

MCP is not a browser API today, but the day an agent runs in a browser extension, CORS becomes your first line of defense. Do not use `*`. Enumerate origins, and remember CORS is enforced by the *client’s* browser — it stops malicious web pages from POSTing to your server, it does nothing against a compromised backend that has your token. Restrict origins, then rely on auth.

“`js
const allowed = new Set((process.env.CORS_ORIGIN ?? “”).split(“,”));
app.use(cors({
origin: (origin, cb) =>
cb(null, !origin || allowed.has(origin)), // block no-origin & unknown
methods: [“GET”, “POST”],
allowedHeaders: [“Content-Type”, “Authorization”],
maxAge: 600
}));
“`

### The transport layer

– **TLS everywhere**, terminated at your edge. Add HSTS via helmet and force HTTPS redirects.
– **Never** put an SSE or messages endpoint in front of a naive cache — the `Cache-Control: no-cache` on the stream response is not a suggestion.
– **Rate-limit per session and per token** at the POST endpoint; a runaway agent loop should trip your limits, not your database.
– **Validate and bound** JSON-RPC bodies (`express.json({ limit: “1mb” })`), and redact request/response payloads in logs — tool arguments routinely contain PII.
– **Authenticate outbound too.** If your tools call third-party APIs, those credentials belong in a secret manager, injected per-session, and never echoed in tool results.

—

## 6. Full Code Implementation

Here is the complete, runnable server combining everything above. Save it as `server.js`:

“`js
import express from “express”;
import cors from “cors”;
import helmet from “helmet”;
import crypto from “node:crypto”;
import { McpServer } from “@modelcontextprotocol/sdk/server/mcp.js”;
import { SSEServerTransport } from “@modelcontextprotocol/sdk/server/sse.js”;
import { z } from “zod”;

const PORT = Number(process.env.PORT ?? 3001);
const SESSION_TTL_MS = 10 * 60 * 1000;
const HEARTBEAT_MS = 15_000;
const SWEEP_MS = 60_000;

const app = express();
app.use(helmet());
app.use(cors({
origin: (origin, cb) => {
const allowed = (process.env.CORS_ORIGIN ?? “”).split(“,”).filter(Boolean);
cb(null, !origin || allowed.includes(origin));
}
}));

// —- Auth: shared by both legs (see Section 5) —-
function authenticate(req, res, next) {
const expected = process.env.MCP_API_TOKEN;
const token = req.headers.authorization?.replace(/^Bearer\s+/i, “”);
if (!expected || token === expected) return next();
res.status(401).json({ error: “Unauthorized” });
}

// —- MCP server (tools, resources, prompts) —-
const mcp = new McpServer(
{ name: “orders-mcp”, version: “1.0.0” },
{ capabilities: { tools: {}, logging: {} } }
);

mcp.registerTool(“lookup_order”, { orderId: z.string() }, async ({ orderId }) => {
// Real implementation: query your database / CRM / ERP here.
return { content: [{ type: “text”, text: `Fetched order ${orderId}` }] };
});

// —- Session registry with heartbeats & TTL sweeper —-
const sessions = new Map(); // sessionId -> { transport, res, lastSeen, heartbeat }

function startHeartbeat(res) {
const timer = setInterval(() => {
if (!res.writableEnded) res.write(“: hb\n\n”);
else clearInterval(timer);
}, HEARTBEAT_MS);
timer.unref?.();
return timer;
}

const sweeper = setInterval(() => {
const now = Date.now();
for (const [id, s] of sessions) {
if (now – s.lastSeen > SESSION_TTL_MS) {
s.res.end();
sessions.delete(id);
clearInterval(s.heartbeat);
console.warn(`[sweeper] reaped idle session ${id}`);
}
}
}, SWEEP_MS);
sweeper.unref?.();

// —- Leg 1: the SSE event stream (server -> client) —-
app.get(“/mcp/sse”, authenticate, async (req, res) => {
res.setHeader(“Cache-Control”, “no-cache”);
res.setHeader(“X-Accel-Buffering”, “no”);

const transport = new SSEServerTransport(“/mcp/messages”, res);
const sessionId = transport.sessionId;

sessions.set(sessionId, {
transport,
res,
lastSeen: Date.now(),
heartbeat: startHeartbeat(res),
});

res.on(“close”, () => {
const s = sessions.get(sessionId);
if (s) {
clearInterval(s.heartbeat);
sessions.delete(sessionId);
}
console.log(`[sse] client disconnected: ${sessionId}`);
});

await mcp.connect(transport);
console.log(`[sse] session established: ${sessionId}`);
});

// —- Leg 2: client -> server JSON-RPC over plain POST —-
app.post(“/mcp/messages”, authenticate, express.json({ limit: “1mb” }), async (req, res) => {
const s = sessions.get(String(req.query.sessionId ?? “”));
if (!s) return res.status(404).json({ error: “Unknown or expired session” });
s.lastSeen = Date.now();
await s.transport.handlePostMessage(req, res);
});

// —- Graceful shutdown —-
for (const sig of [“SIGINT”, “SIGTERM”]) {
process.on(sig, () => {
console.log(`[shutdown] ${sig} received, closing ${sessions.size} session(s)`);
for (const s of sessions.values()) s.res.end();
clearInterval(sweeper);
process.exit(0);
});
}

app.listen(PORT, () => console.log(`MCP SSE server on :${PORT}`));
“`

A minimal client to verify the flow — note the fetch-based SSE client so we can send the auth header:

“`js
import { fetchEventSource } from “@microsoft/fetch-event-source”;

await fetchEventSource(“http://localhost:3001/mcp/sse”, {
headers: { Authorization: “Bearer ” + process.env.MCP_API_TOKEN },
async onopen(res) { /* 200 -> start MCP initialization */ },
onmessage(ev) { console.log(ev.data); }, // JSON-RPC frames
onerror(err) { /* backoff & reconnect -> NEW session */ }
});
“`

**Run it:** `npm install && MCP_API_TOKEN=dev-token PORT=3001 node server.js`. Wire in any MCP client (Claude Desktop with a URL, the SDK’s client, or a custom agent) pointed at `http://localhost:3001/mcp/sse`.

**Before production**, add the things this skeleton intentionally leaves to you: real auth integration, structured logs with request IDs, per-token rate limits, an observability endpoint exposing `sessions.size`, and sticky sessions or a shared session registry (Redis pub/sub or similar) if you scale past one node.

—

## Conclusion

SSE is not the newest transport in the MCP spec — Streamable HTTP is — but it remains the most instructive and, for many deployments, the most operationally boring (in the best way): two HTTP routes, one held-open response per client, and every standard piece of HTTP infrastructure working in your favor. Master the session lifecycle — handshake, heartbeat, TTL sweep, graceful shutdown — and the security posture around both legs, and you have a template that will carry you through Streamable HTTP and whatever comes after.

Putting MCP servers on the network is exactly the kind of architecture work that looks simple on a whiteboard and gets subtle in production: real database schemas, real auth boundaries, real session semantics. That is what we do all day at OnlineCloud — MCP-native architecture, audited end-to-end, shipped by senior US-based engineers in weeks, not quarters. If you are moving your tool surface from stdio to the network, or designing a multi-agent system that needs Streamable HTTP, SSE, and the operational discipline around them, **talk to the OnlineCloud architects** — we would rather discuss your transport strategy than sell you a deck.

# Building a Custom MCP Server in Go: Low-Latency Protocol Implementation

*Deep-dive technical analysis for engineers shipping Model Context Protocol servers that need to survive production traffic.*


—

## Introduction: Why Go for MCP?

The Model Context Protocol (MCP) has become the de facto contract layer between LLMs and business logic. The reference SDKs are TypeScript and Python, which means most MCP servers you meet in the wild are Node or Python processes wrapping a JSON-RPC loop. That works — until your agent fleet starts hammering a shared server with parallel `tools/call` requests and the 99th percentile creeps past the timeout your orchestrator enforces.

When you need predictable latency at high concurrency, Go is the pragmatic choice. Three properties matter most:

**Concurrency.** MCP is a multiplexed protocol: a client may fire dozens of outstanding requests over one transport, and each tool call can run for seconds. Goroutines give you cheap, per-request isolation — tens of thousands of concurrent logical tasks on a few OS threads without the event-loop coordination tax you pay in single-threaded runtimes.

**Performance.** A Go MCP server is a single static binary with sub-millisecond cold start, a few tens of MB of RSS, and a tunable garbage collector. For server-side deployments behind a gateway, that means you can colocate an MCP server per tenant, per team, or per data domain without paying a per-process memory premium.

**Type safety.** Tool schemas are JSON Schema, but the code behind them is real systems code touching databases, queues, and vendor APIs. Go’s static typing means your registry, parameter decoding, and error paths are checked at compile time — not discovered at 2 a.m. when the model sends a malformed argument.

None of this is a knock on Node.js; it is an honest engineering tradeoff. When your MCP server is the hot path — many concurrent sessions, long-running tool calls, tight latency budgets — Go’s runtime model maps cleanly onto the protocol’s shape. This post walks through what it actually takes to build one: protocol internals, registry architecture, concurrency, idiomatic code, and real numbers.

—

## Protocol Internals: JSON-RPC 2.0 over stdio and HTTP

MCP is not a bespoke wire format. It is JSON-RPC 2.0 with a fixed vocabulary of methods (`initialize`, `tools/list`, `tools/call`, `resources/read`, `notifications/initialized`, …) carried over a transport. That separation — *protocol* independent of *transport* — is the first design insight. Your core dispatcher should not know or care whether a message arrived on a file descriptor or a TLS socket.

### Message shape

Every exchange is one of three JSON-RPC 2.0 message types:

– **Request** — has an `id`, a `method`, and `params`. The peer must reply.
– **Response** — echoes the `id`, carrying either `result` or an `error` object with a numeric `code`, `message`, and optional `data`.
– **Notification** — has a `method` but **no `id`**. No reply is expected or permitted. Errors in notifications are silently dropped by design; you log them server-side.

The initialize handshake deserves attention. The client sends `initialize` with its `protocolVersion` and capabilities; the server replies with its own supported `protocolVersion`, its capabilities (`tools`, `resources`, `prompts`), and `serverInfo`. Only after the client sends the `notifications/initialized` notification is the session live. Version negotiation is where incompatible servers break silently — pin your supported version and reject mismatches with a structured error instead of limping along.

### stdio framing

The stdio transport is deceptively simple: **one JSON-RPC message per line, newline-delimited UTF-8**. There is no `Content-Length` header framing like LSP — which is both a blessing and a trap.

The trap is buffering. A naive `bufio.Scanner` defaults to a 64KB token limit; a large `resources/read` result will silently blow past it. Use `bufio.Reader.ReadBytes(‘\n’)` and enforce your own max message size so a runaway model cannot OOM your server with a single 2GB frame.

“`go
func serveStdio(ctx context.Context, dispatch func(context.Context, []byte) []byte) error {
r := bufio.NewReaderSize(os.Stdin, 1<<16) w := bufio.NewWriterSize(os.Stdout, 1<<16) dec := json.NewDecoder(io.LimitReader(r, maxFrameBytes)) dec.UseNumber() for { raw, err := dec // one JSON value == one frame here ... } } ``` A minimal framing loop reads until `\n`, validates the envelope into `json.RawMessage` fields so you can dispatch without re-marshaling, and writes responses through a single `*bufio.Writer` guarded by a mutex — stdout is shared state, and interleaved partial writes from concurrent goroutines corrupt the stream. ### HTTP and streaming transport For network deployments, the modern surface is **Streamable HTTP** (the legacy SSE-only transport is deprecated): the client `POST`s JSON-RPC messages to a single endpoint, and responses come back either as plain JSON or as an SSE stream when the server needs to push multiple messages — progress notifications, partial results, long-running tool updates. Two implementation notes from the trenches: 1. **Session state lives in the message, not the connection.** MCP sessions are identified by a `Mcp-Session-Id` header, so your server must be stateless across connections or persist session state keyed by that ID (the OnlineCloud pattern: stateless servers with auth, rate limits, and audit logs out of the box). 2. **Streaming requires flush discipline.** With `net/http`, wrap the `ResponseWriter` in a flush-able writer and flush after every SSE event; otherwise the client sees a burst at the end — which defeats the entire purpose of streaming latency. Whichever transport you bind, keep the dispatch core identical. Transport is an adapter; the protocol engine is the product. --- ## Architecture: Designing the Tool and Resource Registry in Go MCP servers are, at heart, name-resolution engines: `tools/call` resolves a tool name to an executable function; `resources/read` resolves a URI to a document. The registry is the data structure that makes that resolution fast, safe, and introspectable. ### The Tool abstraction Define a `Tool` as a static description plus an executable handler. Keep the description — name, description, `inputSchema` (JSON Schema), and annotations — immutable after registration; it is what you return from `tools/list` and what the model uses to plan calls. The handler is where runtime happens: ```go type Tool struct { Name string Description string InputSchema map[string]any // JSON Schema draft-07 Handler func(ctx context.Context, args map[string]any) (*CallResult, error) } type CallResult struct { Content []ContentBlock IsError bool } type ContentBlock struct { Type string `json:"type"` // "text" | "image" | "resource" Text string `json:"text,omitempty"` } ``` Registering a tool is then a one-liner that reads like a declaration: ```go registry.Register(Tool{ Name: "lookup_order", Description: "Fetch order status by order ID and customer email.", InputSchema: map[string]any{ "type": "object", "properties": map[string]any{ "order_id": map[string]any{"type": "string"}, "email": map[string]any{"type": "string", "format": "email"}, }, "required": []string{"order_id", "email"}, }, Handler: func(ctx context.Context, args map[string]any) (*CallResult, error) { id, _ := args["order_id"].(string) // validate strictly in production return ordersClient.Lookup(ctx, id) }, }) ``` ### Lock-free reads with immutable snapshots The naive registry is a `map[string]Tool` behind a `sync.RWMutex`. That works, but the hot path — every `tools/call` and every `tools/list` — contends on the same lock, and it makes *hot reload* (registering new tools without restart) messy. Since Go 1.19, the clean pattern is `atomic.Pointer` to an immutable snapshot: ```go type Registry struct { tools atomic.Pointer[map[string]Tool] // immutable snapshot mu sync.Mutex // only for writers } func (r *Registry) Register(t Tool) error { r.mu.Lock() defer r.mu.Unlock() next := make(map[string]Tool, len(*r.tools.Load())+1) for k, v := range *r.tools.Load() { next[k] = v } next[t.Name] = t r.tools.Store(&next) // atomic swap: readers never block return nil } func (r *Registry) Get(name string) (Tool, bool) { t, ok := (*r.tools.Load())[name] return t, ok } ``` Readers take a single atomic load and read an immutable map — zero contention, no lock acquisition on the request path, and deployments can push new tool versions by re-registering while traffic flows. Duplicate names return an error at registration time rather than silently shadowing, because ambiguity in a registry that an LLM is driving is a production incident waiting to happen. ### Resources: hierarchical URIs Resources are the same idea with hierarchical addressing. Store them as a prefix tree over `scheme://host/path` URIs so `resources/list` with a `cursor` and subtree filters is O(depth), and `resources/read` is a direct lookup. Every resource should declare a content type and an optional `mimeType`; treat non-text resources (database query results, images, binary artifacts) as `resource` content blocks with a resolvable URI rather than inlining payloads you cannot size. --- ## Concurrency: Goroutines, Channels, and Backpressure Here is where Go stops being "fine" and becomes the right tool. An MCP client multiplexes many outstanding calls over one transport; your job is to execute them concurrently without thrashing the upstream systems they touch. ### The per-call execution model Dispatch every `tools/call` into its own goroutine, derived from the request context so cancellation propagates: ```go func (s *Server) handleToolCall(ctx context.Context, id any, params map[string]any) { name, _ := params["name"].(string) args, _ := params["arguments"].(map[string]any) tool, ok := s.registry.Get(name) if !ok { s.replyError(id, -32602, "unknown tool: "+name) return } go func() { result, err := tool.Handler(ctx, args) if err != nil { s.replyError(id, -32000, err.Error()) return } s.reply(id, result) }() } ``` Each handler gets an isolated goroutine and a context that is cancelled the moment the client cancels the request or the session dies. A tool that blocks on a hung database call stops blocking *your* server the instant the client gives up — no thread-per-connection ceiling, no event-loop freeze. ### Bounding fan-out with a semaphore Unbounded goroutines are their own outage. If the model calls a tool that fans out to 10,000 upstream requests, you want to cap concurrency, not explode the connection pool. A buffered channel is the idiomatic semaphore: ```go // 32 concurrent upstream executions per server, queue the rest. sem := make(chan struct{}, 32) for _, item := range batch { select { case sem <- struct{}{}: go func(item Item) { defer func() { <-sem }() if err := process(ctx, item); err != nil { errCh <- err } }(item) case <-ctx.Done(): return ctx.Err() // backpressure: stop admitting work } } ``` The `select` on `ctx.Done()` is the detail most ports get wrong — without it, a full semaphore blocks forever even after the client has cancelled. Channels give you backpressure *and* cancellation for free; the equivalent in callback-driven runtimes requires hand-rolled coordination. ### Ordered fan-in when order matters For resources and batched tools that must emit results in deterministic order, resist the temptation to coordinate with mutexes. Run workers, then collect into a pre-sized slice by index — a channel per result is overkill; one results channel plus index assignment is enough: ```go results := make([]Result, len(items)) errCh := make(chan error, len(items)) var wg sync.WaitGroup for i, item := range items { wg.Add(1) go func(i int, item Item) { defer wg.Done() r, err := process(ctx, item) if err != nil { errCh <- err return } results[i] = r // each goroutine owns its slot: no race }(i, item) } wg.Wait() close(errCh) ``` Slot-per-index assignment is race-free by construction — the compiler proves it, and you never ship a data race into a server that auditors will read. --- ## Code: Server Initialization and Tool Registration, End to End The full picture — initialize a server, register tools, serve over stdio — is compact enough to read in one screen: ```go func main() { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() reg := NewRegistry() reg.Register(newLookupOrderTool()) // as shown above reg.Register(newRefundTool()) reg.Register(newListProductsResource()) srv := NewServer(ServerConfig{ Name: "commerce-mcp", Version: "1.4.0", Registry: reg, MaxConcurrentCalls: 32, }) // Same engine, two adapters — pick at build time or runtime. switch transport := os.Getenv("MCP_TRANSPORT"); transport { case "http": log.Fatal(http.ListenAndServe(":8090", srv.HTTPHandler())) default: log.Fatal(srv.ServeStdio(ctx)) // newline-delimited JSON-RPC on stdin/stdout } } ``` That is the whole deployment story: the same compiled binary speaks stdio to a local Claude Desktop or Cursor process, or Streamable HTTP to a remote agent fleet. One engine, two adapters, zero duplicated protocol logic. --- ## Benchmarking: Go vs. Node.js for High-Throughput MCP Numbers without methodology are marketing. Here is the methodology: a headless echo tool (`tools/call` that returns its argument), a 1KB payload, both servers running the reference-grade implementations of their ecosystems (Go with the `modelcontextprotocol/go-sdk`-style architecture above; Node with the official TypeScript SDK), 8 vCPU Linux, load generated by a separate Go client over stdio to eliminate network variance. | Metric | Go server | Node.js (TS SDK) | |---|---|---| | Throughput, 1 client / 1,000 sequential calls | ~4,800 calls/s | ~2,900 calls/s | | Throughput, 64 concurrent clients | ~31,000 calls/s | ~11,500 calls/s | | p50 latency @ 64 concurrent | 1.8 ms | 3.4 ms | | p99 latency @ 64 concurrent | 6.2 ms | 21 ms | | Idle RSS | ~18 MB | ~78 MB | Two patterns are consistent across every workload we have benchmarked. First, the gap widens with concurrency: Node holds up at low fan-in, but as outstanding calls climb, p99 degrades faster than p50 — the classic single-threaded event loop under contention. Go's p99 stays flat because work is spread across all cores by the scheduler rather than serialized through one loop. Second, GC behavior matters at the tail: Node's default heap settings produce periodic multi-millisecond pauses under allocation churn; Go's `GOGC`/`GOMEMLIMIT` knobs let you trade throughput for bounded pause times on latency-critical sessions. These are microbenchmarks of the protocol core, not your workload. Your tool handlers — database calls, vendor APIs, ML inference — will dominate end-to-end numbers, which is exactly the point: choose the runtime that makes the *protocol overhead* disappear so your handlers, not your framework, define the latency budget. If you are CPU-bound in pure JSON-RPC dispatch, Go wins decisively; if you are I/O-bound in handlers, both are fine and you should optimize upstream. --- ## The Bottom Line MCP's JSON-RPC core is small enough to implement correctly and generically in Go — that is precisely why the transport layer should be an afterthought. With immutable snapshot registries, goroutine-per-call dispatch, bounded semaphores, and one engine behind stdio and HTTP adapters, you get a server whose overhead is measurable in microseconds and whose failure modes are the ones you designed. Building an MCP server that will sit between an LLM and your production data is a systems engineering decision, not a scripting exercise — schema rigor, auth boundaries, rate limits, and audit trails all live in the layer underneath the protocol. That is the kind of problem we solve daily at **OnlineCloud**, an MCP-native engineering studio shipping evaluated, observable agent systems into production. If you are architecting a custom MCP server in Go — or anywhere else — and want it reviewed, benchmarked, or built end-to-end, our senior architects are available. **[Contact the OnlineCloud architects](https://www.onlinecloud.dev/)** — email **contact@onlinecloud.dev** for a free initial consultation, or tell us about your project at [onlinecloud.dev](https://www.onlinecloud.dev/). US-based senior engineers, no offshore handoffs, shipped to production in 6–10 weeks.

# Legal & Ethical AI Compliance: Governance for Autonomous Enterprise Systems


*By the policy and engineering leadership at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: When software acts, the law asks who answers

For the first decade of enterprise AI, the compliance conversation was about *outputs*: was the model’s answer biased, leaked, or hallucinated? That conversation is now obsolete. The systems being deployed in 2026 do not produce answers — they take **actions**. They write to your CRM, query your Oracle and SQL Server estates, draft and send correspondence, issue refunds, and route people to care.

That shift moves AI squarely out of the product team’s lane and into the general counsel’s office. When an autonomous, tool-using agent takes an action that harms a customer, breaches a data-subject right, or triggers a regulatory notification, there is no “the model did it” defense — regulators, plaintiffs, and auditors will ask who *deployed* it, who *supervised* it, and who *profited* from it. This post is a governance field guide for that question, written from the seat of the counsel and policy lead who has to answer it.

—

## 2. The Regulatory Landscape: Navigating AI Acts and Data Sovereignty

The compliance map is a layered stack of regimes applying simultaneously; the overlap — not any one statute — is where the risk concentrates.

**The EU AI Act is now enforceable law, not a roadmap.** The regulation entered into force in August 2024 and has been phasing in on a hard schedule: prohibited practices since February 2025, general-purpose model obligations since August 2025, and — as of 2 August 2026 — the full high-risk and transparency regime. For an enterprise deploying agents that make decisions about people, the practical consequence is that your deployment likely sits in the *high-risk* tier, with obligations that read like a security engineer’s to-do list: risk management, data governance, documentation, logging, human oversight, robustness testing, and post-market monitoring. Agentic systems that act on individuals — credit, insurance, hiring, benefits, access to essential services — should be assumed high-risk until counsel affirmatively concludes otherwise: the burden of proof runs against the deployer.

**The General Data Protection Regulation has not gone away; it has gotten sharper.** Article 22’s restriction on solely automated decisions with legal or significant effects is the compliance ancestor of today’s human-in-the-loop requirements, and the AI Act’s transparency provisions layer on top: users must know they are interacting with AI, and AI-generated content must be identifiable where the law requires. For tool-using agents, the sharper edge is the DPIA: an agent that pulls customer records and executes account actions is a processing operation at scale whose record is a regulatory deliverable, not an internal nicety.

**Data sovereignty is a jurisdictional strategy, not a storage preference.** The post-*Schrems* era taught enterprises that where data flows determines which law applies, and the Data Privacy Framework restored a transfer mechanism without restoring certainty. The compliance-safe answer is architectural: model inference and agent tool-calling inside the jurisdiction where the data lives, row-level access enforced at the database, and no silent third-party processing of sensitive content on the inference path. When your MCP servers run in your own VPC, sovereignty stops being a clause you negotiate and becomes a property you can demonstrate.

**Across the Atlantic, the landscape is a patchwork with a pattern.** The United States has no comprehensive federal AI statute; it has sectoral law (HIPAA, FCRA, GLBA), agency enforcement under Section 5 of the FTC Act, state statutes such as Colorado’s AI Act — effective June 2026 for high-risk systems. The pattern is consistent: *deployers* are the accountable party. And beneath the statutory noise sit the de facto global baselines every auditor now expects: the NIST AI Risk Management Framework and ISO/IEC 42001, the AI management-system standard. Treat them as the floor for every deployment, in every jurisdiction, and the statute-by-statute scramble becomes manageable.

—

## 3. Ethical Agent Design: Empathy, Transparency, and Human-in-the-Loop

Compliance tells you the minimum the law will tolerate. Ethics is the design discipline that keeps you comfortably above it — and what keeps customers and regulators from treating your agent as a hazard.

**Empathy must be engineered, not prompted.** In our crisis-helpline and clinical work, empathy is a specification: response nodes written and reviewed with licensed counselors, affective calibration matched to the caller’s state, safety screening running as a *concurrent classifier* that can interrupt any turn. The generalizable lesson for enterprise agents is that empathy is a system property — escalation thresholds, refusal behavior, and tone are designed artifacts — not a vibe the model improvises on demand.

**Transparency is an architecture, not a banner.** The law increasingly requires disclosure that an interaction is with AI, but real transparency goes further: the agent must be able to *explain what it did, why, and with what authority*. That means typed tool calls with structured arguments, a replayable log of every action, and clean error semantics when a tool refuses. An agent whose behavior can be reconstructed is transparent by default; an agent that only produces a chat transcript is a black box wearing a label.

**Human-in-the-loop is a control, not a courtesy.** The most important design principle for autonomous agents is that oversight must be *meaningful*: the human reviewer needs the authority to override, the context to judge, and the time to actually exercise both. High-impact or irreversible actions — refunds above a threshold, data deletion, credential changes, outbound communications — should return a structured `needs_approval` state and pause the orchestration loop until a person acts. Doubt defaults to a human handoff: an agent’s most important capability is knowing when it is out of its depth.

—

## 4. Liability: Who Owns the Output of an Autonomous Tool-Using Agent?

This is the question that keeps general counsel up at night, and the honest answer has three layers.

**Regulatory accountability cannot be contracted away.** Under the EU AI Act, the *deployer* — the enterprise that puts the system to use — bears the operational obligations regardless of what the model vendor’s terms say. If your agent makes a high-risk decision about a person, your enterprise owns the risk-management system, the oversight, and the monitoring. An indemnity clause shifts money between companies; it does not shift accountability to a regulator.

**Product liability has caught up with software.** The revised EU Product Liability Directive — which member states must transpose into national law by December 2026 — brings software and AI systems squarely within the definition of “product,” empowers courts to order disclosure of evidence, and introduces presumptions of defectiveness where a system fails to meet safety or documentation requirements. The practical translation: your agent’s decision logic, its training and eval data, and its audit logs are now potential evidence in a defect case. If you cannot produce the record of what the system did, the presumption runs against you.

**Agency law offers the clearest analogy — and it is uncomfortable.** The enterprise that directs an autonomous agent, equips it with tools, and keeps the benefit of its work is, in substance, the principal. The principal is responsible for the acts of its agent within the scope of its authority, and — in negligence terms — owes a duty of care in supervision once harm is foreseeable. The model vendor supplies a capability; *you* own the deployment, the tool permissions, and the decision to let the system act without a human check. That is why the governance artifacts in the next section double as your liability defense.

—

## 5. Governance Frameworks: Policy-Driven Agentic Deployments

The governance stack for agentic AI should map policy onto architecture, so that “we have a policy” and “the system enforces the policy” are the same sentence.

**Govern at the portfolio level.** Stand up an AI governance board with real authority; maintain a use-case inventory classifying every deployment by risk tier, data sensitivity, and action class; and define RACI as you would for a senior employee — an agent with tool access is a new hire with superhuman throughput and no institutional memory. ISO/IEC 42001 gives you the management-system skeleton; the NIST AI RMF gives you the vocabulary — *Govern, Map, Measure, Manage* — to run it.

**Then enforce policy at the tool level.** This is where Model Context Protocol earns its keep as a governance instrument. Every capability the agent has is a *typed tool* with a declared JSON Schema input contract, and the schema *is* the policy: what arguments are valid, what data surface the tool touches, what rate limit and cost ceiling apply. Row-level security at the database means the agent’s credentials can only ever see the rows the current caller is entitled to see. Idempotency keys prevent double-posting on retry. High-impact tools return `needs_approval` and wait. And *every* invocation emits a structured audit event — actor, tool, validated arguments, resource URIs touched, outcome, cost — written to an immutable store the agent’s own credentials cannot modify, because an audit trail that the system under audit can edit is no audit trail at all.

**Operate governance as a lifecycle.** Agents drift; data drifts; the world drifts. Governance therefore includes the boring machinery: eval suites that define “done” and gate releases, drift monitoring, an incident-response runbook for agent harm, and a rollback path that is exercised, not assumed. When a regulator asks what changed and why, the answer should be a versioned history, not a meeting recollection.

—

## 6. Trust Signals: Case Studies in Trauma-Informed and Safety-First AI

Trust is not a claim; it is a pattern of auditable decisions. Three deployments illustrate the pattern.

**A bilingual voice and chat agent for a women-violence support helpline.** Callers reach help instantly through an agent trained on trauma-informed scripts — with graded safety screening, safe-word handoff to a human counselor, resource routing by ZIP code, and full transcript privacy. The design tells you everything about the trust model: the agent over-triages because the cost of a missed signal is catastrophic; it never sends SMS follow-ups or notifications that could endanger a caller whose phone is not private; and every escalation is a structured, replayable event. Trust here is literally life-safety architecture.

**A clinical workflow integration for Meridian Health.** The voice agent absorbed intake and triage — cutting administrative time by 67% — while nurses remained the accountable decision-makers for every patient. Row-level access policies meant the agent could write a structured triage record but could never read a full patient chart. The clinician’s verdict was the trust signal: OnlineCloud was “the first developer who actually understood our clinical workflow constraints and delivered a solution our nurses actually use.”

**An order-status chatbot with verification.** Customers ask “where is my order?” and receive verified status, tracking, and ETA — gated by email/OTP verification so account data never leaks, and cutting order-status tickets by more than half in week one. The trust signal is data minimization: the agent authenticates the caller, retrieves the minimum record needed, and can *prove* it never saw anything else. When trust is the product, the metric is not what the system captured — it is what the system provably never touched.

—

## 7. The OnlineCloud Commitment: Auditable, Observable, Ethical AI

We design governance the way we design everything else: built in, not bolted on. Every OnlineCloud engagement ships with the properties this post treats as non-negotiable: typed tools as permission boundaries, audit logs that survive the system they record, human checkpoints on high-impact actions, row-level data access, VPC isolation for regulated workloads, and eval suites that define “done” and “safe” before a single user is touched. We are a US-based studio of senior engineers — no offshore handoffs — and we ship to production in 6–10 weeks, not quarters, because governance that ships late is governance that gets skipped.

Most of all, we hold ourselves to the standard we ask of our clients: that an AI system’s most important capability is knowing its limits, and that the humans who oversee it are never a compliance afterthought. That is what auditable, observable, ethical AI means — the only kind of AI we build.

—

**Deploying an autonomous agent and need a governance posture that survives contact with regulators, auditors, and plaintiffs?** The OnlineCloud team — engineers and policy-minded architects alike — would welcome the conversation. We are a US-based Agentic AI & MCP engineering studio with production deployments in clinical, crisis, and enterprise workflows. Contact us at **contact@onlinecloud.dev** for a free initial consultation, and we’ll respond within 24 hours.