Blog

# Designing a Self-Healing Agent Loop: Automated Error Recovery and Retries

*When the code path is generated per-run by a stochastic model, your retry policy can’t be a static snippet. Here is how an SRE makes an agent classify failures, recover from them, and stop amplifying outages.*


## 1. Introduction: The Fragility of Autonomous Agents in a Brittle API World

Every engineer who has run an agent against real APIs knows the death scene: the tool times out, and the agent calls it again with the same arguments, again and again, while the upstream stays down. Nobody wrote a retry loop; the model *became* it.

A deterministic service survives production entropy because its error handling was written once, in code: backoff, jitter, idempotency keys, a circuit breaker. An agent has none of that by default. Its “error handling” is improvised at runtime, under token pressure, by a model that treats every failure as a puzzle rather than a signal to stop. **In an agentic system, the retry policy is generated per-run by the thing that is already failing.**

Production APIs punish this: p99 spikes, 429 rate limits, 5xx during deploys, 2 a.m. maintenance resets. The failure shapes are consistent: the **retry spiral** (identical calls until a ceiling or a human intervenes), the **confident drift** (after two errors the model stops calling tools and invents answers), and the **cascade** (one poisoned result invalidates every downstream step). These failures are behavioral, not technical — the system returns success while doing the wrong thing, or quietly burns budget.

The fix is not to prompt the model to “be more careful.” It is to move error handling out of the context window and into the execution machinery around the model: **classify the failure, pick a recovery strategy, and enforce limits the model cannot override.** That is the same architectural move the Model Context Protocol made for tooling: taking invariants out of prose and putting them into structure. A self-healing loop is what you get when the runtime cares about failure as much as the model cares about the task.

## 2. Error Classification: Model Errors vs. Transient API Failures

You cannot heal what you cannot classify — and you cannot classify with a model. Asking an LLM “what went wrong?” on a 500 costs tokens and produces prose instead of policy. **Classification belongs at the tool boundary, in the runtime**, where every tool wrapper returns a structured error envelope before the model ever sees the failure:

“`json
{
“error”: {
“class”: “transient”,
“code”: “RATE_LIMITED”,
“retryable”: true,
“retry_after_ms”: 30000,
“tool”: “crm.account.update”,
“hint”: “honor Retry-After; do not regenerate arguments”
}
}
“`

Three classes cover nearly every production failure, and each maps to one family of recovery:

– **Transient API failures** — 429s, 5xx, timeouts, resets. A retry *can* succeed: the request was fine, the infrastructure was not. Recovery is deterministic — exponential backoff with jitter, honor `Retry-After`, retry the identical payload under an idempotency key. Never ask the model to “improve” a request that was never the problem.
– **Contract failures** — 400/422 schema rejections, missing resources, expired auth. Also deterministic, in the opposite direction: retrying an identical payload is pure waste. Recovery means repairing the plan — fix the arguments, refresh the credential — then re-executing; a contract failure that recurs after repair is a stop-and-escalate signal.
– **Model errors** — wrong tool chosen, arguments that pass schema but violate business semantics, a plan that contradicts verified results. Not a network class; retrying is meaningless because the same model repeats the same decision. This is the expensive class, because detecting it needs a **verification step** — a read-only check that output matches the goal — not a status code.

Two rules keep it honest. Classify on code and envelope, never payload heuristics: a 500 with an `UPSTREAM_TIMEOUT` body is transient; a 200 with an empty result may be a model error. Treat the classifier as policy, not model judgment — version it with the tool schema, change it without shipping a prompt.

## 3. Recovery Patterns: Backtrack, Plan-Repair, and Tool-Switching

With errors classified, the executor dispatches to one of three patterns from a policy table. They are deliberately few: the runtime must reason about them, and the model must be constrained by them — let the model choose among *visible, eligible* actions; let the runtime decide what is visible and eligible.

**Backtrack — rewind, don’t replay.** Every completed step appends to a checkpoint: plan state, tool result, idempotency key. When an error invalidates the branch, the executor rewinds to the last validated checkpoint — you checkout a known-good state instead of performing surgery on a broken one. The SRE detail that makes this safe: **rewinding context does not unwind side effects.** An order created before the backtrack does not un-create itself. Every side-effecting tool must be idempotent (keyed by step) or compensated (a paired cancel call), or backtracking is just a fancier way to corrupt data.

**Plan-Repair — amend, don’t regenerate.** For contract failures and shallow model errors, the cheapest correct recovery is a surgical amendment of the *remaining* plan — fix the argument that failed validation, drop the step that contradicts verified results. Repair is bounded to two or three rounds with a token budget: an unbounded repair loop is the retry spiral in a nicer jacket. The model may only edit **unexecuted** steps (executed history is immutable, keeping the decision log replayable), and repaired steps pass the same schema and policy validation as the original.

**Tool-Switching — capability pluralism.** The same capability — update an account, enqueue a job — usually lives behind several tools: a synchronous REST tool, a queue-backed async variant, a read-only verification view. The loop needs a **fallback registry**: an ordered list of substitutes per tool, maintained by policy, not improvised mid-crisis. When the primary tool’s breaker is open or its retries are exhausted, the executor promotes the next tool in the registry. The model never “decides” to switch — the runtime simply filters the unavailable tool from its toolset. Switching becomes routing with deterministic preferability.

The dispatch loop tying the patterns together:

“`
while plan.has_steps() and run_budget.remaining():
result = executor.call(step.tool, step.args, idempotency_key=step.key)
if result.ok:
checkpoint(step, result); continue
error = classify(result)

if error.transient and step.attempts < max_retries: # backoff retry sleep(backoff(step.attempts, jitter=True)); step.attempts += 1; continue if breaker.is_open(step.tool): # tool-switching step.tool = policy.fallback_for(step.tool) if step.tool is None: escalate(error); break step.attempts = 0; continue if error.contract or error.model: # plan-repair plan, repaired = repair_plan(plan, error, repair_budget=2) if not repaired: # backtrack plan = last_valid_checkpoint().plan ``` The property that makes this self-healing rather than self-harming: **every path terminates** — retries have a max, repairs have a budget, backtracks land on validated state, and exhaustion ends in a structured escalation: a human handoff, never a silent success. ## 4. Circuit Breakers: Preventing Agentic Retry Storms on Downed Upstreams A single agent retrying a downed API is a nuisance; a fleet of them is an outage amplifier. When the upstream goes down, every concurrent run discovers it at once and starts its own retry campaign against a service that cannot recover under the load they generate. No individual agent misbehaves, yet the aggregate is a self-inflicted DDoS with a token bill attached: the **agentic retry storm**. The arithmetic is brutal. An upstream sheds load with 429s and a 30-second `Retry-After`; a disciplined client retries every 30 seconds, while a model improvising its own policy retries in two — fifteen times the request rate. Across a hundred concurrent runs, that is a 1,500× amplification of the exact load the outage was designed to shed. The breaker makes disciplined backoff an *enforced property of the runtime*, not a suggestion. Agentic breakers follow the classic tri-state pattern with specifics bolted on: - **Per-tool and per-upstream.** State is keyed by tool name and underlying endpoint so one failing tool doesn't poison unrelated ones. Open on five consecutive transient errors or a sustained error rate above ~40%. - **Open, half-open, closed.** While open, calls are refused immediately and the **planner never sees the tool** — it is filtered from the toolset, so the model spends zero tokens proposing calls the breaker rejects. After backoff, a half-open breaker admits one probe: success closes it, failure re-opens it longer. - **Fleet-wide, not per-pod.** In-process state means every instance trips its own breaker while the fleet keeps hammering. Share state via Redis or etcd so the platform presents one breaker per tool, and stagger half-open probes with jitter. - **Break on the right class.** Open on *transient* failures only. Contract and model errors will never succeed on retry; tripping on them only masks a broken schema or a drifting model behind "service unavailable." The same logic extends to the model layer: when a provider degrades in one region, break to a secondary model or region. The breaker does not care whether the thing being hammered is an ERP endpoint or an inference API — and because it enforces limits the model cannot see past, a behavioral problem becomes a solved infrastructure problem. ## 5. Observability: Tracking Recovery Success Rates and "Time to Heal" A self-healing loop that is not instrumented is just a loop with extra steps — you cannot tell whether it is healing errors or converting them into confident wrong answers. Recovery observability adds two metrics to decision logging; both deserve SLOs. **Recovery success rate (RSR)** — the share of erroring steps that resolve to a verified successful outcome without human intervention, broken down by error class and healing pattern. RSR is the loop's health signal: a falling transient-class RSR tracks upstream fragility and miscalibrated retry budgets; a rising model-error RSR means the loop heals model mistakes so efficiently that your eval suite stops noticing the model is drifting. Track per class, per tool, per prompt version — alert on deltas, not absolutes. **Time to heal (TTH)** — wall-clock from first failed attempt on a step to that step completing successfully, or to human handoff. TTH is the direct cost of the uncertainty the loop absorbs and the best single number for comparing agent designs: a retry healed in 400 ms is noise; a model error needing four replans and a backtrack is a design finding. Track p50 and p95 and gate releases on them like latency. Every recovery decision emits an event that makes the healing replayable: ```json { "ts": "2026-09-06T09:14:02.118Z", "trace_id": "7c2e…", "step": 7, "tool": "crm.account.update", "error_class": "transient", "error_code": "UPSTREAM_TIMEOUT", "recovery": { "pattern": "retry_then_switch", "attempts": 2, "fallback_tool": "crm.account.update.async" }, "healed": true, "ttheal_ms": 5400, "cost_usd": 0.031 } ``` Build supporting panels around them — retry counts by error code, repair-budget exhaustion, breaker opens per tool — plus the chart most teams skip: **cost of recovery versus cost of starting over.** When a plan is fourteen steps deep and three are poisoned, healing can cost more than killing the run and restarting fresh. Then write SLOs that hold the loop accountable. The ones we run in production: **≥ 95% of transient errors heal automatically with p50 TTH under five seconds; ≥ 90% of contract errors heal via plan-repair; escalation under 2% of erroring steps; zero silent successes after an exhaustion path.** Alert on RSR falling below target or TTH p95 breaching — never on raw error counts, because a healthy healing loop is supposed to see errors. ## 6. The OnlineCloud Standard: Building Agents That Survive Production Entropy None of this is exotic, and that is the point. A self-healing agent is not a smarter model; it is a runtime that treats the model as the least reliable component and engineers around it. The discipline reduces to a short list — the one we hold every engagement to: **Typed tools with clean error semantics** — every wrapper classifies failures into the envelope before the model sees them. **Idempotency as a precondition** — without an idempotency key, retries and backtracking are off the table. **Recovery policy as data** — retry budgets, fallback registries, breaker thresholds, and repair caps live in versioned configuration, never improvised in a system prompt. **Limits the model cannot override** — breakers, per-run dollar ceilings, and human checkpoints on irreversible actions are enforced in the runtime. **Instrumented from day one** — RSR, TTH, and escalation rate ship with the first vertical slice, not the postmortem. One practice separates evaluated systems from demos: **failure injection.** Part of the eval suite that defines "done" is a chaos pass — kill the upstream mid-run, ratelimit the agent, return a schema-valid but poisoned response, rotate a credential — and the agent must heal within its SLOs. If it cannot survive a simulated outage in week three, it will not survive a real one in month six. That is the difference between an agent that demos well and one your operations team can sleep through: the second has already failed, in a controlled environment, and gotten better at it. Production entropy is not a bug to be eliminated; it is the environment. Upstreams degrade, schemas drift, credentials rotate, and models change their minds. The agents worth running are designed for that world — classifying errors, healing within budget, escalating cleanly when they cannot. At OnlineCloud, we build those agents: US-based senior engineers shipping evaluated, observable agentic systems against Oracle, MS SQL, Postgres, Snowflake, and the CRMs and APIs your teams use every day, typically in 6–10 weeks, with dashboards and on-call owned by the people who built it. If your agents are one 429 away from a thousand-dollar spiral, [contact the OnlineCloud team](https://www.onlinecloud.dev/) and bring the workload you are tired of babysitting — we will show you a loop that heals itself.

# Multi-Agent Conflict Resolution: Implementing Borda Count and Voting Patterns

*By the Principal AI Scientist team at OnlineCloud — Agentic AI & MCP Engineering Studio*


—

## 1. Introduction: When Agents Disagree

Every production agentic system eventually collides with one fact: **the same prompt, tools, and context do not guarantee the same answer.** Sample one LLM ten times at nonzero temperature and you get ten ranked candidates. Put ten differently-instructed agents — a retrieval agent, a code agent, a reasoning agent, each with its own model and tool access — on one question, and you get ten *confident, mutually exclusive* answers. This is not a defect to engineer away; it is the fundamental non-determinism of stochastic language models reasoning over partial information.

Teams react in one of two ways. The first: pick a “best” agent and ignore the rest — cheap, but it inherits that agent’s blind spots. The second: run a naive majority vote over final answers — better, but it silently assumes every opinion is equally trustworthy. A specialist who has worked with this exact schema for three years and a generalist who has seen it for three seconds get equal weight. That is not consensus; it is noise averaging.

The mature position treats disagreement as a **first-class signal** and resolves it with a principled decision procedure. This post walks through what that looks like in practice: Plurality, Borda Count, and Condorcet methods adapted to swarms; weighting that respects confidence and expertise; the central-Judge-versus-peer-to-peer architecture choice; and bounding time-to-consensus so resolution never becomes the bottleneck it was meant to prevent. Working implementations are included throughout — a consensus layer you cannot ship is just a slide.

—

## 2. Voting Algorithms: Plurality, Borda Count, and Condorcet in Swarms

### 2.1 Why ranking beats single-choice ballots

The most important design decision is the *ballot format*. Plurality asks each agent for one preferred answer — easy to collect but pathologically lossy. It discards everything the agent knows about its *second-best* candidate, and when agents disagree because each holds different evidence, second choices frequently encode the compromise first choices cannot see. The fix: ask each agent for a **total or partial ordering** over the candidates. Elicitation is cheap — an LLM can rank ten candidates in one structured-output call — and it unlocks every ranked method in social choice theory.

### 2.2 Plurality: the baseline

Plurality is the counting rule you already know: each ballot contributes one point to its first-ranked candidate; the most points wins.

“`python
def plurality(ballots: list[list[str]]) -> str:
scores: dict[str, int] = {}
for ballot in ballots:
top = ballot[0]
scores[top] = scores.get(top, 0) + 1
return max(scores, key=scores.get)
“`

It is O(N) to tally and trivially explainable — which is why it survives in production. But it fails the moment the swarm splits three ways while every agent’s second choice is the same. Plurality can elect a candidate a majority actively ranks last.

### 2.3 Borda Count: reward the whole ranking

Borda Count scores every position, not just the first. With *m* candidates, an agent’s top choice receives *m−1* points, its second *m−2*, and so on; the highest total wins. It is the workhorse of swarm consensus because it is **easy to elicit, deterministic to tally, and robust to small perturbations** in individual orderings.

“`python
def borda(ballots: list[list[str]]) -> list[tuple[str, float]]:
m = len(ballots[0]) # assume complete rankings
scores: dict[str, float] = {}
for ballot in ballots:
for rank, cand in enumerate(ballot):
scores[cand] = scores.get(cand, 0.0) + (m – 1 – rank)
return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
“`

Implementation notes from the field:

– **Handle incomplete ballots.** Some agents refuse to rank candidates outside their expertise. Padding unranked candidates with zeros silently punishes honesty; prefer *normalized Borda*, scaling each ballot so its total is constant. That neutralizes partial ballots and over-confident agents alike.
– **Decide the tie-break policy in advance.** Borda ties are common in small swarms: break by plurality count, then mean self-reported confidence, then deterministically by hash. A policy invented during an incident is a policy written by whoever is loudest.

Borda’s known weakness is strategic voting — an agent can bury a rival by ranking it last. Inside a swarm this is less malice than *prompt drift*: an agent told to “be decisive” systematically distorts its ranking. Mitigate with explicit instructions to rank by genuine preference and by monitoring ballot distributions for anomalies.

### 2.4 Condorcet: the gold standard (and its cycle problem)

The Condorcet criterion asks a sharper question: *is there a candidate who beats every other candidate in pairwise head-to-head comparison?* Such a Condorcet winner is the most defensible outcome — the candidate a majority prefers against each alternative.

“`python
from itertools import combinations

def pairwise(ballots: list[list[str]]) -> dict[tuple[str, str], int]:
“””Returns { (a, b): number of ballots ranking a above b }.”””
wins: dict[tuple[str, str], int] = {}
for ballot in ballots:
pos = {c: i for i, c in enumerate(ballot)}
for a, b in combinations(ballot, 2):
key = (a, b) if pos[a] < pos[b] else (b, a) wins[key] = wins.get(key, 0) + 1 return wins def condorcet_winner(ballots: list[list[str]]) -> str | None:
wins = pairwise(ballots)
cands = set().union(*(set(b) for b in ballots))
for c in cands:
if all(wins.get((c, o), 0) > wins.get((o, c), 0)
for o in cands if o != c):
return c
return None # no Condorcet winner — cycle
“`

The catch: majority preferences can be *intransitive* — agent A prefers X over Y, Y over Z, yet Z over X. This is Condorcet’s paradox, and in swarms it is not an exotic corner case; agents weighting different evidence dimensions is precisely the structure that produces cycles. When no Condorcet winner exists, fall back to a *completion method* — Ranked Pairs, Schulze’s beatpath, or simply Borda. My production policy is two-stage:

1. If a Condorcet winner exists, return it (and log it — that is strong evidence).
2. Otherwise, resolve with Borda Count and *surface the cycle in the audit trail*.

A cycle is not a failure; it is a diagnostic gift. It tells you the evidence is genuinely polarized and no answer carries majority legitimacy — the exact moment to escalate to a weighted Judge (§4) or a human checkpoint.

—

## 3. Weighted Decisions: Confidence and Expertise

Unweighted voting assumes agent opinions are exchangeable. They are not. Two factors matter most, and both should be explicit.

### 3.1 Calibrated confidence

Have each agent report a confidence score alongside its ranking — but *never* trust the raw number. LLM confidence is notoriously miscalibrated: overconfident out-of-distribution, underconfident on routine inputs. Calibrate reported confidence against historical accuracy the way you would any classifier.

“`python
def borda_weighted(ballots: list[list[str]],
weights: dict[str, float]) -> dict[str, float]:
“””Borda with per-agent weights, e.g. calibrated confidence.”””
m = len(ballots[0])
scores: dict[str, float] = {}
for agent, ballot in ballots:
w = weights.get(agent, 1.0)
for rank, cand in enumerate(ballot):
scores[cand] = scores.get(cand, 0.0) + w * (m – 1 – rank)
return scores
“`

Collect per-agent accuracy over time — against ground truth when available, cross-agent agreement and human review otherwise — and fit a monotonic map from reported confidence to empirical precision. Deploy the map, not the raw score. If an agent’s confidence tracks its accuracy at all, calibration shifts outcomes more than any cleverness in the tally rule.

### 3.2 Expertise as a prior weight

The second weight is *situational expertise*: how relevant is this agent to *this question*? A SQL agent queried on legal jurisdiction should not vote with the authority of a legal agent. Compute a task-expertise score at query time — from tool-usage history, retrieval overlap with the question’s domain, or a lightweight query classifier — and multiply it in. The effective weight of agent *i* on question *q* is:

“`
w_i(q) = calibration_i(confidence_i) × expertise_i(q)
“`

One caution: expert weights can ossify into a self-fulfilling prophecy — the legal agent always wins legal questions, so it gets more legal questions, so its weight grows. Add a mixing term or periodic re-evaluation so minority experts keep a voice, and monitor the *effective number of voters* to catch when the swarm collapses into a single-agent echo. Also cap the weight ratio between strongest and weakest voter (5:1 works), and require the winner to clear a *confidence floor* τ — calibrated probability, not raw margin. If nothing clears τ, the correct output is not the best available answer; it is *abstention plus escalation*.

—

## 4. Architecture: Central Judge vs. Peer-to-Peer Consensus

The tally rule is orthogonal to the *architecture* that collects ballots. Two patterns dominate, mapping onto classic distributed-systems trade-offs.

### 4.1 The central Judge

A single orchestrator (the “Judge”) fans out the question, collects ranked ballots, applies the weighted tally, and returns a verdict. This fits MCP-native orchestration naturally: the Judge is one more agent with tool access, and ballot collection is N tool calls. Judge-based systems are simple to reason about, trivially auditable (one component owns the tally), and easy to make deterministic — the Judge can be a temperature-0 call over pre-computed scores, so the *voting layer* has zero non-determinism even though the voters do not.

The risks are concentration and bias: a single point of failure, a single point of capture, and a tempting prompt-injection target if it alone sees the full picture. Mitigations: keep the Judge *stateless and dumb* — it should tally, not opine, because the moment it injects its own reasoning you have an unweighted extra voter — run it deterministically, and log the full ballot transcript.

### 4.2 Peer-to-peer consensus

At the other extreme, agents exchange and revise rankings directly — a gossip or all-pairs protocol where the aggregate belief state is the ballot and agents iterate toward a fixpoint. This fits federated swarms across organizations, or any setting where a central Judge is itself a latency and trust bottleneck.

P2P consensus inherits decades of distributed-systems machinery: you are effectively running a Byzantine agreement protocol whose “messages” are rankings and whose “faults” are miscalibrated or adversarial agents. Guaranteed agreement under faults means something PBFT-like — O(N²) messages, 3f+1 replication. For most commercial workloads that is overkill: honest agents disagreeing is not the Byzantine generals problem, and treating it as such multiplies infrastructure cost for zero accuracy gain.

My guidance: **start with a Judge, instrument the failure modes, and decentralize only when a concrete requirement — no single point of trust, cross-org boundaries, latency — forces it.** The hybrid is strongest: P2P *deliberation* (agents exchange summaries and revise rankings — this is where new evidence surfaces) followed by Judge-based *resolution* (a deterministic weighted tally on the final rankings). Separate the conversation from the vote and you get the epistemic benefit of consensus with the auditability of a central record.

—

## 5. Performance: Bounding Time-to-Consensus

Consensus layers fail in production when they are slow, because latency is the metric everyone notices. Bound it explicitly with a three-term cost model:

“`
T_consensus = T_elicit + T_tally + T_rounds
“`

– **T_elicit** — one LLM round-trip per agent, fully parallelizable. This dominates: serial elicitation of ten agents is ten latencies; parallel is one.
– **T_tally** — microseconds for any method in §2; all-pairs Condorcet is O(N·m²), trivial at N=20, m=10. Tallying is never your bottleneck; stop optimizing it.
– **T_rounds** — the hidden cost. Multi-round deliberation multiplies T_elicit by the round count.

Three disciplines keep time-to-consensus bounded:

1. **Cap the rounds.** Deliberation returns diminish sharply after round two while latency compounds linearly. Set `max_rounds=2`; round three requires a human or an explicit budget override.
2. **Early-stop on dominance.** If a Condorcet winner exists after round one *and* its margin over the runner-up clears a threshold, declare victory. Most disagreements are shallow — in our eval sets, 60–70% — and you should not pay deliberation cost on questions that were never truly contested.
3. **Time-box and fall back.** Give the swarm a deadline. On expiry, resolve with whatever ballots exist using normalized Borda so partial participation does not bias the tally. A guaranteed latency ceiling is what makes the consensus layer safe to place on the critical path.

Finally, measure it like an SLO: p50/p95 time-to-consensus per question type, per model, per swarm size. When p95 drifts, it is almost always one slow voter — the fix is a per-agent latency cap with abstention, not a faster tally.

—

## 6. Real-World Use Case: Diagnosis and Legal Interpretation

### 6.1 Differential diagnosis in a clinical reasoning swarm

Consider a diagnostic swarm: a radiograph agent, a lab-value agent, a clinical-history agent, and a literature agent, each returning a ranked differential over five candidate conditions. This structure breaks plurality: the radiograph agent is confident in *pneumonia*, the lab agent in *pulmonary embolism*, the others split further — yet *all five* rank *heart failure* second. Plurality inflates the false dichotomy; Borda surfaces the compromise candidate every specialist could live with, which is exactly the differential a prudent clinician wants to work next.

Now add weights: the radiograph agent’s confidence on imaging questions is historically well-calibrated, so its weight is high; the generalist’s is low. Critically, the swarm can flag *its own insufficiency* — if the winning differential’s calibrated probability sits below the escalation threshold τ, the correct behavior is to page a human radiologist, not emit a confident wrong answer. The voting layer just converted a silent misdiagnosis risk into an explicit, documented escalation event.

### 6.2 Statutory interpretation across jurisdictions

Legal interpretation is the adversarial mirror image. Here agents are not collaborators but *advocates*, each briefed with one jurisdiction’s precedent, one canon of construction, one reading of legislative intent. Because voters are deliberately polarized, expect Condorcet cycles: interpretation A beats B in the textualist camp while B beats A among purposivists, and no majority winner exists.

This is where the two-stage policy earns its keep. A Condorcet winner means the question is more settled than the advocates’ posturing suggested — return it. If the tally cycles, the system should *say so*: emit the Borda outcome *labeled with the cycle*, publish the pairwise margins, and escalate to a human review board. The cycle is not a bug; it is a formal proof that the question is genuinely contested — legally meaningful information in itself. In regulated settings, the audit trail — full ballots, weights, pairwise matrix, and the rule that produced the verdict — is often more valuable than the verdict. MCP’s typed, logged tool calls make capturing that transcript a default property of the architecture rather than an afterthought.

—

## 7. Conclusion: Make Disagreement a Feature

Non-determinism is not going anywhere; it is the price of models that generalize. The professional response is not to hide from disagreement but to give it a **decision procedure**: rank-ordered ballots, Borda for robust aggregation, Condorcet for majority legitimacy, calibrated confidence and situational expertise as weights, a deterministic Judge for resolution, bounded rounds for latency, and escalation as a first-class outcome. Done right, the consensus layer turns a swarm’s divergence from a reliability liability into a measurable source of robustness — with the audit log to prove it.

**At OnlineCloud we ship these patterns in production — MCP-native orchestration with typed tools, evaluated outcomes, and consensus layers wired into Oracle, MS SQL, and Postgres.** If “which answer wins” in your multi-agent workload is still decided by vibes or a hardcoded if-statement, that is exactly the conversation we want to have.


—

## Ready to wire consensus into your agent stack?

We are a US-based Agentic AI & MCP engineering studio. Senior engineers scope it, build it, and operate it — no offshore handoffs, no decks, no quarters-long timelines. Most engagements ship to production in **6–10 weeks**.

– **Email:** contact@onlinecloud.dev
– **Web:** https://www.onlinecloud.dev/
– **Support:** Dedicated USA-based support · Free initial consultation · Response within 24 hours

*Tell us which agents in your system are arguing — we will show you the voting layer that settles them.*

# Optimizing Prompt Caching in Anthropic Bedrock for Complex Schemas


*By the OnlineCloud Engineering Team — LLM Infrastructure Engineers*

—

## 1. Introduction: The Quiet Tax on Large Tool Surfaces

Every agentic system hits the same inflection point: the moment the tool belt outgrows the model’s default economics. Early demos carry two or three tools and prefixes stay small. Then production happens. Your MCP server exposes forty tools, each carrying a name, a prose description, and a JSON Schema — and the tools wrapping your Oracle or CRM objects run to hundreds of tokens alone. Add a system prompt with governance rules and output contracts, and every request now opens with a 30,000-to-80,000-token preamble.

Here is the part that surprises platform teams: **that preamble is re-billed and re-processed on every request.** Each invocation of the Messages API is a fresh prefill; the provider charges full input price for every token it processes, system prompt and schemas included. At $3 per million input tokens, a 50,000-token prefix is $0.15 of dead weight before the model emits a single useful word. At millions of requests a month, the tool surface becomes a six-figure line item — and because time to first token (TTFT) grows with prompt-processing cost, the biggest tool set is also the slowest one.

The good news: this cost is almost entirely avoidable. Anthropic’s prompt caching on Bedrock is built for exactly this pattern — a large, stable, repeated prefix — and a well-structured MCP deployment is the ideal workload. This post covers how the cache works, how to structure MCP tool schemas so you hit it, what the 90% input discount looks like in production arithmetic, how to benchmark the TTFT win, and how to govern prompt changes so the cache stays warm on purpose rather than by accident.

## 2. How Anthropic Prompt Caching Actually Works

Prompt caching is a **prefix cache with exact-match semantics at the token level**. You annotate positions in the request with cache breakpoints — a `cachePoint` marker in the Bedrock Converse API (`cache_control` in the raw Anthropic API) — and the platform caches everything preceding each breakpoint. The next request that opens with an identical token prefix is served from cache instead of re-running the model’s prefill stage.

Three mechanics drive every engineering decision:

**Prefix, not semantic, matching.** The cache key is the token stream, not the meaning of the request. One changed byte upstream — a reworded system-prompt sentence, a tool inserted mid-list, a JSON object whose keys serialized in a different order — produces a different stream and a complete miss. Matching is strictly positional: once the stream diverges, everything after it is uncached.

**Minimum size and TTL tiers.** A breakpoint must sit at least 1,024 tokens into the prompt for the 5-minute TTL tier (the 1-hour tier requires a longer prefix). Entries are sliding-window: each read refreshes the TTL, so a hot prefix stays resident indefinitely while a cold one expires minutes after its last request. A prefix that sees traffic only once an hour will never stay warm on the 5-minute tier — that is what the 1-hour tier is for.

**Pricing asymmetry.** Cache reads cost a fraction of the base input price; writes cost a premium:

| Tier | Cache write (per input token) | Cache read (per input token) |
|—|—|—|
| 5-minute TTL | 1.25× base input price | 0.1× base input price |
| 1-hour TTL | 2.0× base input price | 0.1× base input price |

Reads are where the 90% headline comes from; writes are the price of admission, paid once per TTL window per distinct prefix. The strategy is therefore to maximize the read-to-write ratio by keeping the prefix byte-identical across as many requests as possible.

A Bedrock Converse request with breakpoints after the system prompt and after the tool list:

“`python
response = bedrock.converse(
modelId=”anthropic.claude-sonnet-…:0″,
system=[
{“text”: SYSTEM_PROMPT}, # static, versioned, cached
{“cachePoint”: {“type”: “default”}},
],
toolConfig={
“tools”: [
*serialize_tools(TOOL_REGISTRY), # byte-stable, ordered
{“cachePoint”: {“type”: “default”}},
]
},
messages=[{“role”: “user”, “content”: [{“text”: user_query}]}],
)
“`

Bedrock supports multiple breakpoints per request on compatible model versions, so you can cache the fully static region and keep volatile content — per-tenant instructions, retrieved context — behind your own breakpoint. The rule of thumb: **everything static goes before a breakpoint; everything that varies per request goes after one.**

## 3. Structuring MCP Tool Definitions and Schemas for Cache Hits

Getting cache hits is not a configuration task — it is a serialization discipline. In an MCP-native architecture, the MCP server is the source of truth for tool definitions, and an adapter layer converts them into the tool payload of every model request. That conversion must be deterministic, or you will silently invalidate the cache on every request.

**Canonical, byte-identical serialization.** The cache matches on tokens, so two semantically identical requests that serialize differently are two different prefixes. Pick one canonical serialization — stable tool ordering plus fixed JSON key order — and enforce it in every process that builds requests. Beware runtimes that hash-map keys into arbitrary order, and never build tool dicts ad hoc in request handlers. Serialize once from the registry and reuse the bytes.

**Freeze ordering; grow append-only.** Token streams are positional: inserting a tool mid-list invalidates every cached segment after it, fleet-wide. New tools go at the end of the registry; reordering is a cache-invalidation event, not a cosmetic change. Removing a tool is a breaking deployment — version-bump, warm the new prefix, then shift traffic.

**Keep dynamic content out of the cached region.** Tool descriptions and system text must not be templated with per-request, per-tenant, or time-varying values — no “as of {date}”, no environment names, no user clauses inside the cached segment. The most common cache-killer we audit is a system prompt interpolating a timestamp at the top, quietly changing the prefix on every request.

**Curate the giant schemas.** Enterprise tools wrapping real database objects carry most of your prefix tokens — which is precisely why caching matters for them. Trim redundant `description` prose, drop field documentation duplicated between description and `inputSchema`, and avoid scattering the same large schema across near-duplicate tools. Every cached token is billed at write premium and saved at read discount; verbosity has a price either way.

**Right-size the tool surface per agent.** If your MCP server exposes 60 tools but an agent class uses eight, every request pays for the other 52. Partition tool sets by domain — order-status agents get the read-only CRM and order tools, reconciliation agents get finance tools — so each agent class has a smaller, cheaper, more stable prefix, with less drift surface and cheaper re-warm after changes.

**Lint the wire format in CI.** Add a golden-file test that builds the exact request payload and diffs it against the committed artifact. Any change fails the build and forces an explicit decision — which is what you want, because in a cached system a prompt edit is a production change with cost and latency consequences.

## 4. Cost Analysis: Measuring the 90% Discount in Production

The discount is real, but it should be evaluated end-to-end on your own workload. Take a representative profile: a tool-heavy agent with a 12,000-token system prompt and 33,000 tokens of MCP schemas — 45,000 input tokens per request — serving 100,000 requests/day against a Claude-class model at $3/MTok input, $15/MTok output, ~800 output tokens per request.

**Without caching**, every request pays full price for the prefix: 100,000 × 45,000 × $3/MTok = **$13,500/day** in input cost, on top of ~$1,200/day of output tokens.

**With a stable cached prefix**, each 5-minute window pays one 1.25× write and the rest read at 0.1×. At roughly uniform traffic, that is ~288 writes and ~99,700 reads per day:

– Writes: 288 × 45,000 × $3 × 1.25 ≈ **$49/day**
– Reads: 99,700 × 45,000 × $3 × 0.10 ≈ **$1,345/day**
– Total input cost: **~$1,394/day — a 90% reduction**; output cost unchanged.

That ratio holds only while the prefix stays byte-identical and traffic keeps the TTL alive. Fail either and every request pays write premiums — which is why instrumentation is mandatory.

Bedrock exposes the counters you need — `cacheReadInputTokens` and `cacheWriteInputTokens` in the invocation metrics — which you can stream to CloudWatch alongside raw token counts. Track **hit ratio = reads / (reads + writes)** per prompt version, model, and region. Healthy systems sit above 95%; anything below is almost always serialization drift, tool reordering, region-split traffic, or TTL expiry during low-traffic hours. If inter-request gaps routinely exceed five minutes, the 1-hour tier (2× write, same 0.1× read) is worth the premium — run the arithmetic with your actual cadence before choosing.

## 5. Benchmarking TTFT with Cached Context

Cost is half the story. TTFT is dominated by prefill — the model processing the entire prefix before emitting anything — so a 45,000-token preamble is not just expensive, it is slow. Caching removes most of that work from the critical path, and on long prefixes the latency win matches the cost win.

Methodology matters. Measure TTFT from request submission to the first streamed token, and confirm cache hits per request via the read-token counters — otherwise you are comparing a hit population against a miss population and calling it a model improvement. Run an A/B against production-shaped traffic: the same system prompt and tool registry, many different conversations, so you measure real prefix reuse rather than identical repeated queries.

Representative numbers for a 45,000-token prefix on Bedrock (p50/p95 across a production mix):

| Metric | Cache miss (cold prefix) | Cache hit (warm prefix) |
|—|—|—|
| TTFT p50 | 4.9 s | 0.8 s |
| TTFT p95 | 8.2 s | 1.6 s |
| Input cost per request | $0.135 | ~$0.014 |

An 80–85% TTFT reduction is typical at this prefix size, and the absolute gap grows with prefix depth — the larger your schemas, the bigger the win. Two consequences follow. First, **interactive quality changes**: a tool-calling agent that responds in under a second instead of after a five-second pause feels fundamentally different, and gateway timeout budgets stop being a design constraint.

Second, **savings compound across multi-step loops.** An agent resolving a ticket may run a ten-step tool chain, and nearly every step re-sends the full prefix. Cache each step and the 90% input discount and ~85% prefill reduction apply ten times per resolved task — the difference between a loop that costs dollars per completion and one that costs cents. That is why we treat prompt caching as a first-class architectural constraint when designing MCP tool surfaces, not a billing optimization applied afterward.

## 6. Governance: Automating Prompt Versioning for Cache Stability

A cached deployment turns every prompt edit into an infrastructure event. Change one system-prompt sentence and you invalidate the cache for the entire fleet: the next request from every agent pays a write premium and eats a cold prefill. Edit prompts by hand in request handlers and you will do this silently and constantly. The fix is to treat the system prompt plus tool registry as a versioned, deployable artifact with the same discipline as application code.

**The artifact.** The canonical prefix — system text, ordered tool list, serialized schemas — lives in version control (or an S3-backed registry) as one immutable bundle. A build step produces the byte-serialized prefix and its content hash. Runtimes load a pinned version; nothing constructs prompts at request time.

**The CI gates.** Changes to the bundle run through checks encoding the caching rules: a golden-file test asserting byte-identical serialization; an ordering lint enforcing append-only tool growth; a scan rejecting time-varying or per-request interpolation inside the cached region; a token check confirming the prefix still exceeds the 1,024-token minimum. A prompt that fails these gates cannot reach production.

**Versioned rollout with cache warm-up.** Because a release is a cache-invalidation event, deploy it like a risky change: release the new prefix version, run synthetic warm-up traffic to populate the cache before users arrive, then ramp 5% → 100%. This prevents the classic post-release symptom — a normal traffic day billing like a cold start because every request is a cache write. Tag request observability with the prompt version so hit rate and cost are attributable per release.

**Rollback is nearly free.** Entries persist for the TTL after their last read, so reverting to the previous version often restores a still-warm cache within minutes — the platform rolls back with your artifact. That asymmetry (cheap rollback, expensive forward drift) argues for small, reviewed prompt changes over large rewrites.

**The SLO.** Track cache hit ratio per prompt version and alert on drops. A sudden hit-rate cliff is the earliest signal of serialization drift, an accidental reorder, or a rogue templated value — visible in the token counters hours before it shows up in the cost report.

None of this is exotic. It is the release engineering you already run for services, applied to the one artifact most teams still edit by hand — and it is the difference between prompt caching as a lucky discount and prompt caching as a reliable production lever.

## Conclusion: Make Your Schema the Asset

Large tool surfaces and long system prompts are not a mistake — they are the shape of real enterprise agentic systems talking to real schemas. The mistake is paying full price for them on every request. Bedrock prompt caching turns the tax on complex schemas into a 90%-discounted, dramatically faster repeating prefix — but only if you engineer for it: canonical serialization, frozen tool ordering, volatile content quarantined behind breakpoints, and prompt changes governed like deployments.

At OnlineCloud we build MCP-native agentic systems for a living, designing the tool surface, serialization layer, and caching strategy as one system — measured by cost per resolved task and p95 TTFT rather than demo quality. If your agent fleet is paying full freight for a tool belt it re-sends on every call, talk to us. We are a US-based engineering studio of senior engineers — no offshore handoffs — shipping evaluated, observable systems to production in 6–10 weeks.

**Contact the OnlineCloud team today** — [onlinecloud.dev](https://www.onlinecloud.dev/) · [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) · Free initial consultation, response within 24 hours.

# Building a Custom Retrieval-Interleaved Generator (RIG) Architecture


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: Moving Beyond Basic RAG to Dynamic, Interleaved Retrieval

Classic retrieval-augmented generation makes a single, early bet. Embed the question, pull the top-k chunks, stuff them into the prompt, and let the model write one uninterrupted answer against that frozen context. For single-hop lookups over a semi-static corpus, that bet usually pays. The moment questions become multi-hop — reconciling ledgers, comparing contracts against amendments, explaining variances from live ERP rows — the bet is made before the model knows what it needs, and retrieval recall becomes a ceiling no prompt engineering can lift.

The architectural response is to treat retrieval not as a prologue but as a *conversation the generator conducts with its data while it writes*. In a retrieval-interleaved generator (RIG), the model writes what it can defend, emits a control token when its internal evidence runs thin, pauses while a retrieval tool executes against the live system of record, ingests the results, and resumes — possibly several times before the final token. Retrieval becomes a loop the generator drives mid-generation, conditioned on the state of its own output.

This is a different problem than RAG, and different again than the outer agentic loop of plan-act-reflect tool calls, where an orchestrator bounces whole LLM turns back and forth. RIG lives in between: tool-calling granularity *inside* a single response, at sub-turn latency. Done well, it produces answers that are dynamically grounded, numerically verifiable, and traceable to the exact rows behind each sentence. Done badly, it is a latency and cost trap. This post lays out the architecture we use at OnlineCloud — the decision mechanism, the decoding tricks, the verification layer, and the latency budget that keeps it shippable.

## 2. The RIG Pattern: How Generators Decide When to Retrieve Mid-Generation

The core design question is one of control: **who decides that more evidence is needed, and on what signal?** Production systems fuse two decision families.

**Token-level learned decisions.** The model is fine-tuned so the retrieval decision lives in its own output distribution. Extend the vocabulary with control tokens (``, ``, ``) and supervise trajectories with retrieval calls placed where ground truth shows they were needed — before a number only the ledger can supply, before a clause only the contract text contains. Supervised fine-tuning teaches *placement*; a reinforcement pass over outcome rewards teaches *judgment* — when the expected value of another lookup beats the cost of guessing. The result is a generator that retrieves opportunistically from internal state, not from an external planner’s guess.

**Heuristic interrupt gates.** Separate from (or layered on top of) learned placement, an external monitor watches the decoding stream. The cheapest reliable signal is token-level predictive uncertainty: when the softmax over the next token is flat, or the top candidates disagree structurally, the model is improvising — exactly the moment an architect wants a retrieval checkpoint rather than a confident hallucination. A classifier head or a calibrated threshold on sequence log-probability fires an interrupt that forces retrieval before the model commits to a fact-bearing span.

What distinguishes RIG from the outer agentic loop is *coupling*. An agentic workflow decomposes work into whole-turn tool calls — orchestrator asks, tool answers, fresh turn — while RIG keeps the generator’s state (its KV cache, attention pattern, narrative thread) alive across retrieval. The interruption lands at a token boundary, not a turn boundary, and generation resumes from its own hidden state rather than a synthesized transcript. One document can thus hold five interleaved lookups without the coherence loss of five handoffs.

The failure mode to design against is RAG’s mirror image. RAG fails by retrieving too little, too early. RIG fails by retrieving too eagerly — a model that treats `` as a procrastination device, burning round-trips to postpone committing to an answer. The budget controls in Section 5 keep the learned policy honest.

## 3. Implementation: Using Control Tokens and Logit Bias to Trigger Retrieval Tools

Building a custom RIG means owning the decoding loop. Here is the skeleton, in the shape we ship against Oracle, MS SQL, Postgres, and Snowflake backends.

**Vocabulary and fine-tuning.** Reserve three tokens: `` signals a lookup; `` and `` delimit injected results so the model can tell retrieved fact from generated prose. During SFT, annotate trajectories with retrieval points wherever a fact’s source changes, inserting `Q` where Q is the query that would surface the needed rows. During RL, reward trajectories whose outputs pass the verifier from Section 4 and penalize premature or excessive retrieval.

**Decoding with an interruptible loop.** The naive decoder is replaced with a loop that owns the token stream:

“`python
def decode_with_retrieval(model, prompt, tools, gate, budget):
tokens, cache = model.encode(prompt)
for step in range(budget.max_decode_steps):
logits = model.forward(tokens, cache)
if gate.should_retrieve(logits): # uncertainty interrupt
logits[RETRIEVE_ID] += 2.5 # logit bias -> forced trigger
token = sample(logits)
if token == RETRIEVE_ID:
query = model.decode_until(tokens, RETRIEVE_END)
results = tools.execute(query) # typed MCP tool call
tokens += encode(f”{results}“)
budget.retrievals += 1
if budget.retrievals >= budget.max_retrievals: break
tokens.append(token)
“`

Three details decide whether this works in production. **First, the logit bias is a governor, not a crutch.** A +1.5 to +3.0 bias on `` makes the token sampleable exactly when the gate fires while leaving ordinary decoding untouched — the model never *must* retrieve, it is merely made willing. This is how a learned policy sits under a deterministic safety control. **Second, KV-cache continuity.** Pausing at the control token and resuming after evidence injection shifts position IDs mid-sequence, and models with relative position encodings (RoPE and friends) tolerate that poorly. We fine-tune on the exact `` continuation format so the model learns the shift, or recompute the cache from the last stable prefix on deep injections. Skip this and interleaving quietly degrades into confusion. **Third, retrieval is a typed tool surface.** The evidence channel is the output of an MCP-style tool with a JSON Schema contract: queries validate before executing, results carry provenance metadata, and the audit trail records every mid-generation lookup as a first-class event.

The gate needs its own calibration pass. On held-out generation traces, measure the fraction of gate firings that changed the downstream answer: gates that fire on trivia add latency without value; gates that fire too late let hallucinations through. We tune the uncertainty threshold so the *missed-retrieval rate* — fact-bearing spans written with no evidence behind them — stays below what the verification layer can absorb.

## 4. Data Integrity: Verifying Interleaved Facts Against the Source of Truth

Interleaved retrieval raises the bar on trust in both directions. More retrieval means more grounded content — but also more surface where a retrieved fact, a synthesized inference, and a hallucinated number can blur together in one document. A RIG answer is only as good as its ability to prove, sentence by sentence, which source of truth stands behind each claim.

**Claim decomposition.** Before an answer ships, a post-processor splits it into atomic claims — one verifiable proposition per unit: a balance, a date, a policy threshold. For financial content this is mechanical; every figure becomes a claim like *”the GL balance for cost center 4200 in June is $184,220.15.”*

**Two verifier classes.** Numeric and categorical claims verify *deterministically*: re-run the query behind the figure and compare against the live row — byte equality with the source of truth, not semantic similarity. Semantic claims (summaries, classifications, policy interpretations) go through a smaller entailment model that checks whether the claim is *entailed by the retrieved evidence block*. Passages carry provenance — table, row key, query, timestamp — so every verdict has a concrete referent:

“`json
{
“claim”: “GL balance for cost center 4200 is $184,220.15”,
“verdict”: “verified”,
“evidence”: {
“tool”: “ledger.balance”,
“query”: “SELECT SUM(amount) FROM gl WHERE cost_center=4200 AND period=’2026-06′”,
“row_key”: “gl|4200|2026-06”,
“retrieved_at”: “2026-09-06T09:14:22Z”
}
}
“`

Three verdicts, three dispositions: **verified** claims ship with their provenance attached; **unverified** claims trigger one re-retrieval before the system abstains rather than asserts; **contradicted** claims force the generator to revise the span — never to argue with the data. Row-level access controls from your MCP layer apply here too: a claim is only verifiable against rows the caller may see, keeping the integrity guarantee aligned with the security boundary.

This is where RIG earns its keep over RAG. A RAG citation can launder a hallucination — the model cites a chunk it never grounded its answer in. RIG verification is *executed*, not asserted: claim, query, and live row are all on record, and the whole interleaved trajectory replays from the audit log. Compliance teams do not need to trust the model; they need to re-run the math — and with claim-level provenance, they can.

## 5. Performance: Managing the Latency of Multiple Retrieval Round-Trips

Interleaving is a latency multiplier by construction: N retrieval round-trips add N tool executions and N model-resume steps to every response. The architecture only survives if you budget and hide that cost. Four levers, in the order we pull them:

**Cap the hops and make them count.** Every deployment carries a hard retrieval budget — typically 3 to 5 per answer — enforced in the decode loop so a runaway policy cannot inflate p95. Within it, prefer *fewer, wider* lookups to many narrow ones: one query fanning out across a date range beats four single-day queries, because round-trip latency tracks hop count, not bytes returned. Teach this preference in the RL pass: reward trajectories that satisfy the verifier in the minimum number of retrievals.

**Hide latency behind streaming.** Users need the *first* token fast, not the final one. Stream tokens as they decode, so a four-hop answer’s visible latency is the first hop plus the final segment, with intermediate stalls masked by what has already rendered. This turns a 6-second wall-clock answer into a roughly 1.5-second perceived one — the highest-ROI change in production RIG.

**Parallelize and prefetch.** When a segment needs three independent facts, fire the retrievals concurrently and join before resuming. Better: predict the next retrieval from current context — while the model writes the filler between evidence blocks, speculatively run a cheap embedding match on the probable next query and cache it. The decoder consumes the cache only if the trigger fires; otherwise the speculation is discarded. Classic prefetch for a stochastic consumer.

**Keep a semantic cache and a fast path.** Repeated reconciliations hit the same rows: key a semantic cache on the normalized retrieval query and skip the database on identical lookups. And route by query shape — an exact-identifier lookup (`invoice_id = 88132`) should never pay ANN latency when a primary-key hit on the indexed ledger costs under 10 ms versus 50–150 ms for embedding search. Expose both paths and let the generator’s query choose.

| Workload shape | Retrievals | Added p95 latency | Mitigation |
|—|—|—|—|
| Exact-ID lookups | 1–2 | +20–60 ms | PK-index path, semantic cache |
| Multi-hop analytical | 3–5 | +0.5–2.5 s | Parallel fan-out + streaming |
| Full reconciliation | 5–10 | +3–8 s | Budget caps, async verification, human-in-loop review |

The governing rule is the one for any distributed system with a nondeterministic core: *measure per-hop latency in production, set budgets from the tail, and degrade into a handoff, never into a hang.* When the budget is exhausted and the answer is still unverified, surface the partial result with its verification gaps labeled — an honest exception report beats a late, confident guess.

## 6. Case Study: RIG for Complex Financial Reconciliation Tasks

The pattern earns its complexity where questions are numeric, multi-source, and compliance-bound — which is precisely financial reconciliation. Consider the workload we have built RIG for repeatedly: closing the books when the general ledger, the subledger, and the bank statement disagree and every variance must be explained.

A classic RAG approach fails here structurally. The question is not “find the passage about reconciliation” — the answer lives in no document. It lives in millions of rows across three systems of record, and it is different every single day. With RIG, the generator drives the reconciliation like an analyst would, one verified step at a time:

1. **Open.** The generator emits its run summary and ``s the period’s unmatched transaction set — a parameterized query against the subledger’s exceptions view.
2. **Compare.** The model classifies each exception (timing difference, keying error, missing posting) and, where classification depends on policy thresholds, emits a second `` for the approval matrix and tolerance rules.
3. **Recompute.** For every candidate adjustment, the generator does not trust its arithmetic — it ``s the supporting rows and recomputes totals against the live GL, so each proposed entry ties to source rows byte-for-byte.
4. **Verify.** The Section 4 verifier re-checks every figure in the draft exception report against the rows that produced it; contradicted figures force a revision cycle before the report is released.
5. **Compose.** The final output is an exception report in which every number carries a row-level provenance pointer, and the full interleaved trajectory — every query, result set, and revision — is replayable from the audit log for the external auditor.

The measurable outcome is where errors occur. Pre-RIG, single-shot retrieval produced plausible-looking exception reports whose recomputed totals disagreed with the ledgers on a meaningful share of flagged items — the classic confidently-wrong profile. With RIG, the deterministic verifier catches numeric discrepancies at generation time rather than month-end review, and tolerance-policy lookups cut false-positive classifications. Across deployments, manual review time on the exception queue drops by roughly two-thirds, and every figure in the output is re-derivable from a logged query, not defensible in a meeting.

That last property is the real product. A reconciliation no one has to re-audit by hand is not a faster report; it is a different control. RIG turns the generator from a summarizer of whatever it retrieved into an analyst that retrieves on demand and verifies what it writes.

## Conclusion: From Retrieved Answers to Verified Decisions

RIG is the answer to a simple observation: generation is a process, not an event, and retrieval belongs inside it. A generator that can decide mid-sentence that its evidence has run thin, reach into the live system of record, and resume with verified facts produces answers that RAG — with its single early bet — structurally cannot. It costs a decoding loop, control-token fine-tuning, a verification layer, and a latency budget; it buys grounded, replayable, audit-ready output where being wrong is expensive.

If you are building grounded AI against real financial or operational data — Oracle, MS SQL, Postgres, Snowflake, your CRM — and you want an architecture where every number in an answer is traceable to a row, talk to the OnlineCloud architects. We map your data, pick the model class, define the eval that means “done,” and ship a vertical slice in weeks — evaluated, observable, and built by the same senior engineers who scope it, with no offshore handoff. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) or reach us at contact@onlinecloud.dev to explore whether RIG is the right loop for your hardest questions.

# The Math of Embedding Similarity: Cosine vs. Dot Product vs. Euclidean in Production


Ask a RAG team what decides whether their system is good and you will hear a lot of things: the embedding model, the chunking strategy, the reranker, the prompt. Almost nobody says *the distance metric*. It sits in the config file as a one-line default — `metric=”cosine”` — inherited from a tutorial, never questioned, never benchmarked.

That is an expensive mistake. The metric defines the *neighborhood* of every query and sets the recall ceiling for everything downstream: a reranker can reorder a candidate list, but it cannot surface a chunk the metric never retrieved. When retrieval silently returns plausible-but-wrong neighbors, the failure looks like model quality — but it is usually geometry.

Here is the uncomfortable part: for most modern embedding pipelines, cosine, dot product, and Euclidean distance are *not* interchangeable — their rankings only converge under a specific condition most teams never verify. This post walks through the math of each metric, when they agree and when they diverge, how CPUs actually execute these computations, and how to pick a metric for the embedding families you are most likely to run in production: OpenAI, Cohere, and BGE.

## Cosine Similarity: Direction Is All That Matters

Every vector embedding lives in ℝ^d, but cosine similarity deliberately discards almost all of that space. It keeps exactly one property: the *angle* between two vectors.

$$\text{cosine}(a, b) = \frac{a \cdot b}{\|a\| \|b\|} = \frac{\sum_{i=1}^{d} a_i b_i}{\sqrt{\sum a_i^2} \sqrt{\sum b_i^2}}$$

The denominator is the product of the two L2 norms. Geometrically, you are projecting both vectors onto the unit hypersphere — the surface of a d-dimensional ball — and then measuring their alignment there. The result is bounded in [-1, 1]: 1.0 means the vectors point in the identical direction, 0 means they are orthogonal, and negative values mean they point away from each other. The associated *angular distance* is simply arccos(cosine), which is a true distance metric on the hypersphere (it satisfies the triangle inequality, unlike raw cosine similarity).

Why does this dominate LLM search? Because of what the denominator *removes*: magnitude. Language-model embeddings are trained so that paraphrase, topical, and semantic relationships live in *direction*. Magnitude, meanwhile, absorbs all the noise you do not want ranking on — document length, token frequency, vocabulary density, idiosyncratic writing style. Two passages that mean the same thing rarely have the same norm, but they reliably sit at similar angles from the origin. Cosine is the metric that says “I do not care how long your document is, only what it is about.” That is exactly the invariance sentence transformers and retrieval-focused LLM embeddings were trained to respect, which is why cosine has become the default in LangChain, LlamaIndex, and nearly every vector database’s documentation.

One sharp edge: most vector databases define “cosine distance” as 1 − cosine(a, b), bounded in [0, 2]. It is a monotone transform that preserves ranking, but it is not arccos — it is not a true metric and cannot be used for triangle-inequality pruning. If you threshold, know which flavor your engine implements. And because real cosine scores cluster in a narrow band near 0.7–0.95, small differences carry large ranking significance; a fixed cutoff like “0.8 = relevant” is almost always wrong across domains.

## Dot Product: When Magnitude Carries Signal

The dot product is the raw, unnormalized inner product:

$$a \cdot b = \sum_{i=1}^{d} a_i b_i = \|a\| \|b\| \cos(\theta)$$

It is unbounded — it can range from −∞ to +∞ — and it rewards vectors that are both *aligned* and *long*. That coupling is precisely the point when magnitude is learned to encode something real.

The canonical case is collaborative filtering. Matrix-factorization recommenders learn user and item vectors whose dot product predicts interaction strength, and there a vector’s norm is learned signal, not noise: popular items and active users carry larger magnitudes, and the length bonus is exactly the popularity prior the loss function optimized for. Normalize those vectors away and ranking quality measurably degrades. The same logic extends to ad ranking and e-commerce scoring, where evidence strength (clicks, purchases, confidence) is deliberately folded into magnitude so the score becomes `strength × topical alignment`.

Dot product has a second virtue: for many retrieval models it is the *native* score. Dual encoders trained with InfoNCE or in-batch negatives optimize a softmax over inner products; cosine is just the L2-normalized variant of that quantity. When a model card says “trained with cosine similarity,” what was actually maximized was almost always an inner product after normalization.

The trap: if magnitude is *not* meaningful — the common case with raw text embeddings, where norms track verbosity — dot product silently biases results toward long, generic passages. The fix is normalization, which also brings a free performance win we will get to shortly.

## Euclidean Distance (L2): The Geometry of the Space You Actually Built

Euclidean distance is the most intuitive metric and, for unnormalized embeddings, the most dangerous:

$$\|a – b\|_2 = \sqrt{\sum_{i=1}^{d} (a_i – b_i)^2}$$

Unlike cosine and dot product, L2 is a genuine metric over the full space — symmetric, non-negative, zero only for identical vectors, and it satisfies the triangle inequality. That makes it the natural choice for algorithms that *assume* metric geometry: k-means, k-medoids, and metric indexes that prune with triangle-inequality bounds all rely on centroid arithmetic only L2 provides cleanly.

But here is the failure mode. Consider u = (3, 4), v = (6, 8), and w = (4, 3). Vector v is an exact scalar multiple of u — the *same direction*, cosine similarity 1.0 — just doubled in length. Vector w has the same norm as u (both length 5) but points a different way, cosine similarity 0.96. Run the metrics:

– Cosine: v ranks above w (1.0 > 0.96) — semantically correct.
– Dot product: u·v = 50, u·w = 24 — v wins, but only because it is longer.
– Euclidean: ‖u − v‖ = 5, but ‖u − w‖ = √2 ≈ 1.41 — **L2 says w is closer**, even though w is the *less* similar direction.

That inversion is the classic unnormalized-L2 trap: it conflates “different magnitude” with “different meaning.” In text retrieval, where norms are driven by document length and term statistics, L2 on raw embeddings happily ranks a scaled copy of your topic below an unrelated vector that happens to share its norm. Use L2 and you must normalize first — and once you do, the geometry collapses into something very convenient.

For unit vectors, a simple identity connects all three metrics:

$$\|a – b\|^2 = \|a\|^2 + \|b\|^2 – 2(a \cdot b) \xrightarrow{\|a\|=\|b\|=1} 2(1 – \cos\theta)$$

On the unit hypersphere, squared Euclidean distance is a monotone function of cosine similarity. **Ranking is identical.** L2 says 0 is identical and 2 is maximally dissimilar; cosine says the same thing in reverse. Which means the metric choice on normalized embeddings is never a quality decision — it is a performance and calibration decision. That is the single most useful fact in this entire post, and it is the key to the decision framework below.

## Hardware Optimization: SIMD, AVX-512, and the Real Cost of Distance

Similarity search at production scale is not a storage problem, it is an *arithmetic* problem. Every query against a flat index is an O(N·d) scan: for each of N stored vectors you perform d multiply-accumulates. Search 1 million 1024-dimensional vectors and you stream ~4 GB of floats and issue ~2 billion fused multiply-adds — per query, before any HNSW graph traversal. The SIMD width of your CPU decides whether that takes tens of milliseconds or hundreds.

Modern x86 CPUs have been vectorizing this exact workload for two decades, through progressively wider SIMD (single-instruction, multiple-data) units:

– **SSE** (128-bit): processes 4 float32 values per instruction.
– **AVX / AVX2** (256-bit): 8 float32 values per instruction; AVX2 adds fused multiply-add (FMA), where one instruction performs `a*b+c` — the atomic operation of every distance kernel.
– **AVX-512** (512-bit): 16 float32 values per instruction, with two FMA units per core on many server parts — up to ~32 float32 FLOPs per clock cycle, versus 2 for a scalar loop. On paper that is a 16× throughput advantage over scalar code.
– **AVX-512 VNNI / AMX** (newer Xeons): packed 8-bit integer or bf16 dot products, relevant when you quantize embeddings to int8/bf16, multiplying effective throughput further.

The catch is that compute is rarely the only bottleneck. Scanning 4 GB per query at realistic memory bandwidth (~50 GB/s per socket) puts a ~80 ms floor on the naive flat scan that no vector width removes — which is exactly why real engines use IVF to shrink the candidate set and HNSW to replace brute force with graph traversal, and why cache blocking, prefetching, and data layout matter as much as SIMD width. In practice, on one modern core, AVX-512/FMA brings the *compute* cost of a large similarity scan down to roughly a tenth of scalar execution — then memory becomes the wall you actually hit.

This is why the normalization trick matters beyond correctness. Pre-normalize every vector at index time, normalize the query once, and cosine similarity *is* a plain inner product — every engine then runs each comparison as a single fused dot product with no per-vector square root or division. That drops you straight onto the tightest, most SIMD-friendly kernel the hardware offers. Libraries differ in how aggressively they exploit this: FAISS ships hand-vectorized kernels with runtime CPU dispatch, hnswlib and USearch use SSE/AVX/AVX-512 paths (USearch auto-detects and dispatches), and pgvector and Milvus’s Knowhere do the same natively. Whichever you choose, measure on your own data and CPU — AVX-512 can carry a frequency penalty on some parts, and dispatch bugs occasionally fall back to scalar silently.

## The Decision Framework: OpenAI vs. Cohere vs. BGE

With the math in place, the metric question reduces to three checks: **Is the embedding normalized? Is magnitude meaningful? What does the model card actually recommend?** Here is how the three dominant embedding families answer:

| Embedding family | Typical dims | Normalized by default? | Recommended metric | Production note |
|—|—|—|—|—|
| **OpenAI** text-embedding-3-small / 3-large | 1536 / 3072 (Matryoshka-reducible to 256–1024) | No — API returns raw vectors | Cosine (per OpenAI docs) | Normalize client-side, then inner product ranks identically and runs faster |
| **Cohere** embed-english-v3.0 / embed-multilingual-v3.0 | 1024 | **Yes** — unit-length output | Dot product | Since vectors are unit-normed, dot product ≡ cosine; use inner-product kernels |
| **BGE** bge-base/-large-en-v1.5, bge-m3 | 768 / 1024 / 1024 | No | Cosine (per model card) | Prepend the retrieval instruction prefix; L2-normalize, then inner product in FAISS/Milvus |

**OpenAI.** The `text-embedding-3` family is not normalized at the API boundary, and OpenAI’s own guidance recommends cosine similarity. Two consequences follow. First, store raw vectors with dot product and you inherit a length bias you never asked for. Second, the Matryoshka `dimensions` parameter lets you truncate to, say, 512 dims for cheaper storage and faster scans — since truncation preserves relative ordering reasonably well, re-rank the top few hundred with the full-dimension cosine.

**Cohere.** The v3 models return unit-length embeddings by construction — the API normalizes for you. That single fact dissolves the whole debate: on unit vectors, dot product and cosine produce *identical rankings*, so search with the plain inner product and let your engine run its fastest kernel. If a teammate “helpfully” re-normalizes Cohere vectors, nothing breaks — just check you are not dividing twice or storing unnormalized copies in a mixed index.

**BGE.** BAAI’s models (including bge-m3) are trained with cosine-oriented objectives and the model card says cosine — but the model card *also* tells you to prepend `”Represent this sentence for searching relevant passages:”` to short queries, and that instruction is the higher-leverage knob. For ANN search, the standard production pattern is: encode with the instruction, L2-normalize both sides, then index with inner product — you keep cosine’s exact ranking while unlocking the fastest kernels.

If you are not on one of these families: (1) check whether the training objective or model card implies normalized embeddings or magnitude-meaningful scores; (2) measure ranking agreement between cosine and inner product on ~1,000 held-out queries — and if a reranker or threshold relies on absolute scores, agreement on *order* is not enough; (3) pick the fastest kernel among the metrics that agree, and never let a library default decide for you. Benchmark on *your* chunks and query distribution with recall@k and MRR — top-1 accuracy on a toy eval set will lie every time.

## The Bottom Line

The distance metric is not a config detail; it is a modeling assumption about what your embeddings mean. Cosine says direction encodes semantics and magnitude is noise. Dot product says magnitude encodes signal — popularity, confidence, engagement. Euclidean says the space is metric in the raw-coordinate sense, which is only true after normalization. On unit vectors, all three agree on ranking and the decision becomes purely about kernel speed and score calibration. Off the unit sphere, they disagree — silently, and in ways that poison retrieval quality without ever raising an error.

At OnlineCloud, we build production retrieval systems for teams who are done guessing: evaluated RAG pipelines, vector indexes sized and tuned to real hardware, and agent architectures wired into your actual databases. If you want your embedding stack audited — or you are shipping a retrieval system next quarter and want the metric, index, and eval strategy decided by engineers who have done it in production — we should talk.

**[Start a conversation with OnlineCloud](https://www.onlinecloud.dev/)** — free initial consultation, US-based senior engineers only, no offshore handoffs, response within 24 hours at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev).

# Multi-Tenant MCP Architecture: Secrets Isolation & Environment Sandboxing


*By the security engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: Scaling MCP Across Hundreds of Client Environments

The Model Context Protocol (MCP) gave every LLM host a standard way to discover and invoke tools, resources, and prompts. But the protocol deliberately says nothing about *tenancy*. There is no tenant identifier in a tool call, no boundary in the wire format, and no notion of “who pays for this inference.” That is the platform engineer’s job — and it becomes existential the moment you run one MCP fleet for more than one customer.

The failure mode is not exotic. It is the platform that starts as a demo — one desktop config pointing at one server with one database — and then wins six enterprise pilots. Six becomes sixty, then six hundred agents per day, each with its own Postgres, its own S3 bucket, its own API keys into the customer’s Salesforce, Snowflake, and Stripe. The naive move is one server per tenant: correct, but unreachable at fleet scale. A hundred tenants mean a hundred deployments, a hundred patching windows, a hundred drift-prone IAM roles. The platform move is **shared control planes with isolated data and execution planes** — isolation that is structural, not procedural.

Everything below answers one question first: **what breaks if this layer is the only thing standing between tenant A and tenant B’s data?** If the answer is “everything,” the layer is load-bearing and must be enforced by the platform, never requested of the model. We cover secrets isolation, network sandboxing, protocol-layer session boundaries, tool sandboxing, and governance. The recurring theme: **every boundary must be a mechanism, not a convention.**

## 2. Secrets Isolation: AWS Secrets Manager vs. HashiCorp Vault per Tenant

Start with the most dangerous object in the system: the credential. A multi-tenant MCP server holds hundreds — the customer’s warehouse password, their model-provider key, their OAuth refresh tokens into a SaaS estate. The cardinal rule is that **no tenant secret may ever be decrypted by code reachable from another tenant’s session.** That eliminates the two most common architectures at once: a shared environment-variable bag, and a shared secrets table with a `tenant_id` column. A table with a tenant column is a *filter*, not a boundary — every rotation job, debug endpoint, or SQL error becomes a potential cross-tenant read. Secrets management in a multi-tenant platform is not about storage; it is about **decryption proximity**: not where the ciphertext lives, but which principal can decrypt it, and which session can invoke that principal.

### AWS Secrets Manager: the VPC-adjacent default

The clean pattern is a **per-tenant KMS key, a per-tenant secret, and a tenant-scoped IAM role** the MCP server assumes only while serving that tenant’s traffic:

“`json
{
“Statement”: [{
“Effect”: “Allow”,
“Action”: “secretsmanager:GetSecretValue”,
“Resource”: “arn:aws:secretsmanager:us-east-1:123456789012:secret:tenant-acme/*”
}, {
“Effect”: “Allow”,
“Action”: “kms:Decrypt”,
“Resource”: “arn:aws:kms:us-east-1:123456789012:key/tenant-acme”
}]
}
“`

The KMS key is the load-bearing wall: even a misconfigured IAM policy that lets a role *read* the secret blob fails at `kms:Decrypt` without the tenant key, and every fetch lands in CloudTrail with the role ARN for per-tenant forensics. Watch the cost model at scale, though — Secrets Manager bills per secret and per 10,000 calls, so cache values in memory with a short TTL, force-refresh on rotation events, and budget hard against uncached fetches.

### HashiCorp Vault: when the boundary must be dynamic

Vault earns its place when tenancy is more dynamic than “one static key per customer.” With **dynamic database credentials**, the server never holds a standing password at all: when tenant A’s session needs warehouse access, the server calls Vault with a policy-bound token, and Vault mints a **unique, time-boxed Postgres credential for that session**, revoking it when the session ends. A leaked secret’s blast radius drops from “the customer’s password, forever” to “a synthetic credential valid for fifteen minutes, usable only from the tenant’s network path.” For long-running agent sessions, that is the difference between a reportable breach and a non-event:

“`hcl
path “database/creds/acme-warehouse” {
capabilities = [“read”]
}
“`

The trade-offs are real. You now operate Vault — an HA stateful service with unseal ceremony and upgrade discipline, and the most-targeted service in your estate. Our heuristic: choose **Secrets Manager when tenancy is stable and cloud-native fits**; choose **Vault when you need ephemeral, session-scoped credentials** — mandatory for regulated customer data where standing passwords are unacceptable. Either way, the platform rule is identical: **the tenant boundary is the decryption key, never the application code.**

## 3. Network Sandboxing: VPC Endpoints and Security Groups

Secrets isolation answers “who can decrypt.” Network isolation answers “who can even *reach* the thing.” We build it with two primitives: **security groups as tenant firewalls** and **VPC endpoints as private, metadata-free ingress**.

The architecture is a **hub-and-spoke topology**. The MCP control plane (shared servers, session registry, rate limiters) lives in a hub VPC; each tenant gets a **spoke** — or a dedicated account for regulated workloads — containing only their data plane: Postgres, S3, warehouse connections. Spokes have no Internet Gateway, no NAT, no public subnets. The only path in is a peering or Transit Gateway attachment to the hub, filtered by a security group that references the hub’s MCP group ID — a *group*, not a CIDR:

“`
┌──────────── HUB VPC ────────────┐
│ ALB ─► MCP ASG (sg-mcp) │
│ │ │ │
│ Session Registry Secrets │
│ (sg-control) (sg-sec) │
└───────┬──────────┬──────────────┘
peer │ │ peer
┌───────▼──────┐ ┌─▼─────────────┐
│ Spoke A │ │ Spoke B │
│ sg-tenant-a │ │ sg-tenant-b │
│ Postgres/S3 │ │ Postgres/S3 │
└──────────────┘ └───────────────┘
“`

Tenant A’s database group allows inbound **only from `sg-mcp`**, only on the DB port, only over the peering link. Tenant B’s group is identical in shape, entirely disjoint in effect. Because rules reference groups rather than IPs, autoscaling the fleet never breaks tenant connectivity — and because there is no spoke-to-spoke route, a compromised spoke cannot pivot laterally.

The second primitive is **VPC endpoints for every service the tenant path touches** — S3, Secrets Manager, STS, DynamoDB. If the MCP server reached S3 over the public endpoint, every object fetch would transit the internet, outside the scope of your security groups and flow logs. Endpoints keep tenant data-plane traffic on the AWS backbone, inside your routing table, where policy applies uniformly. Two rules complete the posture: **never put a NAT gateway in a spoke** (spoke egress is how exfiltration happens), and **enable VPC Flow Logs on every attachment**, aggregated where agents cannot reach them — when an incident asks “did tenant A’s session touch tenant B’s bucket?”, flow logs are the only answer that survives broken application logging.

## 4. Protocol Layer Isolation: Enforcing `Mcp-Session-Id` Boundaries

Secrets and networks isolate at the infrastructure layer. But the MCP server is shared code — one process namespace, one connection pool, one log buffer — so the protocol layer must enforce a boundary no infrastructure policy can: **every tool call executes in exactly one tenant session, and the server proves it on every request.**

The boundary token is the session. In the MCP Streamable HTTP transport, the server issues a session identifier that the client returns on every request — commonly `Mcp-Session-Id`. The platform rule: **`Mcp-Session-Id` is a capability, not a claim.** The server must never trust the header alone; it resolves the session server-side, verifies its authenticated principal, and binds *everything* downstream — the secret-decryption role, the database connection, the tool allowlist, the rate-limit bucket — to the resolved session, never to header-derived strings:

“`
1. Resolve sess_9f3c… in the session registry
→ tenant: acme, principal: ops-bot, scopes: [salesforce:read]
2. Unknown / expired / revoked → 401, no handler runs
3. Bind tenantID = acme (server-side, not from input)
4. Open acme’s DB connection pool; assume acme’s decrypt role
5. Execute tool against acme’s allowlist; write audit event
“`

Three enforcement details separate a real boundary from theater. **First, no cross-session state in the handler:** connection pools are per-tenant, caches key by the *resolved* tenant ID, and logging redacts anything outside it — a bug that lets one session read another’s cache entry is a leak no network policy catches. **Second, sessions are short-lived and revocable:** an agent session can run for hours, so the registry (Redis or DynamoDB) carries a TTL refreshed on activity, and revocation events — rotated credentials, terminated employees, abuse thresholds — kill every live session for a tenant in seconds. **Third, IDs are opaque, high-entropy, and tombstoned:** a 128-bit random value encoding no tenant identity means the header is useless without registry access, and closed sessions fail closed on replay instead of resurrecting dead context. Write the server as if the header is attacker-controlled — on any shared host, it is.

## 5. Tool Sandboxing: gVisor or Firecracker for Untrusted Code

The layers so far isolate tenants from *each other*. Tool sandboxing isolates the platform from the **tools themselves** — and from the model invoking them. This matters most once tenants register custom tools, or the server executes code on behalf of a prompt-injectable LLM. The rule: **no tenant code ever executes in the MCP server process, or even on the same kernel.** Two technologies dominate, at different points on the security-to-overhead curve.

**gVisor** is a user-space kernel that intercepts the application’s syscalls and implements them in Go behind a layer that distrusts the host kernel’s syscall surface. It runs as a normal process via `runsc`, boots in milliseconds, and drops into containerd as a runtime class — operationally light. The pattern is one sandbox per tool invocation or short session: read-only root filesystem, no network by default, tmpfs for scratch. Its sweet spot is **high-frequency, low-privilege calls** — data transforms, retrieval steps, schema introspection — at thousands per minute where virtualization would be uneconomical. Its trade-off: it shares the host kernel’s posture, a strong boundary against application exploits but not against a hostile host.

**Firecracker** is a microVM — a minimal VMM booting a stripped Linux guest in ~125 ms with under 5 MiB of overhead. Each sandbox is a *real VM* with its own kernel and device model and no shared syscall surface: the right answer when tools run **tenant-supplied or model-generated code with network access to customer data**, where a sandbox escape would mean cross-tenant compromise. Its cost is operational — guest images, kernel updates, and snapshotting to keep a warm pool.

The platform pattern is a **two-tier execution model**: Tier 1 (gVisor, per call) for built-in trusted tools at tens-of-milliseconds latency; Tier 2 (Firecracker, per session) for tenant-registered or code-generating tools, each microVM wired into its own tenant’s network namespace — plugging straight into the VPC endpoints and security groups above, so even a fully compromised sandbox can only reach its own spoke. Whichever tier, three rules hold: **no outbound network by default** (egress only via an audited allowlist), **no host filesystem or Docker socket** (all I/O through the session-bound MCP tool interface), and **hard wall-clock and memory caps enforced by the sandbox** — a tool that can run forever is a tool that can mine cryptocurrency on your neighbor’s capacity, and “the model said stop” is not enforcement.

## 6. Governance: Multi-Tenant Cost Attribution and Rate-Limit Enforcement

Isolation without governance is unmanageable; governance without isolation is unfair. The final layer answers two business questions — **who pays, and who is hogging the platform** — from the same session-bound telemetry the security layers already produce.

**Cost attribution** starts with the session as the cost unit. Every invocation emits an event carrying the resolved tenant ID, tool, sandbox tier, wall-clock time, CPU, memory, tokens, and downstream calls (secrets lookups, database queries, model inference). Aggregated per tenant per day, that stream becomes a cost report — but only if tenant ID is **unforgeable in telemetry**, taken from the same resolved-session context as the security layers, never from client-supplied fields. Direct attribution (tenant → tool → resource) covers the common case; **shared-pool attribution** covers the control plane no single tenant caused — a platform fee plus usage-weighted allocation, recomputable from the audit log so finance trusts it and tenants can dispute it in the open.

**Rate limiting** is where governance meets availability, and multi-tenant makes naive limits dangerous. A global limit lets one tenant starve everyone; frozen per-tenant limits waste idle capacity. The workable design is token buckets enforced **at the session router**, keyed by resolved tenant, at three horizons:

“`json
{
“tenant”: “acme”,
“rps”: { “capacity”: 50, “refillPerSec”: 5 },
“concurrent”: { “max”: 20 },
“tokensPerDay”: { “capacity”: 500000 }
}
“`

A per-second burst cap; a concurrency cap that doubles as the memory-exhaustion defense (each concurrent session pins sandbox or connection resources); and a daily token budget that maps directly onto the billing ledger. Enforcement lives at the **router, before expensive work begins**, and an over-limit tenant gets a structured 429 with `Retry-After` so their agents back off gracefully. Limits must also be **adaptive, not static**: near the daily budget, throttle or queue rather than kill mid-task; with idle fleet capacity, let a well-behaved tenant burst. Static limits are a policy; adaptive limits driven by the same telemetry that feeds billing are a platform.

Close the loop by writing every rejection, throttle, and sandbox denial into the same immutable audit store as the security events — governance data and security data are the same data from different seats. A tenant spiking to 10x normal volume is either scaling up or exfiltrating, and the SOC and the billing team should be watching the same dashboard when it happens.

—

**Building a multi-tenant MCP platform — or retrofitting isolation into one that already shipped? Talk to the OnlineCloud architects.** We are a US-based engineering studio — MCP-native, senior engineers only, no offshore handoff — and we deliver session-bound secrets isolation, VPC-sandboxed data planes, and audited cost governance in production. Contact us at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/) for a free initial consultation. We respond within 24 hours.

# Implementing Asynchronous Tool Execution: Handling Multi-Minute Jobs in MCP

*Deep-dive for engineers who need MCP tool calls to survive what synchronous RPC cannot: ETL jobs, batch inference, and data exports that run for minutes while agents, proxies, and humans wait.*


—

## 1. Introduction: The Failure of Synchronous JSON-RPC for Long-Running Agent Tasks

Model Context Protocol is JSON-RPC 2.0 under the hood: a client sends a `tools/call` request with an `id`, and the server must eventually answer *that* `id` with a `result` or an `error`. The correlation is elegant and stateless-friendly — and completely wrong as a contract for work that takes minutes.

Consider what actually happens when a tool call runs long. Most MCP deployments terminate at an HTTP boundary — an nginx `proxy_read_timeout` defaulting to 60 seconds, a cloud load balancer dropping idle connections, an API gateway enforcing its own ceiling. If the job survives the network, the agent harness is next: orchestrators impose per-turn and per-tool timeouts so a stuck call cannot consume a session forever. Over stdio the problem is worse — the server process is pinned to the client that spawned it, with no way to shed load. Three independent layers — proxies, hosts, and the LLM application loop — all assume a tool call resolves in seconds.

The workloads enterprises actually want agents to run violate that assumption exactly: exporting two million CRM rows, regenerating embeddings for a document corpus, backfilling a normalized schema across Oracle and MS SQL. Done synchronously, they turn a crisp RPC protocol into a parking lot: the connection idles, the agent stalls mid-plan, and every layer above interprets the silence as failure.

The fix is architectural: stop treating *tool execution* and *job completion* as the same event. **A tool call should return a decision quickly; the work continues outside the request/response lifecycle.** The client gets an opaque job handle, the server tracks a durable state machine, and progress and results travel on channels designed for asynchronous delivery. This is also the direction the protocol itself is moving — the 2025-11-25 MCP revision introduced **Tasks**, an experimental but SDK-supported primitive for “call now, fetch later” execution. Tasks standardized what production teams had already built by hand: job IDs, status polling, progress notifications, and a state store that outlives any single connection. That pattern is what this post is about.

—

## 2. MCP Async Patterns: Job IDs and Progress Notifications

Before any fancy queuing, two protocol mechanisms do most of the work: *progress tokens* and *job handles*.

### Progress tokens and `notifications/progress`

MCP has supported progress reporting since its earliest revisions. A client that wants progress attaches a token to request metadata:

“`json
{
“jsonrpc”: “2.0”,
“id”: 1,
“method”: “tools/call”,
“params”: {
“name”: “run_export”,
“arguments”: { “source”: “crm”, “destination”: “warehouse” },
“_meta”: { “progressToken”: “tok-export-41” }
}
}
“`

The server then emits zero or more notifications referencing that token:

“`json
{
“jsonrpc”: “2.0”,
“method”: “notifications/progress”,
“params”: {
“progressToken”: “tok-export-41”,
“progress”: 51234,
“total”: 2000000,
“message”: “row batch 26 of ~100”
}
}
“`

Three spec rules matter operationally: progress values **must** increase monotonically, notifications **must stop** at completion, and tokens **must be unique across active requests** — monotonic counters collide across sessions, so use UUIDs or ULIDs. Under Tasks, a progress token stays valid for the entire task lifetime, so this plumbing carries forward.

### Job handles: making the tool return fast

A progress token alone does not make a tool asynchronous — the server still holds the JSON-RPC response hostage until the work ends. The job-handle pattern breaks that coupling: the tool returns immediately with a description of the job, not the result:

“`json
{
“jsonrpc”: “2.0”,
“id”: 1,
“result”: {
“content”: [
{
“type”: “text”,
“text”: “Job 01J5QX2F3X0VX0 accepted. Poll get_job({\”jobId\”:\”01J5QX2F3X0VX0\”}) or wait for notifications/progress. Retained 24h.”
}
],
“structuredContent”: {
“jobId”: “01J5QX2F3X0VX0”,
“status”: “queued”,
“pollIntervalMs”: 5000,
“ttlMs”: 86400000
}
}
}
“`

The JSON-RPC exchange now completes in milliseconds, and the job lives in a namespace the client can interrogate: a `get_job` tool, a `job://` resource URI, or — under Tasks — the standardized `tasks/get`, `tasks/result`, `tasks/cancel`, and `tasks/list` methods over a `working` / `input_required` / `completed` / `failed` / `cancelled` state machine. Job IDs gate access to state and results: generate them unguessable, bind them to the creating session’s auth context, and enforce that binding on every follow-up call — a `tasks/get` from another tenant should fail as if the job never existed.

—

## 3. State Polling vs. Callbacks: Designing a Robust Event-Driven Tool Execution Loop

Given a job handle, how does the agent learn the job finished? Two mechanisms exist, with complementary failure modes.

**Polling** is requestor-driven and authoritative. The client calls `get_job` or `tasks/get` on an interval, and the server returns ground truth from durable storage. Polling survives everything: dropped connections, missed notifications, server restarts, reconnects. Its costs are latency (you discover completion up to one interval late), agent turns and tokens, and server load — hence the server-suggested `pollInterval` and backoff as jobs age.

**Callbacks** — progress and status notifications pushed over the live stream — are receiver-driven and best-effort. They give near-zero latency and keep the UI alive, but are fire-and-forget: a notification emitted while the client is disconnected is gone, and one racing a reconnect can be delivered twice or not at all. Notifications can **never** be your source of truth, because delivery is not guaranteed.

The robust design is therefore not a choice but a combination: **poll for truth, listen for speed.** The client’s loop looks like this:

“`
job = call(“run_export”, {progressToken}) # returns immediately
while job.status not in {completed, failed, cancelled}:
wait_for_notification(timeout = job.pollIntervalMs)
if timeout or reconnect: # notification lost or stream dead
job = call(“tasks/get” / “get_job”, {jobId})
else:
apply_status_update(notification.payload) # reconcile, never trust alone
result = call(“tasks/result” / “get_job”, {jobId}) # exactly the sync result, delayed
“`

Two properties make this loop robust: **notifications are an acceleration layer over a pollable state** (every message is idempotent to apply, every missed message recoverable), and **the loop always terminates by polling** — the terminal fetch is an authoritative read, so no duplicate or reordered notification can corrupt the outcome. Server side, keep the same discipline: state transitions are append-only (a `completed` task never returns to `working`, even if a late worker event says so), and terminal results stay retrievable long after notifications stop.

—

## 4. Architecture: Using Redis Pub/Sub or SQS for Background Job Status Propagation

The job-handle pattern forces a topology change. If the MCP server executes tools inline, async buys you nothing — the process is still blocked. In production you split the system: **gateway nodes** own MCP sessions, streams, and protocol state, while a **worker fleet** executes jobs. They scale independently — the gateway is I/O-bound and chatty, the worker CPU- and database-bound. Between them you need a *queue* for dispatch, a *bus* for status propagation back to the session owner, and a *state store* as the authoritative record of every job.

Redis and SQS are the two workhorses for queue-and-bus, and choosing between them is a durability-vs-latency trade, not a religious one.

**Redis Pub/Sub** gives single-digit-millisecond fan-out. Workers publish progress events to a channel; gateway nodes subscribe and forward them as `notifications/progress` to whichever session owns the job. It is operationally simple — you almost certainly already run Redis — and its fire-and-forget semantics suit notifications, which are best-effort by protocol design. The trap is treating it as a system of record: messages with no subscribers are dropped, a restarting gateway misses every event published while down, and nothing is persisted or replayable. So with Redis the rule is: **use Pub/Sub only for the delta channel, and keep authoritative job state in Redis keys with TTLs** (a `SET` for status, `HSET` for counters, `EXPIRE` for retention). A reconnecting client polls the keys, not the pub/sub history.

**SQS** (or any durable queue) inverts the guarantees. Messages persist, survive restarts, replay on failure, and move across AZs; the visibility-timeout mechanism doubles as a natural worker lease — a worker that dies mid-job releases the message for retry, and DLQs catch poison jobs. The costs are latency (hundreds of milliseconds minimum per hop), no native fan-out (pair SNS topics with queue subscriptions), and at-least-once delivery, which *forces* idempotent workers. When the job is an enterprise data export that auditors will ask about — or when a lost completion event means re-running a two-million-row extract — durability wins.

| Decision | Redis Pub/Sub | SQS (+ optional SNS) |
|—|—|—|
| End-to-end latency | ms | 100ms–seconds |
| Message persistence | None | Durable, replayable |
| Delivery semantics | At-most-once (fire & forget) | At-least-once |
| Restart recovery | Reconcile from state store | Replay from queue |
| Poison handling | Manual (state store flags) | Native DLQ |
| Ops footprint | You operate it | Managed |
| Best for | Live progress fan-out, low-latency UX | Job dispatch, compliance, cross-AZ reliability |

The reference topology combines both: a queue for dispatch, a durable state store as the source of truth, and a Redis Pub/Sub channel for low-latency progress fan-out to connected gateways. The MCP client never touches the queue — it talks to the gateway, which translates internal events into protocol notifications and answers polls from state. That indirection lets one surface serve stdio, SSE, and Streamable HTTP clients without leaking infrastructure.

—

## 5. Error Handling: Managing Timeouts and Partial Failures in Long-Lived Sessions

An async job multiplies your failure surface, because every handoff is a new place to lose work. Enumerate them: the request never reaches the queue; a worker crashes mid-job; the state store evicts the job before the client fetches it; a notification is dropped; the session disconnects and reconnects under a new transport; the *agent* times out even though the job is healthy; and — the nastiest case — the job *partially* fails, completing 48 of 50 shards.

**Idempotency is the foundation.** If a client retries a timed-out submission, you must not spawn a second export of the same two million rows. Accept an idempotency key (or derive one from a stable hash of tool name and arguments) and return the *same* job ID for the same logical submission. Tasks leaves idempotency to implementers — real networks do not — so build it yourself: check the key before enqueueing, and make the check and enqueue atomic.

**Leases prevent zombie work.** Every executing job carries a lease the worker must renew by heartbeat. Redis: `SET job:ID lease NX EX 60`, renewed every 20 seconds; a worker that dies stops renewing, the lease expires, and another worker re-claims the job. SQS gives you this for free via the visibility timeout. Add a fencing token — the worker writes its token into every state update, and a stale worker whose lease was reclaimed loses the write — so two workers can never both believe they own the export.

**Decouple the agent’s patience from the job’s lifetime.** The client’s tool-call timeout firing is *information*, not a cancellation order. The job keeps running, and the result must be retrievable after the client reconnects — which is exactly why `tasks/list` exists: a restarted agent rediscovers in-flight work instead of orphaning it. Retain terminal results for the TTL the server committed to (hours to days, not seconds), and make terminal states immutable: once `completed` or `failed`, no late event may move it.

**Model partial failure honestly.** A batch job that fails on 2 of 50 shards should not fail the whole job. Return a structured result that separates protocol errors from execution outcomes:

“`json
{
“content”: [{ “type”: “text”, “text”: “Export completed with 48/50 shards.” }],
“structuredContent”: {
“status”: “completed”,
“partial”: true,
“completed”: 48,
“failed”: [{ “shard”: “orders_2019”, “error”: “ORA-04030: out of process memory” }],
“retryCommand”: “run_export”
}
}
“`

The client can retry exactly the failed shards with the same idempotency key — cheap, safe, visible recovery. Under Tasks, execution failure is `status: failed` with diagnostics, while malformed requests stay JSON-RPC errors; keeping those layers separate keeps error handling composable. And put metrics on the loop — enqueue latency, lease expiry rate, time-to-terminal — because the earliest symptom of trouble in a job system is a metric, not an error.

—

## 6. Implementation: An Async MCP Tool with Progress Updates in Node.js

Here is the pattern in one Node.js sketch: an MCP gateway process (Express + the official SDK over SSE) and a worker sharing a Redis store. The tool returns in milliseconds; the worker does the minutes-long work; progress flows back as `notifications/progress`.

“`js
// gateway.mjs — MCP server exposing an async tool
import { McpServer } from “@modelcontextprotocol/sdk/server/mcp.js”;
import { SSEServerTransport } from “@modelcontextprotocol/sdk/server/sse.js”;
import { createClient } from “redis”;
import { z } from “zod”;
import { createId } from “@paralleldrive/cuid2”;

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const server = new McpServer(
{ name: “data-mcp”, version: “1.0.0” },
{ capabilities: { tools: {} } }
);

// In-memory: activeToken -> { transport, sessionId }
// Used to forward progress to the right live session.
const tokenIndex = new Map();

server.registerTool(
“run_export”,
{
source: z.enum([“crm”, “erp”, “datawarehouse”]),
destination: z.string(),
},
async ({ source, destination }, extra) => {
const jobId = createId(); // unguessable, sortable
const progressToken = extra._meta?.progressToken ?? `job:${jobId}`;

// Durable state: source of truth for polling/reconnect.
await redis.hSet(`job:${jobId}`, {
status: “queued”, progress: “0”, destination,
token: progressToken, createdAt: Date.now(),
});
await redis.expire(`job:${jobId}`, 86_400); // 24h retention

// Keep a mapping so the worker can find this session later.
tokenIndex.set(progressToken, jobId);

// Dispatch — worker picks this up within ms.
await redis.lPush(“jobs:pending”, JSON.stringify({
jobId, source, destination, progressToken,
}));

return {
content: [{
type: “text”,
text: `Queued export job ${jobId}. Poll get_job, or expect progress notifications.`,
}],
structuredContent: { jobId, status: “queued”, pollIntervalMs: 5000, ttlMs: 86_400_000 },
};
}
);

// Worker progress events arrive here; forward as MCP notifications.
// Gateway subscribes once and routes by token -> live session.
async function forwardProgress(channel, raw) {
const evt = JSON.parse(raw);
const job = await redis.hGetAll(`job:${evt.jobId}`);
const session = tokenIndex.get(job.token); // resolve to live stream
if (!session) return; // client disconnected — state store covers recovery
session.transport.session.sendNotification({
method: “notifications/progress”,
params: { progressToken: job.token, progress: evt.progress, total: evt.total, message: evt.message },
});
}
await redis.subscribe(“job:events”, forwardProgress);

// get_job: the poll side of “poll for truth, listen for speed”
server.registerTool(“get_job”, { jobId: z.string() }, async ({ jobId }) => {
const job = await redis.hGetAll(`job:${jobId}`);
if (!job?.status) throw new Error(`Unknown job: ${jobId}`);
return { content: [{ type: “text”, text: JSON.stringify(job) }] };
});

// HTTP wiring (SSE transport) — abbreviated, see companion post on SSE.
const sessions = new Map();
import express from “express”;
const app = express();
app.get(“/sse”, async (req, res) => {
const t = new SSEServerTransport(“/messages”, res);
sessions.set(t.sessionId, t);
await server.connect(t);
});
app.post(“/messages”, express.json(), async (req, res) => {
const t = sessions.get(String(req.query.sessionId));
if (t) await t.handlePostMessage(req, res);
});
app.listen(3001);
“`

“`js
// worker.mjs — drains jobs, updates state, publishes progress
import { createClient } from “redis”;
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

while (true) {
const [, raw] = await redis.brPop(“jobs:pending”, 0); // blocking pop
const { jobId, source, destination, progressToken } = JSON.parse(raw);

const lease = await redis.set(`lease:${jobId}`, workerId, { NX: true, EX: 60 });
if (!lease) continue; // another worker claimed it — idempotent skip

try {
await redis.hSet(`job:${jobId}`, { status: “running”, worker: workerId });
const total = await countRows(source);
for await (const [i, batch] of streamBatches(source, destination)) {
await renewLease(jobId); // heartbeat
await redis.publish(“job:events”, JSON.stringify({
jobId, progress: i, total, message: `batch ${i}/${total}`,
}));
await insertBatch(batch); // the actual minutes-long work
}
await redis.hSet(`job:${jobId}`, { status: “completed”, progress: total });
await redis.publish(“job:events”, JSON.stringify({ jobId, done: true }));
} catch (err) {
await redis.hSet(`job:${jobId}`, { status: “failed”, error: String(err) });
// or push to a retry/DLQ list with the idempotency key intact
} finally {
await redis.del(`lease:${jobId}`);
}
}
“`

On the client side, attach `_meta.progressToken` and treat `notifications/progress` as a hint layer only — always confirm the terminal state via `get_job` before acting. If your SDK has adopted Tasks, the shape maps directly: send `task: { ttl }`, poll `tasks/get`, and block on `tasks/result` for the final payload; the durability, lease, and idempotency machinery above is unchanged.

—

## Conclusion

Synchronous JSON-RPC is the right tool for the wrong job the moment a tool call exceeds an HTTP proxy’s patience. The fix is architectural: keep the RPC fast by returning a job handle, make the job a durable state machine in a store the client can poll, and use notifications only as an acceleration layer over that truth. Queue the work for a fleet that outlives any connection, lease it so crashes cannot orphan it, and make retries idempotent so a client cannot double-run an export. That combination — fast ack, durable state, event-driven progress, pollable truth — is what turns MCP from a demo protocol into something that drives real enterprise workloads for minutes at a time.

This is the kind of architecture that looks simple on a whiteboard and gets subtle in production: session ownership across a gateway fleet, state stores, idempotency keys, and knowing which channel is authoritative. It is what the senior engineers at OnlineCloud build every day — MCP-native systems with Redis and AWS at the core, audited end-to-end, shipped in weeks, not quarters. If you are putting long-running tools behind MCP, **talk to the OnlineCloud architects** — we would rather argue about your lease timeout than sell you a deck. [Contact us](https://www.onlinecloud.dev/) or write to contact@onlinecloud.dev — free initial consultation, response within 24 hours.

# MCP to Oracle Implementation: Deep-Dive on VPD Policies at the Protocol Layer


*By the database security engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: Why Agentic SQL Access Is Inherently Dangerous

Before we talk about Virtual Private Database (VPD) policies, let’s be honest about the threat model that makes them necessary. An agent with SQL access is not a faster analyst. It is a new class of database principal, and it violates nearly every assumption the Oracle security model was designed around.

The first violation is *judgment*. A human analyst carries context: which cost center they belong to, which accounts they may touch, which tables are too sensitive to query casually. An LLM carries none of that. It has a system prompt, a tool description, and a stochastic sampler. Its notion of “the right rows” comes from whatever text was in its context window — including text an attacker injected through a fetched document, an email thread, or a poisoned web page. Hand an agent a connection string and a free-text SQL tool, and you have handed your prompt-injection attack surface the keys to your data dictionary.

The second violation is *impersonation*. Legacy application access is mediated by application code that authenticates the end user and then shapes every query around that identity. Agentic tool calls bypass the application entirely. The MCP (Model Context Protocol) server is the application now — and if it connects to Oracle with a single shared, privileged credential, then every end user behind the agent inherits the same database identity. Row-level ownership, tenancy boundaries, and segregation-of-duties controls silently stop meaning anything. A tool that answers “show me this customer’s invoices” for one user can, under a shared credential, answer it for every customer.

The third violation is *amplification*. SQL is a Turing-complete, set-oriented language where one malformed predicate can scan a billion rows, one omitted predicate can return a full table, and one `MERGE` can rewrite a ledger. A human who does that is caught by review. An agent that does it has already moved on — and may have serialized the entire result set into its context window, where it becomes training signal, cached memory, or a prompt-injection payload.

You cannot prompt your way out of this. The fix is a database that is physically incapable of returning a row the principal may not see — regardless of what the model asks or what string an attacker smuggled in. That is exactly what Oracle Virtual Private Database provides, and why VPD belongs at the heart of any production MCP-to-Oracle bridge.

## 2. Oracle Virtual Private Database: The Principles of FGAC

Virtual Private Database is Oracle’s implementation of **Fine-Grained Access Control (FGAC)**. The principle is disarmingly simple: a security policy is attached directly to a table or view, and Oracle’s kernel appends a predicate to *every* SQL statement that references the object — regardless of the tool, the driver, the connection pool, or the human (or model) behind it.

The policy is a PL/SQL function you write, registered through the `DBMS_RLS` package:

“`sql
— Policy function: returns the predicate Oracle appends to every statement
CREATE OR REPLACE FUNCTION mcp_sec.f_tenant_scope (
p_schema VARCHAR2, p_object VARCHAR2
) RETURN VARCHAR2 AS
v_tenant VARCHAR2(64);
BEGIN
v_tenant := SYS_CONTEXT(‘mcp_ctx’, ‘tenant_id’);
— FAIL CLOSED: no context, no rows. Never return NULL here.
IF v_tenant IS NULL THEN
RETURN ‘1 = 0’;
END IF;
RETURN ‘tenant_id = SYS_CONTEXT(”mcp_ctx”, ”tenant_id”)’;
END;
/

BEGIN
DBMS_RLS.ADD_POLICY(
object_schema => ‘FIN’,
object_name => ‘INVOICES’,
policy_name => ‘mcp_tenant_scope’,
function_schema => ‘MCP_SEC’,
policy_function => ‘f_tenant_scope’,
statement_types => ‘SELECT, UPDATE, DELETE’,
sec_relevant_cols => ‘AMOUNT,TAX_ID’,
policy_type => DBMS_RLS.CONTEXT_SENSITIVE
);
END;
/
“`

Three properties make this the right control for agentic access.

**Transparency.** The predicate is invisible to the caller. The agent issues `SELECT * FROM fin.invoices` and receives only its own tenant’s rows — no `WHERE` clause it can argue with, drop, or inject around. The model never knows the boundary exists, which means it cannot be social-engineered into crossing it.

**Enforcement inside the kernel.** VPD is not an application-layer filter; the predicate is compiled into the execution plan by the optimizer. No code path — tool, SQL*Plus session, or `EXECUTE IMMEDIATE` trick — reaches rows without passing through it. That matters because the model’s tool arguments are untrusted input; defense must live below the layer an attacker can influence.

**Column-level reach.** Through `sec_relevant_cols`, VPD can mask sensitive columns (replacing them with `NULL`) while leaving the rest of the row visible — useful when an agent legitimately needs invoice metadata but must never receive `TAX_ID` or PII columns.

One rule makes or breaks a deployment: *fail-closed discipline*. A policy function that returns `NULL` or an empty string when the context is missing disables restriction entirely — a guard quitting his post because nobody badged in. Every policy function must return `1 = 0` on missing context and treat a NULL context as an attack signal worth auditing, not a configuration convenience.

## 3. The MCP Integration: `DBMS_SESSION.SET_CONTEXT` Inside the Tool Invocation Loop

VPD predicates need a per-principal value to filter on, and Oracle’s mechanism for supplying that value is the **application context**: a namespace of name-value pairs associated with a database session, read by SQL through `SYS_CONTEXT`. Application contexts are how VPD deployments have bound end-user identity to row security — and the MCP tool loop is a new, unusually demanding caller of that machinery.

The binding happens inside the MCP server’s tool handler. The critical architectural decision is where the MCP Oracle connector sits relative to the LLM:

“`
LLM Host ──MCP (streamable HTTP / SSE)──▶ MCP Server (Oracle tools)
│
1. authenticate OAuth2 token
2. resolve principal → tenant_id
3. bind Oracle session context
│
▼
Oracle DB ──VPD predicate──▶ Rows
“`

Each tool handler in the MCP server follows a strict sequence: **authenticate, resolve, bind, execute, clear**. If your MCP server uses a session pool (and it should — connect cost alone rules out per-call sessions), the pooled session is a shared, reusable asset that must be re-scoped on every checkout:

“`java
// MCP tool handler — “get_customer_invoices”
ToolResult handle(Invocation inv) {
var claims = verifyOAuthToken(inv.principalToken()); // 1. authenticate
String tenantId = identityResolver.mapToTenant(claims.sub()); // 2. resolve
try (var conn = pool.getConnection()) { // 3. bind
conn.prepareCall(“{call mcp_sec.set_principal(?, ?, ?)}”)
.setString(1, claims.sub())
.setString(2, tenantId)
.setString(3, inv.requestId())
.execute();
try (var rs = conn.prepareStatement(inv.sql()) // 4. execute
.executeQuery()) {
return rowMapper.apply(rs);
} finally {
conn.prepareCall(“{call mcp_sec.clear_principal()}”).execute(); // 5. clear
}
}
}
“`

The PL/SQL side is where the security lives. A namespace can only be written by the package it is bound to (`CREATE CONTEXT … USING`), which means the database login used by the MCP pool **cannot forge a context attribute directly** — it can only invoke the package, and the package decides what it will accept:

“`sql
CREATE OR REPLACE CONTEXT mcp_ctx USING mcp_sec.set_principal;

CREATE OR REPLACE PACKAGE mcp_sec AS
PROCEDURE set_principal(p_sub VARCHAR2, p_tenant VARCHAR2, p_req VARCHAR2);
PROCEDURE clear_principal;
END;
/

CREATE OR REPLACE PACKAGE BODY mcp_sec AS
PROCEDURE set_principal(…) IS
BEGIN
— Server-side validation: never trust raw tool arguments.
— p_sub/p_tenant arrive only after OAuth verification at the MCP edge.
DBMS_SESSION.SET_CONTEXT(‘mcp_ctx’, ‘oauth_sub’, p_sub);
DBMS_SESSION.SET_CONTEXT(‘mcp_ctx’, ‘tenant_id’, p_tenant);
DBMS_SESSION.SET_CONTEXT(‘mcp_ctx’, ‘request_id’, p_req);
DBMS_APPLICATION_INFO.SET_MODULE(‘mcp:’ || p_req, ‘tool_execute’);
DBMS_SESSION.SET_IDENTIFIER(p_sub); — visible in Unified Audit
END;

PROCEDURE clear_principal IS
BEGIN
DBMS_SESSION.CLEAR_CONTEXT(‘mcp_ctx’);
DBMS_APPLICATION_INFO.SET_MODULE(NULL, NULL);
DBMS_SESSION.SET_IDENTIFIER(NULL);
END;
END;
/
“`

The `finally` block that clears the context is not hygiene — it is the security boundary between one agent user and the next. A pooled session retaining the previous principal’s context is a cross-tenant read channel waiting for a scheduling coincidence. Context bleeding of this kind is the most common VPD-in-pooling bug I have reviewed, and exactly what an MCP tool loop amplifies by multiplexing many end users over few sessions.

## 4. Security: From the OAuth2 Principal to the SQL Predicate

Row-level security is only as trustworthy as the identity chain that feeds it. VPD does not care what your OAuth2 token *claims* — it cares what the application context contains, because that is what the predicate compares against. Every hop in that chain — token claims → context attributes → SQL predicate — must be validated and must fail closed.

**Hop one: the token.** The MCP server terminates OAuth2 (for MCP hosts, the Authorization Code flow with PKCE per the MCP authorization specification) and verifies signature, issuer, audience, and expiry on *every* tool call — not once per connection. The claim that anchors the chain is `sub`, the stable principal identifier, which your identity provider guarantees is unique per human or service account.

**Hop two: claims to context.** The tool handler must never pass raw model-provided arguments into the context. The tenant and any scoping attributes are *derived server-side* from the verified `sub` by your identity resolver — the same directory that powers the rest of your authorization estate. An unmapped `sub` returns an authorization error, never a default “no filter.”

**Hop three: context to predicate.** Inside the policy function, `SYS_CONTEXT(‘mcp_ctx’,’tenant_id’)` is interpolated into a SQL predicate string. That string is assembled by *your* code, not user input — but treat it as injection-prone anyway. Validate attribute values with `DBMS_ASSERT` (or a whitelist regex) before embedding them, bind with literals only after validation, and always fail closed when the attribute is absent or malformed:

“`sql
IF NOT mcp_sec.validate_tenant_id(v_tenant) THEN
RETURN ‘1 = 0’; — invalid value is the same as no value: deny
END IF;
“`

**Why this defeats the agentic attack classes.** Prompt injection cannot manufacture rows, because the predicate is appended below any SQL the model can influence. Tool misuse cannot escalate scope, because even a “correct” tool call is filtered by the context of whoever’s token authorized it. And a compromised MCP server cannot become a tenant-wide read, because the database login itself holds no broad data privilege that VPD does not narrow — audit the login’s grants so that table privileges exist precisely *because* VPD constrains them, not despite it.

One additional hardening worth shipping: restrict the MCP database account so it can only reach protected tables through paths where the context has been set. A logon trigger that raises when a statement references a VPD-protected table while `mcp_ctx.tenant_id` is null turns “forgot to set context” from a silent data-classification incident into a loud, audited failure at connect time.

## 5. Audit Trails: Reconciling MCP Decision Logs with Oracle Unified Audit

VPD tells you what a principal *can* see. Audit tells you what the system *actually did* — and with an agent in the loop you need two independent records, because they answer different questions. **MCP decision logs** answer *why*: the tool name, the model’s arguments, the request ID. **Oracle Unified Audit** answers *what really happened at the data layer*: the exact SQL parsed, the database user, the client identifier, the bind values. Neither record suffices alone; the security value is in the *reconciliation*.

To make reconciliation possible, every MCP tool execution must stamp the database session with correlation identifiers that flow into the audit trail. In the package above, that is `DBMS_SESSION.SET_IDENTIFIER(p_sub)` — which Unified Audit records as `CLIENT_IDENTIFIER` — and `DBMS_APPLICATION_INFO.SET_MODULE(‘mcp:’ || request_id, …)`, recorded as `MODULE`/`ACTION`. The request ID is generated by the MCP host at conversation start and logged by the decision logger, so the join key exists end to end:

“`sql
CREATE AUDIT POLICY mcp_fin_access
ACTIONS SELECT, UPDATE, DELETE ON fin.invoices,
SELECT ON fin.customers
WHEN ‘SYS_CONTEXT(”USERENV”,”MODULE”) LIKE ”mcp:%”’
EVALUATE PER STATEMENT;

AUDIT POLICY mcp_fin_access;

— Reconciliation: did every audited statement have a logged decision?
SELECT ua.os_user, ua.client_identifier, ua.module, ua.sql_text,
ua.event_timestamp, dl.request_id
FROM unified_audit_trail ua
LEFT JOIN mcp_decision_log dl
ON dl.request_id = REGEXP_SUBSTR(ua.module, ‘mcp:([a-z0-9-]+)’, 1, 1, NULL, 1)
WHERE ua.module LIKE ‘mcp:%’
AND ua.event_timestamp > SYSTIMESTAMP – INTERVAL ‘1’ HOUR;
“`

The anomalies are where the investigation lives:

– **Audit rows with no matching decision log** mean SQL executed on the MCP credential that no tool call can explain — a policy bypass, a compromised server, a manual session using the same account, or a driver-side retry. All four are worth knowing about.
– **Decision logs with no matching audit rows** mean a tool claimed success the database never executed — a dropped connection, a misrouted request, or (worse) an audit policy that missed a table. Empty reconciliation results are how you discover your audit coverage gap.
– **Context and identity drift**: audit rows whose `CLIENT_IDENTIFIER` does not match the decision log’s principal indicate the context was stale or overwritten — the pooling bug from Section 3, caught in the wild.

For column-level sensitivity, layer `DBMS_FGA` fine-grained audit policies on top of the VPD-protected objects so that access to specific columns (tax identifiers, account numbers, PII) generates its own auditable events. Keep the MCP account out of `AUDIT_ADMIN` and `AUDIT_VIEWER`, set a retention policy on `UNIFIED_AUDIT_TRAIL` that survives the application’s own log rotation, and treat the decision log as a tamper-evident append-only stream (hash-chained or written to object storage with WORM retention). When an incident responder asks “what did the agent do and why,” the two logs answer in one join.

## 6. Performance: Evaluating the Latency Overhead of Session-Scoped Policies

VPD is not free, and an MCP tool loop multiplies the cost of getting the policy configuration wrong, because it runs many small statements across pooled sessions. In our benchmark harness — a 19c database, 10M-row `INVOICES` table partitioned by tenant with an index on `(tenant_id, invoice_date)`, an MCP server issuing point lookups through a 20-session pool — the overhead breaks down into three measurable components.

**Context binding.** A `DBMS_SESSION.SET_CONTEXT` call costs on the order of tens of microseconds. Done once per tool call (not per statement), it is noise — we measured ~0.1–0.3 ms added to a ~2 ms request. The failure mode is doing it *per statement* inside the tool handler: with agent loops issuing 5–10 statements per call, that compounds into visible tail latency for zero security benefit. Bind once at checkout, clear once at check-in.

**Predicate generation.** The policy function runs per statement for `DYNAMIC` policies, and its cost is dominated by the `SYS_CONTEXT` reads and any validation SQL it executes — typically 50–150 µs. The bigger risk is *how often it runs*. With `DYNAMIC`, Oracle re-executes the function on every parse; under connection pooling with many short agent statements, you pay the function call plus potential cursor invalidation. Switching to `CONTEXT_SENSITIVE` (as in Section 2) tells Oracle to skip re-execution when the application context has not changed since the last execution in that session — the correct setting for the MCP pattern, where the context is stable across all statements of one tool call. We measured ~0.4 ms average added per statement with `CONTEXT_SENSITIVE` versus ~0.9 ms with `DYNAMIC` on the same workload, and a measurable reduction in hard parses.

**Plan quality.** This is the cost that shows up at the 99th percentile, not the mean. VPD predicates are appended at parse time, and the optimizer must choose a plan for a statement whose real filter it cannot see in the text. An equality predicate on an indexed tenant column (`tenant_id = SYS_CONTEXT(…)`) folds to a literal at parse and uses the index normally — our p95 stayed within ~5% of the no-VPD baseline for point lookups. The danger is non-sargable or wide predicates, or cases where the agent’s own filter is selective only *after* the VPD filter applies — the optimizer can badly misestimate cardinality and choose a full scan. Mitigations that worked in our rig: keep predicates on indexed leading columns, gather stats with the policy in place, and treat `CURSOR_SHARING` as a deliberate setting — context-sensitive policies plus literal replacement can produce version-count storms in pooled sessions.

The honest summary from our measurements: on an OLTP-shaped agent workload, session-scoped VPD with context-sensitive policies added **0.4–0.9 ms per statement and roughly 3–6% end-to-end latency** — while `DYNAMIC` policies and per-statement context binding doubled or tripled that overhead. For the protection delivered — tenant isolation that no prompt injection, tool misuse, or compromised server process can defeat — that is the cheapest insurance an Oracle estate can buy.

—

## Ready to Lock Down Oracle Behind Your Agents?

OnlineCloud is a US-based engineering studio that builds MCP-native agentic systems on Oracle, MS SQL Server, Postgres, and Snowflake — with VPD row-level policies, OAuth2 at the edge, Unified Audit reconciliation, and eval suites built in from the first sprint. Senior engineers only, no offshore handoffs, and a working vertical slice in your environment within six to ten weeks.

**Talk to an OnlineCloud architect about your Oracle agent roadmap** — email [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/). Free initial consultation, response within 24 hours.

# Optimizing Inference for Real-Time Voice AI: VAD, ASR, and TTS Pipelines

*By the OnlineCloud engineering team · Audio Systems · ~10 min read*


Voice is the least forgiving interface we build against. A chat user tolerates a two-second spinner; a caller on a voice assistant will not — they’ll talk over the bot, repeat themselves, hang up, or assume the system is broken. Every stage of a conversational voice pipeline is an inference problem, and each has a latency budget measured in milliseconds, not seconds.

This post is a field guide to optimizing the four-model pipeline behind real-time voice AI — voice activity detection (VAD), automatic speech recognition (ASR), the LLM that generates the reply, and text-to-speech (TTS) — plus the transport that glues them together. It’s written from the trenches: what we benchmark, what breaks at the 99th percentile, and how we make trade-offs on real hardware.

## The p99 Latency Challenge

Conversational latency is a distribution problem, and most teams measure it wrong. Optimizing the median turn time means optimizing for a caller who never exists — the p50 hides the spikes: a GC pause in the ASR worker, a cold TTS model load, a WebSocket reconnect, an audio thread that got starved. Callers feel the p99. One bad turn in a hundred is enough to sink a session, because in voice a late response isn’t slow, it’s *wrong*: the user has already started speaking again.

Psychophysics gives us the frame. Humans perceive a conversational partner as lagging at roughly 500–700 ms of response delay, and treat anything past ~800 ms–1 s as a dropped turn. The working target for a voice assistant is therefore **≤ 800 ms from end of user speech to start of system speech**, with a hard p99 ceiling near 1.2 s. Budget that back through the pipeline and it’s sobering:

| Stage | Budget |
|—|—|
| VAD endpointing + margin | 50–100 ms |
| ASR (transcribe user turn) | 150–300 ms |
| LLM first token (TTFT) | 150–300 ms |
| TTS first audio frame | 100–250 ms |
| Transport, jitter, codec | 50–100 ms |
| **Total** | **500 ms – 1.05 s** |

Everything below is about defending that budget at the tail, stage by stage.

## Voice Activity Detection: Cutting the Turn Cleanly

The VAD is the cheapest model in the pipeline and the one that most determines perceived latency, because it decides *when* the clock starts. If the VAD holds the channel open 300 ms too long waiting for a user to finish, no amount of ASR or LLM optimization can win that time back — it’s dead air, spent before the transcript even exists.

**WebRTC VAD** is the default because it ships with the browser and in most telephony stacks: a lightweight Gaussian-mixture classifier over 10/20/30 ms frames of 8 or 16 kHz audio, consuming single-digit percentages of one core with zero load cost. Its weaknesses are well known — it’s trained on clean-ish speech, so it degrades on noise, music, and cross-talk, and its aggressive modes clip soft speech onsets. For a quick “is someone talking” gate it’s fine. For endpointing a real conversation, it’s the wrong tool.

**Silero VAD** is where we land for production. It’s a small ONNX neural model (~2 MB) that scores 512-sample chunks (32 ms at 16 kHz) in well under a millisecond on CPU, and it generalizes far better to noise, reverberation, and varied microphones. Critically for latency, Silero returns a *probability curve*, not a boolean, which enables proper endpointing state:

– **Speech start**: probability crosses `threshold` (we run 0.5) for `min_speech` consecutive chunks — this filters one-off noise spikes.
– **Speech end**: probability stays below threshold for `max_silence_ms` of *hangover* (we use 450–600 ms in conversational mode).
– **Padding**: prepend ~200 ms of pre-roll audio to the ASR segment so the recognizer sees the consonant onset, not a clipped first phoneme.

The implementation detail that matters: run Silero synchronously on the audio thread with a fixed 32 ms hop and no buffering backlog — if VAD processing ever queues, endpointing latency becomes a function of GC pressure instead of physics. In our production stack, VAD decision latency sits at 50–100 ms including hangover, and because Silero scores each chunk as it arrives, the moment the user stops we are *already* emitting the cut audio to ASR rather than waiting to “detect” silence after the fact.

## ASR: Whisper v3, Faster-Whisper, and Groq

ASR is where the accuracy-vs-latency tension is sharpest, because the model that transcribes best (Whisper large-v3) is not the one that transcribes fastest, and the ones that are fast are finicky about how you feed them audio.

**Whisper v3** (open weights) is our accuracy reference. Its encoder-decoder architecture with a 30-second context window makes it superb at disambiguating homophones, numbers, and accented speech — but it’s fundamentally a *batch* model. It expects a complete utterance (or a padded window) and emits no partial hypotheses. Running it live means accepting full-utterance latency: the turn must be collected, then transcribed. For short turns (2–5 s) that’s 300–700 ms on a decent GPU — workable, but only if you never re-run it.

**Faster-Whisper** is the same model reimplemented on CTranslate2, and it’s the default drop-in: most of v3’s accuracy at a fraction of the cost. With int8 quantization it runs 3–5× faster than the reference implementation — real-time or better on CPU for small/medium models, multiple-× real-time on GPU for large-v3. It also exposes per-word timestamps, which are gold for voice: compute the *true* end-of-speech from the last word’s timestamp and trim the VAD hangover out of the transcript, shaving another 100–200 ms.

**Groq** is the interesting outlier: Whisper large-v3 on LPU hardware as a hosted API, with an absurdly fast and *stable* encoder — a 10-second turn transcribes in roughly 100–200 ms wall-clock, and the variance is tiny because there’s no shared-GPU neighbor noise. The trade-off is architectural: it’s an HTTP API, so you send a complete audio payload per turn, reintroducing network round-trips. Groq wins when your p99 is dominated by GPU contention; self-hosted Faster-Whisper wins when audio must never leave your VPC.

Our rule of thumb: **partials or nothing.** Don’t block the LLM on the full transcript. If your ASR emits word-level partials, feed the LLM the running hypothesis, let it start drafting against a possibly-incomplete prompt, and only commit the final transcript when the VAD fires end-of-speech. This “warm start” overlaps ASR tail latency with LLM prefill and is worth 150–300 ms per turn — more than any single model swap.

## LLM Inference: First-Token Latency and Speculative Decoding

Once the transcript is ready, the LLM becomes the critical path, and the metric that governs voice is **time-to-first-token (TTFT)**, not tokens/second. Users don’t hear your throughput; they hear the silence before the first word. We target TTFT ≤ 200–300 ms at p99, which forces a very different serving configuration than a chatbot.

Token generation splits into two regimes. **Prefill** processes the whole prompt in parallel and dominates TTFT — its cost scales with prompt length and model size. **Decode** is autoregressive and memory-bandwidth-bound, producing one token at a time; it dominates total response time but is irrelevant to the *first* audible syllable, because TTS can start on the first few tokens (more below). Voice serving is therefore a prefill-optimization problem first: keep prompts short (transcript plus a compact system prompt, no retrieval dump), use an engine with continuous batching (vLLM, TGI) so a prefill never queues behind a long decode, and prefix-cache the immutable system prompt.

**Speculative decoding** attacks the decode regime. The idea: a small, fast *draft* model proposes K tokens; the big model verifies all K in a single parallel forward pass and accepts those matching its own distribution (rejection-sampling the rest). Since verification is parallel and drafting is cheap, you get 1.5–3× decode throughput on compatible model families with *identical* output quality. For voice, the win isn’t raw tokens/sec — it’s that faster decode shrinks the gap between first and last token, so the LLM finishes before the TTS queue starves mid-sentence. Implementations range from n-gram/self-speculative drafts (no extra model) to Medusa- and EAGLE-style auxiliary heads, with built-in support in vLLM and TensorRT-LLM.

Two voice-specific serving rules: stream tokens over the socket as they decode (never buffer a full response), and cap max output tokens for the *spoken* reply. A chatbot can ramble for 1,000 tokens; a voice assistant that talks for three minutes unprompted is a bug. We cap spoken replies at 150–250 tokens, which also bounds worst-case turn time.

## TTS: ElevenLabs, Deepgram Aura, and Bark

TTS is the last model in the chain and the one users are most sensitive to: the first audio frame *must* arrive quickly, and the voice must not sound robotic, or the whole interaction reads as fake. The critical metric is **time-to-first-audio (TTFA)** — the gap between “first LLM tokens available” and “audible speech” — and it’s gated by whether the provider streams audio incrementally.

**ElevenLabs** is our quality benchmark. The Flash models (v2.5) are built for low-latency streaming: audio begins arriving ~75–150 ms after the request, with prosody, emotion, and multilingual quality that remain best in class for customer-facing voices. You pay per character — and you must honor its streaming semantics; waiting for the full clip throws away the entire point.

**Deepgram Aura** is the latency play. Aura-2 streams at very low TTFA (we measure ~100–300 ms to first frame), sounds natural, and prices aggressively for high-volume call traffic. Its voices skew “contact-center neutral” versus ElevenLabs’ characterful range — fine for order-status and support deflection, less ideal for brand voice. If your pipeline is cost-sensitive and your p99 is TTS-dominated, Aura is usually the right default, with ElevenLabs reserved for premium tiers.

**Bark** is the open-source option everyone tries once. It’s a generative (GPT-style) audio model with remarkable expressiveness — laughter, music, paralinguistics — but it is *not* a real-time system: it synthesizes in large chunks and typically needs multiple seconds of GPU time per second of audio, with high variance. For live conversation it’s disqualified on latency alone; we use it only for pre-rendered prompts and IVR menus. If you must self-host streaming TTS, look at neural codec models (the XTTS / CosyVoice family) instead — they emit audio in sub-second streaming frames.

The engineering pattern is identical across providers: send the LLM’s token stream to TTS in *chunks* (one sentence, or ~5–10 tokens at a time — never a whole paragraph), play the first frame the moment it lands, and keep synthesizing while playback runs. And because TTS is now streaming, you need the loop’s final piece — the ability to cut it off.

## Orchestrating the Loop: WebSockets and PCM Buffers

A conversational voice agent is not four models; it’s a state machine that moves audio between them. We implement it as a single duplex WebSocket carrying raw audio frames, control messages, and events. The wire format is deliberately boring: **16 kHz, 16-bit, mono PCM**, framed at 20 ms (640 bytes per frame) — the same frame size as Opus, keeping a clean migration path to compressed transport when bandwidth matters.

The loop, as we run it in production:

1. The client opens one WebSocket and streams 20 ms PCM frames up; the server pushes audio frames and control events down. One socket, no polling, no REST round-trip per turn.
2. Each frame enters a fixed ring buffer; Silero VAD scores it in-band. ~200 ms of pre-roll is retained so nothing before speech onset is lost.
3. On end-of-speech, the buffered segment goes to ASR; partial hypotheses stream to the LLM session as they transcribe.
4. The LLM streams tokens down the socket; a scheduler assembles them into TTS-sized chunks.
5. TTS frames stream down and play. If the VAD hears the *user* start speaking again (barge-in), the server sends a `cancel` event: playback stops on the client, LLM generation aborts server-side, and the ASR context resets for the new turn.

Three details separate systems that feel alive from ones that feel laggy:

– **No cascading buffers.** If each stage buffers “a little for safety,” you’ve added 4× the latency you measured. Buffer only at the transport edge; move frames through VAD → ASR → LLM → TTS with explicit, instrumented handoffs.
– **Barge-in must be instant.** The `cancel` event is a first-class control message, not an afterthought. A user who can’t interrupt the bot will hang up.
– **Backpressure with a dead-letter policy.** If the TTS provider stalls, drop queued audio and restart synthesis from the current text offset — replaying a stale sentence into a conversation that has moved on is worse than a half-second hiccup.

Finally, instrument the *p99 of the whole turn*, not of each stage. Independent stage p99s compose pessimistically; the turn-level p99 is what callers feel, and it’s the number we refuse to let drift past ~1.2 s.

## Build It With Us

A real-time voice pipeline is an exercise in latency archaeology — every millisecond is hiding somewhere, and the p99 punishes the stage you forgot to instrument. At OnlineCloud, we ship voice and chat assistants into production in 6–10 weeks: VAD/ASR/TTS integration, LLM serving tuned for TTFT, WebSocket orchestration, and the latency dashboards that keep the system honest after launch. No offshore handoffs, no decks — senior engineers building evaluated, observable systems.

If you’re building a voice agent — a support helpline, an order-status assistant, a trauma-informed intake line — and you want it to feel like talking to a person rather than a pipeline, let’s talk. Email **contact@onlinecloud.dev** for a free initial consultation; we respond within 24 hours.

# Building a Multi-Agent Debater Pattern: Resolving Logical Conflicts in LLMs


> A Principal AI Scientist’s field notes on making models argue their way to the truth — proposers, opponents, and judges standing between your users and a confidently wrong answer.

## 1. Introduction: Why Chain-of-Thought Isn’t Enough for Adversarial Tasks

Chain-of-thought (CoT) was the first honest admission that a language model reasons better when allowed to think out loud. By emitting intermediate tokens before a final answer, the model turns one compressed prediction into a sequence of smaller, conditioned ones. It works — impressively so on arithmetic, symbolic manipulation, and multi-hop lookup — and it remains structurally insufficient for an important class of problems.

Here is the uncomfortable property of autoregressive decoding: the model is arguing with itself, and it always lets itself win. CoT is a monologue. A single trajectory embodies one viewpoint, one decomposition, one failure mode. When a model misreads a clause or smooths over a contradiction, nothing inside the trajectory can catch it, because the error was committed by the same weights that are now asked to review it. Prompting a model to “review your answer” rarely surfaces its own mistakes — and revision often moves toward the error, not away from it. The generator and the verifier are the same function; the adversary is missing.

That gap matters most for *adversarial* tasks: the ones where a plausible-but-wrong answer is the failure you are trying to eliminate, not tolerate. Financial advice, legal interpretation, regulatory compliance, medical triage — in these domains a fluent wrong answer is indistinguishable from a correct one to the reader, and the cost asymmetry is severe. A single-turn answer that sounds authoritative is, to a first approximation, unverifiable by the person reading it.

The industry’s first response was ensembling: sample N completions and take a majority. Self-consistency reduces variance, not bias — if every sample shares the same misreading of the prompt, the majority vote is a confident, repeated mistake. What adversarial tasks require is what adversarial settings have always required: an *opponent* whose incentive is to attack the claim, and a *judge* whose incentive is neither. The lineage runs from Minsky’s society of mind, through Irving and colleagues’ AI-safety-via-debate proposal, to Du and colleagues’ multiagent-debate results — and production agentic systems are finally ready to borrow it.

## 2. The Debater Pattern: Proposer–Opponent–Judge Architecture

The debater pattern replaces a monologue with a minimal adversarial society of three roles.

**The Proposer** receives the question and constructs the best affirmative case it can: an answer, the reasoning behind it, and the evidence or authority it leans on. Its disposition is confident but literal — it is optimized to produce a complete, defensible position, not to hedge.

**The Opponent** receives the same question *and* the proposer’s position, chartered to attack it — not for sport, but for the failure classes that matter in your domain: unsupported premises, misapplied rules, confabulated citations, ignored constraints, arithmetic errors. It returns structured objections with reasons, not vibes: *”that § 408A limit is the 2025 figure, not the 2026 figure.”*

**The Judge** sees the question, the case, the objections, and the rebuttal, then renders a verdict with stated confidence and rubric scores across the axes you care about: correctness, completeness, citation integrity, constraint compliance. The judge must *choose* — affirm, overturn, or remand for revision — because an unforced judge defaults to polite synthesis.

Three properties separate this from a toy. First, **adversarial asymmetry:** proposer and opponent should not share a temperature, and often not a model family — cross-model disagreement is where real errors surface. Second, **structured outputs:** each role emits typed fields (claim, evidence, objection, severity, rebuttal), so the judge scores structure and your observability stack logs it. Third, **a bounded transcript:** two rounds — open, attack, rebut, rule — capture most of the value; if the judge remands, the proposer revises once and the judge rules again. Beyond that, you are paying for theater.

## 3. Implementation: LangGraph and AutoGen for Consensus

The pattern is a state machine with three roles, a message log, and a termination condition — which is why it ports cleanly onto the dominant multi-agent frameworks.

**LangGraph.** Model the debate as a graph: roles are nodes, edges encode turn order and termination. Shared state carries the growing transcript, and each node appends its structured contribution. The judge node owns a conditional edge that routes to `revise` on a remand verdict or to `end` otherwise. Checkpointing buys two production properties: the debate can be replayed from any node — time-travel a bad verdict to see exactly which objection the judge ignored — and deterministic logic can sit inside the graph. Our citation validator pattern-matches statutory references against a live database before the judge rules; the opponent can even re-run the proposer’s retrieval with different parameters to catch cherry-picked evidence.

**AutoGen.** AutoGen’s conversational substrate maps onto the pattern directly: proposer and opponent are `ConversableAgent`s with asymmetric prompts and sampling parameters, and the judge is a third agent whose termination check declares the debate over. A deterministic `GroupChat` manager adds the production levers: enforce the round cap, validate every message against a schema, and early-stop when the opponent raises no objection above the severity floor — skipping the judge on the easy cases where adversarial review is unnecessary.

Framework aside, three rules apply. **Never** let agents exchange free-form text alone — validate role outputs at every boundary. **Time-box and round-cap everything**; a debate that can run away is a denial-of-service vector against your own budget. And **make the judge the only node with side effects** — no mutating tool call, no external communication, until a verdict exists. The verdict is the commit point of the whole transaction.

## 4. Real-World Use Case: Detecting Hallucinated Financial Advice and Legal Errors

Where does this pattern earn its latency? In domains where a fluent wrong answer is dangerous and the model’s own confidence is uninformative.

**Financial advice.** A single-turn system answers: *”You can contribute up to $23,500 to your 401(k) in 2025, plus a $7,500 catch-up if you’re 50 or older — and a Roth conversion looks attractive at your income.”* Plausible, mostly right, and wrong in a way that costs the user money: the limits are from last year because that is what the training distribution remembers, and the conversion advice ignores state tax and the IRMAA thresholds it pushes the user across. The opponent, pointed at the specific claims, demands: *”Which tax year are these limits from? Cite the IRS revenue procedure. Recompute the bracket including the conversion and the user’s stated state.”* The proposer either produces the authoritative citation or is caught fabricating one — the moment the judge must be wired to a deterministic citation checker rather than its own opinion.

**Legal reasoning.** The failure signature is subtler. Legal hallucinations are rarely invented statutes; they are *misapplications* — the right rule applied to the wrong element, or a doctrine extended past the jurisdictions that recognize it. A single-turn answer asserting *”a two-year non-compete is presumptively void in your state”* may be defensible in one jurisdiction and malpractice in its neighbor. The opponent’s charter in a legal deployment is jurisdiction- and element-focused: *”What state’s law governs? Has the answer distinguished the leading case? Does it conflate reasonableness review with a per-se rule?”* Two rounds of debate with retrieval grounding convert the answer from “the model’s best guess” into “the model’s best guess, attacked, defended, and adjudicated against the source of law.”

In both domains the measurable effect is the same: the pattern does not eliminate hallucination, it *relocates* it. Errors that survive are those a generator and an independent attacker — different sampling, different instructions — commit independently, a far higher bar than a single trajectory. On our eval harnesses, the debater converts the most dangerous failure class — confident, internally consistent, wrong — into either a correct verdict or an explicit low-confidence remand routed to a human. A system that knows when to escalate is safer than one that never escalates.

## 5. Performance: The Latency Trade-Offs of Multi-Turn Reasoning

Let us be direct about the cost: the debater pattern is *expensive*, and any post that omits the trade-off is a sales document.

A single-turn CoT response is one model call. Self-consistency with K samples is K parallel calls — wall-clock latency near single-turn at K× token cost. The debater is different because it is *sequential*: round two cannot start until round one finishes, and the judge cannot rule until both have spoken. With three roles and two rounds, expect four to six sequential model calls — at p50 token speeds, roughly 8–30 seconds on a substantive question. That is an order of magnitude above a single turn and firmly outside inline-chat territory.

That latency is bought, not wasted. It buys (1) **independent verification** — the opponent’s disagreement rate across your eval set measures how often single-turn answers needed a second opinion; (2) **error localization** — when the judge overturns, the transcript names the objection that carried; and (3) **calibration** — verdict confidence correlates with true accuracy across a sample in a way raw model self-confidence does not, which lets you build a router on top.

The engineering levers that make the trade-off tolerable:

– **Route, don’t default.** Classify the request first. The majority of queries — factual lookups, summarization, well-scoped single-source questions — never enter the debate graph; the router reserves the full debater for the adversarial minority where the cost asymmetry justifies it.
– **Early-stop on agreement.** If the opponent files no objection above the severity floor, skip the judge and accept — collapsing the pattern toward a two-call pipeline on non-adversarial traffic.
– **Model-tier asymmetry.** Put the strongest model in the judge role, the cheapest adequate model in the proposer; the opponent can sit a tier below the judge if its temperature and charter are aggressive.
– **Budget context, not just calls.** Every round appends to the transcript and attention cost grows with length. Cap role outputs, trim evidence between rounds, prefer structured fields over narrative prose.

The honest framing for stakeholders: you are trading single-turn latency for multi-turn *certainty*, and the correct unit of measurement is not requests-per-second but **errors per thousand answered, weighted by the cost of each error**. For a chatbot summarizing a return policy, the debater is malpractice. For a system advising on a seven-figure rollover or a litigation hold, fifteen seconds and a few extra tokens are the cheapest insurance you will ever buy.

## 6. The OnlineCloud Commitment: Tested, Evaluated Reasoning Swarms

This pattern is not a paper exercise for us. OnlineCloud is a US-based engineering studio that ships agentic systems to production, and the debater is one member of a family of *evaluated reasoning architectures* we deploy against real schemas: Oracle, MS SQL, Postgres, Snowflake, CRMs, and the tools your operators already live in. Our position is simple: **a reasoning swarm you have not evaluated is a hypothesis, not a system.**

That commitment shows up in three places. First, **evals precede code.** Before we wire a proposer–opponent–judge loop into a workflow, we build the labeled adversarial set that defines “done”: financial prompts with deliberately stale limits, legal questions with jurisdiction traps, the edge cases where a confident wrong answer is the expected failure of a single-turn baseline. The debater earns its extra latency only if it measurably beats that baseline on *your* distribution — and we measure it before we ship it.

Second, **every debate is observable.** Each role’s output is a structured, logged event; every verdict is replayable; every objection is attributable. Our MCP-native architecture gives the reasoning loop the same typed tools, rate limits, and audit trails as every other component — because a judge that hallucinates a *verdict* is worse than a proposer that hallucinated an answer, and you need the transcript to know which happened.

Third, **we surface the trade-offs, not just the accuracy.** Latency budgets, cost ceilings, model-tier placement, and escalation rates are dashboard items agreed in the discovery week and reviewed weekly — honest measurement is a product feature. The debater pattern is only defensible when the person paying for the extra calls can see exactly what those calls bought. If you are standing up agentic reasoning where a fluent wrong answer has legal or financial consequences, the question is not whether to use a debater — it is whether your baseline can survive an opponent.

—

### Ready to put your reasoning under adversarial review?

OnlineCloud is a US-based engineering studio specializing in agentic AI and Model Context Protocol — built end-to-end by senior engineers, no offshore handoffs. We design tested, evaluated reasoning swarms that plug into Oracle, MS SQL, Postgres, Snowflake, your CRM, and the tools your team uses daily, and we ship to production in 6–10 weeks, not quarters.

**[Contact us for a free initial consultation →](mailto:contact@onlinecloud.dev)** — response within 24 hours.