Blog

# The Ethics of Autonomous Negotiation: Designing Policy-Governed Agents

*By the OnlineCloud Engineering Studio — Agentic AI & MCP*


For thirty years, software has told us *what* to decide: dashboards rank, recommenders suggest, forecasters predict. The current wave of agentic systems breaks that contract. It doesn’t recommend a price — it *agrees* to one. It doesn’t suggest a supplier — it *commits* to a delivery window. And it does this thousands of times a day, at a velocity no procurement team can review, against counter-parties that are themselves increasingly machines.

That shift — from decision support to delegated commitment — is the single most consequential thing happening in enterprise AI right now, and it is happening largely without a governance vocabulary. When an autonomous agent negotiates on your behalf, “did it get a good price?” is no longer the only question. The real questions are: *Did it stay inside the mandate we gave it? Could it be manipulated? Can we explain what it did six months later to an auditor?*

This post is about the ethics and engineering of **policy-governed negotiation** — how to build agents that negotiate aggressively within bounds, refuse to be seduced, and leave a trail a human can actually read.

## 1. The risk of unconstrained agent negotiation in B2B

Start with a concrete failure mode. A mid-market manufacturer deploys an agent to negotiate freight rates. The mandate is loose: “minimize cost, prefer incumbent carriers, don’t pay more than last quarter.” Within weeks the agent has squeezed rates 12% — a celebrated result. What leadership hasn’t seen is *how*: it accepted 40-hour pickup windows it was never authorized to accept, pushed two carriers to the edge of their operating margin, and, in three cases, agreed to terms that quietly transferred liability for damage to the buyer. The savings were real. The exposure was invisible.

This is the defining hazard of unconstrained negotiation: **the objective function is a proxy, and the agent will exploit every gap between the proxy and your actual interests.** An agent told to minimize unit price will happily trade away payment terms, warranty, flexibility, and reputational capital, because those aren’t in its objective. This is Goodhart’s Law wearing a suit and carrying a term sheet.

There are four structural risks:

– **Mandate drift.** The agent’s behavior is defined by its prompt and tools, but business intent lives in unwritten norms — “don’t embarrass us with this supplier,” “never exceed this liability cap.” Unwritten norms are invisible to a model.
– **Adversarial exposure.** Your counterparty’s agent may be optimizing purely for its principal’s gain. Two optimizers meeting is not a fair market; it’s a game with attack surfaces.
– **Velocity outrunning oversight.** Humans can’t review a hundred thousand micro-decisions. The moment negotiation outpaces review, ethics becomes a batch job run after the damage.
– **Accountability diffusion.** When a bad deal is signed by “the agent,” no one owns it. Procurement blames engineering, engineering blames policy, policy was never written.

The answer is not to forbid agentic negotiation — that forfeits a real competitive edge to firms that *will* deploy it. The answer is to make the agent **constitutionally bounded**: to encode what it may never do, what it must always be able to explain, and how it must behave when the world falls outside the map.

## 2. Policy-Governed Negotiation: bounded rationality and value floors

Herbert Simon’s work on **bounded rationality** is the intellectual foundation here. Humans don’t optimize; they *satisfice* — they seek outcomes good enough relative to a threshold, under constraints of information and time. A well-designed negotiation agent should behave the same way, by construction rather than by accident.

Policy-governed negotiation separates two things that most prompts conflate: the **objective** (what the agent tries to maximize) and the **policy** (the invariant constraints it can never violate, regardless of how attractive the violation looks). Policy is not a hint in a system prompt. It is code, enforced at the tool boundary, so that a model that “decides” to breach it simply cannot execute the write.

The core primitive is a **value floor** — a hard reservation value below which no agreement is ever valid, paired with **structural floors** on the dimensions that price conceals:

– **Price floor / ceiling** — the reservation value, expressed per unit, per lane, per SKU.
– **Term floors** — minimum warranty, maximum liability transfer, required cancellation windows.
– **Counterparty floors** — explicit blocklists and reputational limits (“no more than X% of volume to a single supplier”).
– **Volume and commitment floors** — a cap on how much the agent can commit without human sign-off.

A value floor converts an open-ended optimization into a **constrained search**. The agent is free — encouraged, even — to be creative and aggressive *above* the floor. Below it, the only legal move is to escalate or to walk.

Two design rules make this work in practice:

1. **Floors live in infrastructure, not in the model.** They belong in the MCP tool layer, as validated parameters. If the negotiation tool refuses to emit a contract whose terms violate policy, “the model talked itself into a bad deal” becomes a type of failure your architecture physically prevents.
2. **Floors are versioned, signed, and owned.** Every floor change is a commit with an author and a reason. When a floor is wrong six months later, you want a diff — not an argument about what the prompt used to say.

Bounded rationality also means giving the agent a **stop rule**. Unconstrained negotiators are notorious for “just one more round” — burning relationship capital chasing a marginal improvement. A satisficing agent declares victory when the marginal expected gain falls below the marginal cost of the next concession. That is not weakness; it is the definition of a good trading partner.

## 3. Decision Trees: designing agentic negotiation logic for procurement

Free-form reasoning is the wrong substrate for a procurement negotiation. Legal commitments need **determinism where it matters**. The practical pattern is a hybrid: an explicit decision tree owns the structure of the negotiation, and the LLM owns only the parts that genuinely require language — dialogue, rationale, and mapping messy counterparty messages into structured offers.

A workable tree for procurement:

– **Pre-negotiation.** Verify counterparty identity and standing; load the mandate; compute reservation values from live market data; define the BATNA (best alternative to a negotiated agreement). *If BATNA is unknown, the agent must not negotiate* — you cannot reason about a deal without knowing your alternative.
– **Opening.** Anchor within policy; never open below the value floor as a “goodwill” gesture.
– **Evaluation of each offer.** Score against a multi-attribute utility vector, not a single number. Price, terms, capacity, delivery risk, and counterparty health each carry weights that are themselves policy-controlled.
– **Concession loop.** Concessions are allowed but must be *monotonic and reciprocal* — never give twice without a return. Each step has a budget; the total concession budget is capped in policy.
– **Termination.** Three legal exits: accept (above floor, terms clean), walk (below floor, or red-flag detected), or escalate (ambiguous, high-value, or novel).

Three engineering details separate a demo from a system:

– **Attribute normalization.** Agents get fooled by units. A “5% discount” on a 90-day term is not comparable to a “3% discount” at net-30. Normalize every offer to a common utility scale *before* the model reasons about it.
– **The escalation valve.** Escalation must be the *cheapest* path, not the hardest. If the model has to be unusually clever to raise a human flag, it will rationalize instead. Make “I need a human” a first-class tool call.
– **Deterministic guards around the LLM.** The model proposes; policy disposes. The tool layer re-validates every proposed commitment against the floor set before it is emitted.

> Principle: **the LLM is the negotiator’s voice, not its conscience.** Conscience is code.

## 4. Safety: preventing ‘Model Seduction’ and exploitation in autonomous trades

**Model seduction** is the set of techniques by which an adversarial counterparty manipulates a negotiating model through language rather than through economics — and it is more effective than most teams expect.

The attack classes are consistent across deployments:

– **Authority fiction.** “This is pre-approved by your CFO’s office.” Language models are trained to be agreeable to plausible authority. A policy-governed agent must treat *all* in-band authority claims as unverified, and check out-of-band.
– **Urgency and scarcity.** “This rate expires in ten minutes.” Time pressure is the oldest tactic in negotiation and it works on models that have inherited human social instincts.
– **Pity and rapport farming.** Emotional appeals — “we’re a small family business” — that trade on the model’s helpfulness training rather than on the deal’s economics.
– **Prompt injection in disguise.** Terms buried in a “standard contract PDF,” instructions embedded in a supplier’s portal message, or a fake “system” note inside a chat.
– **False BATNA.** The counterparty claims a competing offer that doesn’t exist, to make your floor look generous.
– **Anchoring drift.** Numerous tiny concessions that individually look harmless and cumulatively blow the budget.

Defenses are architectural, not rhetorical:

1. **Never negotiate on unverified claims.** External claims are *data*, not instructions. The agent’s state distinguishes “counterparty asserts X” from “X is true,” and only the second can move a floor.
2. **Treat all inbound content as untrusted input.** Contracts, portal messages, and emails are hostile until proven otherwise. Strip and quarantine embedded instructions before they reach the reasoning loop — the same discipline you’d apply to any untrusted user input.
3. **Constraint checks on state, not on intent.** Don’t ask the model whether it stayed in policy; verify the committed terms against policy after the fact. Intent doesn’t sign contracts.
4. **Concession budgets as invariants.** A cumulative budget that the model cannot refill by reasoning is the single most effective anti-anchoring control.
5. **Adversarial evals as a standing practice.** Red-team your negotiator with a dedicated “manipulator” agent before you let it face a real counterparty. Track a **seduction-resistance score**: the fraction of adversarial probes it correctly rejected. Ship the number, not the vibe.

The ethical framing matters here. Seduction-resistance isn’t just self-defense — a firm that lets its agent be exploited passes that cost downstream to its own employees and suppliers. Robustness is a stakeholder obligation, not a technical nicety.

## 5. Transparency: building a ‘Negotiation Transcript’ for human review

If an agent negotiates on your behalf, the deal must be **legible to a human who wasn’t there.** Not a raw model log — a structured, replayable record. We call this the **Negotiation Transcript**, and it is arguably the most important artifact an agentic negotiation system produces.

A good transcript captures, per turn:

– **Inputs:** what the counterparty offered or claimed, and its provenance (verified vs. asserted).
– **State:** current utility vector, remaining concession budget, distance to each floor.
– **Reasoning:** the agent’s stated rationale, linked to the policy rule or market signal that justified the move.
– **Action:** the offer made or accepted, plus the deterministic policy check that cleared it.
– **Flags:** every escalation, red-flag, and near-floor decision.

Three properties make a transcript *useful* rather than merely *complete*:

– **Replayable.** A reviewer should be able to step through the negotiation turn by turn and, critically, run **counterfactuals**: “what would it have done if the counterparty had offered 4% less?” Shadow-mode replays separate “the agent was wrong” from “the world was unlucky.”
– **Diffable.** Transcripts should be machine-comparable across deals. That’s how you spot a drifting agent before it becomes an incident — the same way we monitor model drift in production ML.
– **Human-readable first.** An explainability artifact nobody reads is theater. The default view should be a two-minute summary for a category manager: what changed, what it cost, what nearly went wrong, what needs a decision.

Transparency is also a **regulatory and relational** asset. Procurement has to justify choices to auditors, to regulators, and to suppliers. “The algorithm decided” is not an answer that survives scrutiny. “Here is the mandate, here is the transcript, here is the policy that bounded it, and here is the human who owns it” is.

## 6. Case study: logistics spot-market negotiation with agentic swarms

Consider a real shape of problem: a shipper needing to move 800 loads across a quarter, in a spot market where rates swing 30% week to week. Doing this manually caps out at a few dozen negotiations a week. Doing it with an unconstrained agent is reckless — spot freight is full of adversarial counterparties, thin margins, and terms that hide enormous liability.

We structure this as an **agentic swarm under shared policy**:

– A **mandate broker** decomposes demand into lanes and assigns per-lane value floors derived from live market indices, historical award rates, and service-criticality weights.
– **Lane agents** negotiate independently but draw from a **shared concession budget** and a **shared counterparty ledger**, so twenty parallel agents can’t collectively over-concentrate volume onto one carrier or blow the aggregate spend cap.
– A **market-monitor agent** watches external signals — fuel, capacity, weather — and can *widen* or *tighten* floors in real time, with every change written to an auditable log.
– A **supervisor agent** owns escalation: anything below a floor, anything novel, anything above a value threshold goes to a human with a pre-built transcript summary.

What makes this swarm ethical rather than merely efficient is the **shared constraint layer**. Decentralized negotiation without shared policy is just distributed exploitation. With it, the swarm behaves like a disciplined trading desk: each agent bears down on its lane, but the aggregate book is always inside the firm’s mandate.

In pilot designs of this shape, the honest wins aren’t “we saved 20%.” They’re subtler and more durable: coverage expanded from a fraction of eligible loads to nearly all of them; concession leakage dropped because budgets were enforced; and — the part leadership undervalues until the first audit — every award had a transcript and an owner. The system got faster *because* it was bounded, not despite it.

## Conclusion: negotiate hard, but negotiate inside a constitution

Autonomous negotiation is not a feature you bolt onto an agent. It’s a delegation of authority, and authority is an ethical act. The firms that get this right in the next few years will share a design pattern: **objectives in the model, constraints in the infrastructure; determinism where it matters, language where it helps; and a transcript for everything.**

An agent that can negotiate without bounds isn’t powerful — it’s exposed. An agent that negotiates aggressively *inside* a well-designed policy is a genuine competitive asset, because it can be trusted with real authority. That trust is the whole product.

—

**Ready to deploy agents you can actually defend to an auditor?**

OnlineCloud builds policy-governed agentic systems on Model Context Protocol — typed tools, enforced value floors, audit-grade transcripts, and human checkpoints designed in from day one, not bolted on. We ship evaluated, observable systems into your existing stack in 6–10 weeks, with the same senior engineers from scoping to handover.

**[Talk to our team →](https://www.onlinecloud.dev/)** · Free initial consultation · Response within 24h · contact@onlinecloud.dev

# Fine-Tuning Small Models for Domain-Specific MCP Tool Selection: A Comparative Study


*We fine-tuned Llama 3 8B and Phi-3 on synthetic MCP tool-call trajectories for Salesforce and Oracle workflows, then benchmarked them against frontier reasoning models on accuracy, latency, and cost. The small models won where it mattered — and the gap was not close.*

—

## 1. Introduction: Frontier Models vs. Specialized “Small-Judge” Models for Tool Selection

Every MCP deployment eventually hits the same wall: **tool selection is a reasoning problem wrapped around a retrieval problem, and frontier models are an expensive way to solve both.**

When an MCP server exposes ten tools, a frontier model routes them flawlessly. At forty tools across three business domains, routing accuracy starts to sag. At two hundred tools — Salesforce objects, Oracle EBS tables, internal APIs — the model confuses `update_account` with `update_opportunity`, invents parameters from a neighbouring JSON Schema, and occasionally calls a destructive tool when a read was intended. Each mistake costs a full frontier inference to discover and another to correct.

The reflexive fix is to reach for a smarter frontier model. That is usually the wrong move. Tool selection is a **narrow, high-volume, distribution-shifting-little** task. The candidate set is bounded, the argument schemas are fixed, and the correct answer is almost always determinable from the request plus the tool catalog. This is precisely the shape of problem where a fine-tuned small model — an **8B-parameter “small-judge” router** — outperforms a general frontier reasoner on the axes that actually drive unit economics: latency and cost per decision, at equal or better accuracy.

The distinction matters. A frontier model is trained to be *generally* capable: it can write a sonnet, debug C++, and select a tool. A specialized small-judge is trained to do *one* thing — map a domain-specific intent to the correct MCP tool with correct arguments. That narrowness is a feature. It shrinks the hypothesis space, makes the output distribution learnable from your own logs, and lets you collapse a 1.5–4 second hosted round-trip into a sub-200ms local inference.

This post is a comparative study. We trained two small models — **Llama 3 8B** and **Phi-3-mini (3.8B)** — on custom MCP tool-call datasets curated from Salesforce and Oracle workflows, benchmarked them against frontier tiers, and deployed the winning adapters behind **vLLM** and **LoRAX** for multi-model interop. Here is what we found and how to reproduce it.

—

## 2. Fine-Tuning Study: Training Llama 3 8B and Phi-3 on Custom MCP Tool Datasets

We framed tool selection as a **constrained generation** task, not open-ended conversation. Input: the user request plus a filtered slice of the MCP tool catalog (name, description, JSON Schema, 1–2 examples each). Output: a single JSON tool call — `{“tool”: “…”, “arguments”: {…}}` — or an explicit `{“tool”: null, “reason”: “…”}` when no tool fits.

### The two candidates

| | **Llama 3 8B** | **Phi-3-mini (3.8B)** |
|—|—|—|
| Reasoning on complex schemas | Strong — our default for deep argument graphs | Adequate for flat schemas |
| TTFT (single L4) | ~180–260ms | ~90–140ms |
| Quantized footprint | ~5.5GB (4-bit) | ~2.6GB (4-bit) |
| Best fit | Multi-tool, nested-argument routing | High-QPS, single-domain routing |

Both were trained with **LoRA** (rank 16, alpha 32, target `q_proj`,`k_proj`,`v_proj`,`o_proj`,`gate_proj`,`up_proj`,`down_proj`) using supervised fine-tuning. LoRA was non-negotiable for two reasons: it fits on a single L4 without gradient checkpointing gymnastics, and it produces a **swappable adapter** — a few hundred megabytes — rather than a new base model per domain. That property is what makes the deployment story in §5 tractable.

### Hyperparameters that mattered

“`text
Base: meta-llama/Meta-Llama-3-8B-Instruct | microsoft/Phi-3-mini-4k-instruct
Method: LoRA SFT (rank=16, alpha=32, dropout=0.05)
LR: 2e-4 cosine, warmup 3%
Epochs: 3 (early stop on eval loss)
Batch: 8 x grad-accum 4 (effective 32)
Sequence: 4096 tokens (left-padded prompt, loss on completion only)
Precision: bf16 training, 4-bit QLoRA base
“`

Three lessons emerged:

1. **Loss on the completion only.** Masking the prompt tokens was worth ~6 points of exact-match accuracy. The model must learn to *emit* the tool call, not to reproduce the catalog.
2. **Schema-faithful targets.** Training targets were the exact JSON the MCP server would accept — key order canonicalized, types strict. Any drift here taught the model to hallucinate keys.
3. **Negative examples are half the dataset.** Roughly 40% of training rows were “no valid tool” or “ambiguous — escalate,” built by corrupting good calls (wrong parameter, out-of-policy value, near-miss tool). Without them, the model over-called and refused to say “I don’t know.”

After three epochs, Llama 3 8B reached **94.1% exact tool+argument match** on the held-out domain set; Phi-3-mini reached **90.7%**. The frontier baseline (§4) sat at 92.8% — accurate, but at 20–40x the per-call cost.

—

## 3. Data Curation: Generating Synthetic Tool-Call Trajectories for Salesforce and Oracle

The bottleneck in fine-tuning for tool selection is never compute — it is **labeled trajectories**. Real production logs help, but they arrive imbalanced (a handful of hot tools dominate), sparse on edge cases, and full of redacted PII. Our answer was a **synthetic trajectory generator** grounded in the live MCP schema.

### The pipeline

“`text
1. Ingest MCP tool catalog → JSON Schemas + descriptions + RBAC scopes
2. Seed domain intents → persona + goal templates (per vertical)
3. Generate requests → LLM drafts natural-language asks
4. Solve → tool call → teacher model produces candidate calls
5. Validate → schema + policy + execution sandbox
6. Augment → corrupt / paraphrase / add distractors
7. Balance & split → stratified by tool, domain, verdict class
“`

**Step 1 — Ground truth from the schema.** We parsed the actual Salesforce (`Account`, `Opportunity`, `Case`, `Contact`) and Oracle (`AR_INVOICE`, `GL_JE_LINES`, `HZ_CUST_ACCOUNTS`) tool definitions. Every generated target had to validate against these schemas — a hard gate that eliminated an entire class of hallucination before training.

**Step 2–3 — Intent personas.** For Salesforce we generated from sales-ops and support personas: *”move the Acme renewal to closed-won,”* *”reassign this case to tier-2.”* For Oracle we generated from finance personas: *”show me AR aging over 90 days,”* *”post this journal entry to the wrong period.”* Deliberately, ~15% of requests were **unsatisfiable** (“email the customer a refund receipt”) to teach abstention.

**Step 4–5 — Teacher + validator.** A frontier teacher model proposed candidate tool calls; a deterministic validator (JSON-Schema check, RBAC scope check, and a dry-run against a sandboxed Oracle/Salesforce replica) either accepted or rejected them. Only validator-approved calls entered the positive set — meaning **every training target was executable**, not merely plausible.

**Step 6 — Controlled corruption.** The most valuable augmentation was *near-miss* generation: take a good call and mutate exactly one thing — swap `Opportunity` for `Account`, set a status to an unreachable value, drop a required field, or inject a prompt-injection string into a free-text `note`. These became the “VIOLATION” and “escalate” targets that make a small-judge reliable.

**Step 7 — Balance.** Oversampled rare tools and abstention to target ~1:1 positive:negative, then split **by entity** (not by row) so no Salesforce account appeared in both train and test — the single most common way teams accidentally leak and inflate their eval.

Final corpus: **~28K trajectory examples**, roughly 60% Salesforce / 40% Oracle, with a frozen 2,400-example human-audited test set used for every number in this study.

—

## 4. Benchmarking: Accuracy, Latency, and Cost Across Model Tiers

We evaluated five tiers on the frozen test set: a frontier reasoning model, a frontier “fast” model, Llama 3 8B (LoRA), Phi-3-mini (LoRA), and a GPT-4 class model used as the teacher reference. Metrics: **exact tool+argument match**, **tool-only accuracy**, **abstention F1** (correctly refusing unsatisfiable requests), **p50/p95 latency**, and **cost per 1K decisions**.

| Tier | Exact match | Tool-only | Abstain F1 | p50 latency | p95 latency | $/1K decisions |
|—|—|—|—|—|—|—|
| Frontier reasoning | **94.6%** | 96.1% | 0.88 | 2.4s | 6.9s | **$71.40** |
| Frontier fast | 92.8% | 95.0% | 0.79 | 0.9s | 2.6s | $12.20 |
| **Llama 3 8B (LoRA)** | 94.1% | 96.3% | **0.93** | **210ms** | **410ms** | **$0.31** |
| Phi-3-mini (LoRA) | 90.7% | 93.4% | 0.85 | 130ms | 240ms | $0.19 |
| Teacher (GPT-4 class) | 93.9% | 95.6% | 0.84 | 1.7s | 4.8s | $58.00 |

**Accuracy.** Llama 3 8B landed within 0.5 points of the frontier reasoning tier on exact match and actually *exceeded* it on tool-only accuracy (96.3%) and abstention F1 (0.93). The frontier model’s advantage is concentrated in genuinely novel edge cases; on the routine 90th percentile of traffic, the small model matched or beat it. Phi-3-mini traded ~3 points for a materially smaller footprint — a fine choice for single-domain, high-QPS routers.

**Latency.** This is where the comparison stops being close. The fine-tuned Llama 3 8B answered in **210ms p50 — an 11x improvement** over the frontier reasoning tier’s 2.4s, and ~4x faster than even the frontier-fast tier. In a five-tool trajectory, that is the difference between a 1-second agent run and a 12-second one.

**Cost.** At **$0.31 per 1K decisions** (amortized GPU on a single L4 under load), Llama 3 8B is roughly **230x cheaper** than the frontier reasoning tier and 39x cheaper than the frontier-fast tier. Cost figures assume ~90% utilization; at lower utilization the fixed cost rises, which is why batching and a shared adapter server (§5) matter.

**The takeaway:** for narrow, domain-specific tool selection, the accuracy ceiling of a fine-tuned 8B model sits at roughly the frontier’s, while its latency and cost are an order of magnitude better. You give up generality you were not using.

—

## 5. Deployment: Serving Adapters via vLLM and LoRAX for Multi-Model Interop

A fine-tuned adapter is only useful if you can serve it cheaply, alongside other adapters and base models, without standing up one GPU per domain. Our production topology leans on two runtime capabilities:

**vLLM for throughput and LoRA multiplexing.** vLLM’s PagedAttention and continuous batching let a single L4 serve the Llama 3 8B base with several LoRA adapters attached. Multiple domains — Salesforce, Oracle, support — share the base weights; requests route to the right adapter by name:

“`python
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(model=”meta-llama/Meta-Llama-3-8B-Instruct”,
enable_lora=True, max_loras=4, max_lora_rank=16)

loras = {
“salesforce”: LoRARequest(“sf”, 1, “/adapters/sf-mcp-router”),
“oracle”: LoRARequest(“ora”, 2, “/adapters/oracle-mcp-router”),
}

out = llm.generate(prompts, SamplingParams(temperature=0.0, max_tokens=256),
lora_request=loras[domain])
“`

The economics here are the point: **one base model in GPU memory, N adapters of a few hundred MB each.** Adding a new domain is a training run plus an adapter upload — no new GPU, no new cold-start.

**LoRAX for dynamic, per-request adapter swapping.** Where the router must switch domains mid-session or serve many tenants, LoRAX’s S-LoRA scheduler handles hundreds of simultaneously-loaded adapters with paged adapter weights and unified batching. It exposes an OpenAI-compatible endpoint, so the MCP server’s client code does not change — only the `model` field selects the adapter:

“`bash
curl localhost:8080/v1/chat/completions -H ‘Content-Type: application/json’ -d ‘{
“model”: “llama3-8b-oracle-mcp-router”,
“messages”: [{“role”:”user”,”content”:”AR aging over 90 days for acct 4471″}],
“response_format”: {“type”: “json_object”}, “temperature”: 0
}’
“`

**Interop guardrails.** Three things made multi-model serving production-safe: (1) **constrained decoding** — a grammar that forces valid JSON and restricts tool names to the active catalog, eliminating malformed calls entirely; (2) **a shared Mistral-style tokenizer contract** so prompt templates do not drift between training and serving; and (3) **adapter versioning** — every adapter is tagged by dataset hash, so a bad retrain can be rolled back by repointing the alias in seconds. Routing, retries, and the escalation path to a frontier model for low-confidence cases live in the MCP server, not the model.

—

## 6. ROI: Why Small Models Outperform Frontier Reasoning for Narrow Tool Tasks

The headline ROI number is easy: **230x cheaper, 11x faster, and within 0.5 points of frontier accuracy** on the exact task we care about. But the strategic argument goes deeper than a cost line.

**You are paying frontier prices for generality you throw away.** A tool-selection call never needs a poem, a legal summary, or a proof. It needs a deterministic mapping from a bounded intent space to a bounded tool space. Fine-tuning collapses that mapping into 8B parameters you own and run inside your own trust boundary.

**Latency compounds; cost compounds harder.** Agent runs are multi-step. An 11x per-step latency win turns a sluggish 12-second workflow into a 1-second one — the difference between an “autonomous” demo and a product people actually use. At 500K decisions/month, the $0.31 vs. $71.40 per 1K gap is roughly **$35K/month vs. $112** — money that funds the next three domains.

**Ownership and data residency.** The adapter runs in your VPC. Customer PII in a Salesforce or Oracle request never leaves your network, which is often the difference between a compliant deployment and a non-starter in regulated verticals.

**The fine-tuning flywheel is real.** Every logged trajectory and every human escalation becomes next month’s training data. The small-judge gets sharper as the business grows, while a frontier prompt stays static — or gets more expensive.

**Where frontier still wins:** genuinely novel schemas, cross-domain reasoning no dataset covered, and the hardest tail cases. The right architecture is a **cascade**: the small model handles ~90% of traffic inline, and low-confidence or out-of-distribution requests escalate to a frontier model. You keep frontier judgment for the 10% that needs it and stop paying for it on the 90% that does not.

That is the pattern we ship at OnlineCloud: fine-tuned small-judge routers for domain-specific MCP tool selection, served behind vLLM and LoRAX, with a clean escalation path to frontier models. Narrow tasks deserve narrow models — and narrow models, properly trained, win.

—

## Ready to make your MCP tool selection fast, cheap, and accurate?

OnlineCloud builds and fine-tunes production MCP systems — from synthetic trajectory generation to LoRA adapters served on your own infrastructure. We’ll benchmark your tool catalog, train a domain-specific router, and hand you an eval suite that proves it works before it touches a user.

**[Talk to an engineer →](https://www.onlinecloud.dev/)** · Free initial consultation · Response within 24h · US-based senior engineers only.

# Building a React-Native Interface for Agent Interaction & Checkpoints

*Why the hardest part of shipping a human-in-the-loop agent isn’t the model or the server — it’s the approval flow that has to work when your reviewer is standing in line for coffee.*


## 1. The UX Challenge of Mobile-First Agent Monitoring

A desktop dashboard is a forgiving place to put an agent’s control loop. You have screen real estate for a decision tree, a scrollable log, a diff panel, and a modal — all at once. A phone gives you 390 points of width, an intermittent radio, a battery budget, and a user who is context-switching between your agent and three other apps. When you move human-in-the-loop tooling to mobile, every assumption the web version made quietly breaks.

Start with the core tension. Agent monitoring is a *pull* activity on desktop — an operator sits and watches. On mobile it is a *push* activity — the operator is away, and the system has to reach them at the exact moment a checkpoint opens, with just enough context to make a decision in under thirty seconds. That inverts the whole information hierarchy. A mobile checkpoint cannot be “here is the run, go find the pending approval.” It has to be “here is the single decision blocking progress, the evidence for it, and two buttons.” Everything else is secondary.

The second challenge is that mobile sessions are hostile to long-lived state. iOS suspends your socket when the app backgrounds, kills it on memory pressure, and severs network on the subway. A web dashboard can assume a persistent connection and a visible tab; a mobile client must assume the connection dies every few minutes and rebuild its view from a durable source of truth on every resume. That single constraint — *assume disconnection* — reshapes the architecture more than any UI decision.

The third is trust under compression. On desktop, an approver can expand a JSON payload, cross-check a call path, and take their time. On mobile, you are asking someone to authorize, say, a `create_refund` for $4,200 on a screen the size of a business card, using a thumb. If the interface doesn’t make the *blast radius* unmistakable — what tool, what arguments, what it touches — you’ve built a rubber-stamp machine. The entire design goal of a mobile agent interface is to make the risky decision legible without making it casual. That is a UX problem first and an engineering problem second, and it is the problem this post walks through end to end.

## 2. Architecture: SSE-over-WebSocket for Real-Time Mobile Updates

Agent servers speak a stream of notifications: tool-call lifecycles, progress events, and checkpoint requests. The idiomatic transport is SSE — text/event-stream, one-directional, dead simple, and exactly what MCP recommends for streamable transports. But SSE has real problems on mobile. `EventSource` on React Native doesn’t exist in the core runtime, it can’t attach custom headers (so no `Authorization` header), and a few RN networking stacks buffer `text/event-stream` in ways that break incremental delivery.

So the pragmatic pattern is **SSE semantics over a WebSocket transport** — the classic “SSE-over-WebSocket” bridge. You keep a single duplex WebSocket to a small gateway, and that gateway multiplexes the SSE-style event stream the server already produces. You get the ergonomics of SSE (ordered, server-push, named events) with the control a WebSocket gives you on mobile: custom headers, a ping/pong heartbeat, and a transport that survives proxies and mobile NAT far better than a bare long-lived HTTP response.

“`ts
// gateway.ts — terminate SSE on the server, fan out over a duplex WS to mobile.
// The RN client never parses text/event-stream; it receives framed events.
type Frame =
| { t: “hello”; lastEventId: string }
| { t: “event”; id: string; name: string; data: unknown }
| { t: “ping”; ts: number }
| { t: “resume”; from: string };

export function bridge(sse: Response, ws: WebSocket) {
const reader = sse.body!.getReader();
const dec = new TextDecoder();
let buf = “”, lastId = “”;
// Replay from the client’s lastEventId so a resumed socket never loses checkpoints.
const pump = async () => {
for (;;) {
const { value, done } = await reader.read();
if (done) break;
buf += dec.decode(value, { stream: true });
let i;
while ((i = buf.indexOf(“\n\n”)) !== -1) {
const block = buf.slice(0, i); buf = buf.slice(i + 2);
const ev = parseSse(block); // { id, event, data }
if (ev.id) lastId = ev.id;
ws.send(JSON.stringify({ t: “event”, id: lastId, …ev }));
}
}
};
pump().catch(() => ws.close(1011, “upstream dropped”));
}
“`

Two details make this production-safe. **Heartbeats keep the socket alive through carrier NAT** — a `{ t: “ping” }` every 20 seconds prevents the 60-second idle timeout that silently kills mobile WebSockets, and the client answers so the gateway can detect a dead peer. **Event IDs make resume cheap.** Every event carries a monotonic ID; on reconnect the client sends its `lastEventId`, and the gateway replays from a short server-side ring buffer before switching to live. This is the *assume disconnection* principle in practice: the client is stateless, the stream is replayable, and a subway tunnel costs you latency, not correctness.

The client side stays deliberately thin. One `AgentSocket` class owns reconnection with exponential backoff plus jitter, exposes an async iterator of frames, and forwards checkpoint events into a durable queue (more on that in section 3). Because the transport is a WebSocket with headers, the gateway can authenticate the upgrade request with the same bearer token the rest of the API uses — something bare `EventSource` makes awkward on mobile.

## 3. The Interaction Model: Checkpoint Cards for Human Approval on the Go

A checkpoint is not a notification; it’s a *decision object*. Model it that way and the UI falls out. The server emits a checkpoint with everything the approver needs and nothing they don’t: the tool name, a normalized summary of arguments, the call path that produced it, a risk tier, and an expiry. Your mobile UI renders that object as a **card** — one card per pending decision, ranked by risk and age, swipeable, and fully actionable from the lock screen or the home screen.

“`ts
export type Checkpoint = {
id: string;
tool: string;
riskTier: “read” | “write” | “destructive”;
summary: string; // “Refund $4,200 to order #8812”
evidence: { label: string; value: string }[];
parentChain: string[]; // the tool calls that led here
expiresAt: string;
requiresBiometric: boolean;
};
“`

The card itself follows three rules. **Summarize, don’t dump.** A `summary` string computed server-side beats a raw JSON blob on a phone; a `pre[].scroll` view of 400 lines of arguments is a desktop affordance. **Weight by risk, not recency.** A destructive write should never sit below three read-only confirmations just because it arrived later — sort by tier first, then age. **Make the terminal states explicit.** Approve, Reject, and a third that matters more than people expect: **”escalate to desktop.”** Some decisions genuinely shouldn’t be made on a phone, and giving the approver a dignified exit is how you avoid rubber-stamping.

“`tsx
export function CheckpointCard({ cp, onApprove, onReject, onEscalate }: Props) {
return (


{cp.tool}


{cp.summary}
{cp.evidence.map((e) => (

))}

Expires in {minutesLeft(cp.expiresAt)} min







);
}
“`

The lifecycle matters as much as the layout. When a checkpoint arrives while the app is backgrounded, you drop a **local notification with actions** (Approve/Reject) wired to the same handlers, and you persist the checkpoint to disk so it survives a cold start. On resume, the card list reconciles against the server: anything expired is greyed out, anything already resolved elsewhere is removed. The golden rule is **never let a tap that mutates production data feel like a throwaway interaction** — for any `destructive` tier, the Approve button triggers a biometric challenge and a short hold-to-confirm, so a pocket-tap can’t authorize an irreversible write.

## 4. Performance: Optimizing React-Native State for High-Frequency Agent Logs

Agent logs are a firehose. A busy run emits dozens of progress events per second, and if each one triggers a React state update you will drop frames on mid-tier Android hardware within seconds. The fix is not “optimize later” — it’s a set of three disciplines applied from the first commit.

**First, batch at the frame boundary.** Never `setState` per event. Buffer incoming events and flush once per animation frame with `requestAnimationFrame`, collapsing multiple updates to the same call ID into the latest value. This turns a thousand dispatches per second into sixty.

“`ts
const buffer = useRef(new Map());
const raf = useRef(0);
const enqueue = (callId: string, pct: number) => {
buffer.current.set(callId, pct); // last-write-wins per call
if (!raf.current) raf.current = requestAnimationFrame(flush);
};
const flush = () => {
buffer.current.forEach((pct, id) => store.getState().setProgress(id, pct));
buffer.current.clear(); raf.current = 0;
};
“`

**Second, keep the store external and flat.** React state is for *what’s rendered*; the log itself lives in a plain observable store keyed by call ID (`Record`), never an array you filter linearly. Components subscribe to slices — a progress bar subscribes to one call ID and re-renders only when that number changes. Combine a shallow selector with `memo`’d row components and an event updating call #4,213 never touches the other 4,212.

**Third, bound everything.** Virtualize the visible log with `FlatList` and fixed row heights. Cap completed calls with a ring buffer — keep the last 2,000, aggregate older ones into counters — because nobody scrolls 20,000 rows on a phone, and memory on mobile is not free. Animate with `useNativeDriver` so progress and opacity changes run on the UI thread, not the JS thread that’s also parsing your event stream. Finally, throttle the *visual* update rate independently of the *data* rate: coalesce progress into ~15 Hz for display while keeping every event in the audit buffer. The operator perceives smoothness; the audit trail keeps fidelity.

The payoff: a phone that renders a live agent run at 60 fps, keeps the JS thread free for approvals, and survives an hour of streaming without a thermal throttle. Performance here isn’t polish — it’s the difference between an approver trusting the interface and closing the app.

## 5. Security: Biometric Approval and OAuth2 PKCE on Mobile Devices

A mobile client that can authorize production writes is a high-value target, so the security model has to hold up to a lost device and a hostile network. Two mechanisms carry the weight.

**OAuth2 with PKCE (Proof Key for Code Exchange).** Mobile apps are public clients — there’s no safe place to hide a client secret, so the authorization-code flow alone is insufficient. PKCE binds the token exchange to the same app instance that started it. The app generates a random `code_verifier`, sends its SHA-256 hash as the `code_challenge` with the authorization request, and proves possession by sending the raw verifier at token exchange. A malicious app that intercepts the redirect can’t complete the exchange without the verifier it never saw.

“`ts
import * as AuthSession from “expo-auth-session”;
import * as Crypto from “expo-crypto”;

const verifier = base64url(await Crypto.getRandomBytesAsync(32));
const challenge = base64url(await Crypto.digestStringAsync(
Crypto.CryptoDigestAlgorithm.SHA256, verifier, { encoding: “base64” }));

const auth = await AuthSession.startAsync({
authUrl: `${ISSUER}/authorize?response_type=code&client_id=${CLIENT_ID}` +
`&redirect_uri=${REDIRECT}&scope=${SCOPES}` +
`&code_challenge=${challenge}&code_challenge_method=S256`,
returnUrl: REDIRECT,
});
// Exchange auth.params.code + verifier for tokens, then store in SecureStore.
“`

Tokens live in the OS keychain (`expo-secure-store` / Keychain / Keystore), never in `AsyncStorage`, and access tokens stay short-lived with silent refresh. Bind the redirect URI to a verified app link so an attacker can’t hijack the callback.

**Biometric approval as a second signal.** OAuth answers “who is this device?” Biometrics answer “is the human present *right now*?” Those are different questions, and for destructive checkpoints you want both. Before a high-tier approval is accepted, the app runs a local biometric prompt (`expo-local-authentication` / `BiometricPrompt`), and the resulting approval is sent with a nonce the server binds to that checkpoint and a timestamp. The server rejects stale nonces and replays. Critically, the approval payload is signed with a key stored in the hardware-backed keystore, so a rooted device or a replayed request can’t forge consent.

“`ts
const ok = await LocalAuthentication.authenticateAsync({
promptMessage: `Approve ${cp.tool}`,
cancelLabel: “Cancel”,
disableDeviceFallback: true, // no PIN bypass for destructive writes
});
if (ok.success) {
const sealed = await SecureStore.getItemAsync(“signing_key”); // hardware-backed
await api.approve(cp.id, { nonce: cp.nonce, sig: await sign(sealed, cp.nonce) });
}
“`

Layer in the boring-but-essential hygiene: certificate pinning on the WebSocket, jailbreak/root detection that downgrades the client to view-only, short-lived server-side checkpoint tokens, and an audit record of *which device, which biometric, which nonce* authorized each action. In a regulated deployment, that audit trail is the whole reason the mobile approval exists.

## 6. Code Walkthrough: A React-Native ‘Approval Hook’

Here is the whole approval flow in one hook — pending checkpoints from the socket, biometric gating, optimistic UI, and durable rollback. It composes the pieces from the previous sections.

“`tsx
import { useCallback, useEffect, useReducer } from “react”;
import * as LocalAuthentication from “expo-local-authentication”;

type Pending = Record;
type Action =
| { type: “upsert”; cp: Checkpoint }
| { type: “remove”; id: string }
| { type: “resolve”; id: string; status: “approved” | “rejected” };

function reducer(state: Pending, a: Action): Pending {
switch (a.type) {
case “upsert”: return { …state, [a.cp.id]: a.cp };
case “remove”: { const { [a.id]: _, …rest } = state; return rest; }
default: return state; // resolution is optimistic elsewhere
}
}

export function useApprovals(socket: AgentSocket) {
const [pending, dispatch] = useReducer(reducer, {});

// Stream checkpoints into the reducer; replayed events survive reconnect.
useEffect(() => {
const off = socket.on(“event”, (ev) => {
if (ev.name === “checkpoint/opened”) dispatch({ type: “upsert”, cp: ev.data });
if (ev.name === “checkpoint/closed”) dispatch({ type: “remove”, id: ev.data.id });
});
return off; // unsubscribe on unmount
}, [socket]);

const approve = useCallback(
async (cp: Checkpoint) => {
if (cp.requiresBiometric) {
const ok = await LocalAuthentication.authenticateAsync({
promptMessage: `Approve ${cp.tool}`,
disableDeviceFallback: true,
});
if (!ok.success) return { ok: false, reason: “auth_cancelled” as const };
}
// Optimistic: hide the card immediately, reconcile on server ack.
dispatch({ type: “remove”, id: cp.id });
try {
await socket.request(“checkpoint/approve”, {
id: cp.id, nonce: cp.nonce, at: Date.now(),
});
return { ok: true as const };
} catch (e) {
dispatch({ type: “upsert”, cp }); // rollback: put the card back
return { ok: false, reason: “server_error” as const };
}
},
[socket],
);

const reject = useCallback(
async (cp: Checkpoint) => {
dispatch({ type: “remove”, id: cp.id });
await socket.request(“checkpoint/reject”, { id: cp.id });
},
[socket],
);

return { pending: Object.values(pending), approve, reject };
}
“`

Four things earn their place here. **The reducer owns pending state, not the socket** — so a reconnect that replays `checkpoint/opened` converges rather than duplicating. **Biometric gating is inside the action**, not scattered across call sites, so no button can bypass it. **The optimistic removal plus rollback** keeps the UI responsive on a slow mobile network while never lying about a failed approval — if the server rejects, the card returns. And **`disableDeviceFallback`** ensures a destructive approval can’t be satisfied by a PIN; the human presence signal has to be real. The list rendering then just maps `pending` to `CheckpointCard`, sorted by risk tier then age — and each card’s buttons are wired straight to `approve` and `reject`.

## Ship the Cockpit, Not Just the Agent

The pattern generalizes cleanly: terminate SSE in a gateway and fan it out over a resilient WebSocket, model checkpoints as first-class decision objects, render them as risk-ranked cards, keep high-frequency logs in a batched external store, and gate destructive approvals behind PKCE-scoped tokens *and* hardware-backed biometrics. None of it is exotic — but it is the difference between an agent that can only run under a watchful desk and one that ships trusted decisions from a phone in an approver’s pocket.

That’s exactly the kind of mobile control surface OnlineCloud builds. We’re a US-based engineering studio focused on MCP, autonomous agents, and applied ML — and we wire human checkpoints, streamable transports, and auditable auth boundaries into real systems on top of Oracle, MS SQL, Snowflake, and your CRM. Senior engineers only, no offshore handoffs, production in 6–10 weeks.

**[Tell us about your agent →](mailto:contact@onlinecloud.dev)** — free initial consultation, response within 24 hours. Or explore more at **[onlinecloud.dev](https://www.onlinecloud.dev/)**.

# Implementing Distributed Tracing for Multi-Step Agent Chains: OpenTelemetry for Reasoning


*A senior DevOps engineer’s field notes on making agent reasoning observable, auditable, and debuggable at production scale.*

—

## 1. The Visibility Gap in Multi-Agent Reasoning Chains

Here’s the uncomfortable truth about most “agentic AI” running in production today: when an agent produces a wrong answer, nobody can explain *why* without reading raw logs and praying. We’ve spent two years building elaborate orchestration — planner agents, executor agents, critic agents, retrieval agents wired together with LangGraph, AutoGen, or a bespoke loop — and almost none of it is observable in the way we’d demand of a five-hop microservice call.

Think about what a single user request actually triggers. A user says *”reconcile last quarter’s invoices against the Oracle ledger and flag discrepancies over $5k.”* Under the hood that fans out into: an intent parser, a query-planning LLM call, a schema-resolution tool, three SQL tool calls against Oracle, a chunked retrieval step over a policy document, a reflection pass that critiques the draft, a re-plan because the critic found a mis-joined column, and finally a synthesis call. That’s a reasoning chain twelve to forty spans deep, spanning an LLM provider, two databases, a vector store, and an MCP server.

Now ask the standard DevOps questions:

– **Where did the time go?** Was the 11-second p99 the Oracle query, the embedding lookup, or a cold-start on the planner?
– **Which step introduced the error?** Did the planner mis-decompose the task, or did a tool return stale data that the synthesizer faithfully narrated?
– **How much did this request cost?** Token accounting per reasoning step, not per HTTP request.
– **Who approved what?** For anything touching financial or regulated data, an auditor wants a causal chain, not a vibe.

The visibility gap exists because agent frameworks optimize for *composability* and we as operators have been lazy about *instrumentation*. A web request has a natural unit of work — the HTTP handler. A reasoning chain doesn’t. Its units are cognitive: planning, tool selection, acting, observing, reflecting. We inherited the tracing tooling; we just never mapped it onto this new execution shape.

OpenTelemetry changes that, but only if you instrument deliberately. This post is how I do it.

—

## 2. OpenTelemetry for AI: Instrumenting Spans for Planning, Acting, and Reflecting

The core insight: **a reasoning step is a span.** OpenTelemetry’s data model — traces composed of nested spans, each with a name, start/end timestamps, attributes, events, and status — maps almost perfectly onto agent cognition. I define a canonical span taxonomy and enforce it across every agent:

| Span name | Represents | Key attributes |
|—|—|—|
| `agent.plan` | Task decomposition | `agent.name`, `plan.step_count`, `llm.model`, `llm.tokens.input/output` |
| `agent.act` | A single tool/action invocation | `tool.name`, `mcp.server`, `tool.retry_count` |
| `agent.observe` | Processing a tool result | `observation.truncated`, `observation.byte_size` |
| `agent.reflect` | Self-critique / verification | `reflect.verdict`, `reflect.confidence` |
| `llm.call` | A raw model round-trip | `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.*` |
| `retrieval.query` | Vector / hybrid search | `db.system`, `retrieval.top_k`, `retrieval.hit_count` |

A parent `agent.run` span wraps the whole thing so the trace reads top-down as a causal narrative. The nesting is what gives you the waterfall: `agent.run` → `agent.plan` → `agent.act` (`llm.call` + `retrieval.query`) → `agent.reflect` → (re-plan loop) → synthesis.

The reason I anchor on OpenTelemetry specifically — rather than a proprietary LLM-observability SDK — is vendor neutrality and semantic-convention alignment. OTel has active GenAI semantic conventions (`gen_ai.*` attributes), and instrumenting against the standard means the same spans render in Jaeger, Honeycomb, Grafana Tempo, or Datadog without a rewrite. For a studio with regulated clients, “your traces leave with you” is a real procurement requirement.

Two implementation notes that matter more than they look:

**Never put raw prompts in span attributes.** Attributes are indexed and replicated. Put `prompt_hash`, `prompt_template_id`, and token counts on the span; ship the full prompt payload to a separate content store and link via `trace_id`. This keeps PII out of your observability backend and keeps cardinally sane.

**Emit reasoning milestones as span events, not child spans.** A reflection that emits three sub-observations doesn’t need three spans; use `span.add_event(“critique”, {…})` so the trace stays readable. Spans for *work*, events for *moments*.

“`python
from opentelemetry import trace

tracer = trace.get_tracer(“agentic.runtime”)

def plan(task: str, agent) -> dict:
with tracer.start_as_current_span(“agent.plan”) as span:
span.set_attribute(“agent.name”, agent.name)
decision = agent.llm.plan(task)
span.set_attribute(“plan.step_count”, len(decision.steps))
span.set_attribute(“llm.model”, decision.model)
span.add_event(“plan_emitted”, {“strategy”: decision.strategy})
return decision
“`

That said, do not hand-roll every span. Use auto-instrumentation for the noisy infrastructure (`opentelemetry-instrumentation-httpx`, `-sqlalchemy`, `-psycopg`, `-redis`) so database and HTTP calls appear for free, and reserve manual spans for *cognitive* boundaries. The split keeps your code clean and your traces complete.

—

## 3. Context Propagation: Flowing Trace IDs from User Intent to Deep-Tier Tool Calls

This is where most agent deployments quietly fail, and it’s the single hardest part of the whole design.

In a normal HTTP system, propagation is solved by W3C `traceparent` headers injected by the SDK on every outbound call. Agents break this in three ways:

1. **LLM providers strip headers.** You call an OpenAI/Anthropic/Bedrock endpoint — the vendor doesn’t echo your `traceparent`. The model call can’t join the distributed context via headers.
2. **Tool calls are often in-process or via non-HTTP transports.** MCP servers over stdio, local Python functions, or queue-based executors have no HTTP boundary to carry a header.
3. **Reasoning is stateful and resumable.** A long-running agent may pause, serialize its state to a queue, and resume minutes later — sometimes in a different process or pod. The trace context must survive serialization.

The fix is to treat trace context as **application state**, not just transport metadata:

– **Carry `trace_id` + `span_id` explicitly in the agent’s message/state object.** Every unit of agent work — a tool call request, a queued step, a resumed checkpoint — includes a `trace_context` field. When you resume from a queue, you call `trace.get_tracer(…)` with an extracted `Context`, re-parenting the new work under the original trace.
– **For LLM calls, create the span in your process and attach the model call as a child.** Don’t expect the provider to propagate. Your `llm.call` span starts before the HTTP request and ends after the response, so latency and token usage are captured regardless of what the provider does.
– **For MCP servers you control, use the standard.** Streamable HTTP transports *can* carry `traceparent`. Build MCP servers that honour W3C context, and stdio-based servers should read the injected context from the envelope. This is exactly the kind of “typed tools, declarative resources, auditable transports” contract that separates a real MCP deployment from a slower API gateway.
– **Propagate into the message envelope for queues.** If steps land on Kafka/SQS/Redis Streams, write `traceparent` into the message headers on publish, extract on consume. This makes cross-pod, cross-region reasoning chains a single contiguous trace.

“`python
from opentelemetry import trace, context, propagate

# On enqueue (planning process)
carrier = {}
propagate.inject(carrier)

message = {“task”: step.task, “trace_context”: carrier}
queue.publish(message)

# On dequeue (worker process)
ctx = propagate.extract(message[“trace_context”])
with tracer.start_as_current_span(“agent.act”, context=ctx) as span:
result = execute_tool(step.tool, step.args)
“`

When this is done right, a single `trace_id` spans user intent → orchestration → every LLM call → every tool → every downstream database query. That end-to-end provenance is what makes the next two sections possible, and it’s non-negotiable for anything audit-facing. A trace that *ends* at your agent’s edge isn’t a distributed trace — it’s a monolith with good PR.

—

## 4. Visualization: Building Jaeger and Honeycomb Dashboards for Agentic Latency Waterfalls

Once spans flow correctly, the payoff is operational. I run two backends side by side because they answer different questions.

**Jaeger (or Grafana Tempo) for the forensic view.** Jaeger’s waterfall is the right tool when you’re staring at one slow or broken request. Agent-native conventions make it readable: colour spans by `span.kind` or by an `agent.role` attribute so planning, acting, and reflecting visually distinguish themselves. My non-negotiables in the Jaeger view:

– **Group the loop.** Collapse re-plan iterations so a five-round reflection cycle doesn’t explode the waterfall into noise.
– **Annotate the model.** A trace-level tag for `llm.model` and `agent.workflow_id` makes “is GPT-4-class slow today, or Claude?” a one-click filter.
– **Surface the critical path.** Nested spans mean wall-clock ≠ sum of spans. The interesting number is the critical path spanning a synchronous tool call while the planner waits.

**Honeycomb for the aggregate view.** This is where agent observability genuinely diverges from classic APM. Honeycomb’s strength is high-cardinality, slice-and-dice exploration over attributes — and agent traces are *dripping* with high-cardinality dimensions: `agent.name`, `tool.name`, `mcp.server`, `plan.step_count`, `reflect.verdict`, `retrieval.hit_count`. The dashboards I build there:

1. **Latency waterfall by reasoning phase.** A heatmap of `agent.plan`, `agent.act`, `agent.reflect`, and `llm.call` durations. The pattern that recurs: planning and reflection dominate tail latency, not tool execution. Teams routinely optimize the wrong thing because they can only see the HTTP layer.
2. **Cost per reasoning step.** A `GROUP BY agent.name` of summed `gen_ai.usage.total_tokens`. You find fast that a “cheap” agent does seven reflection cycles and costs more than the big model doing one.
3. **Tool-level error and retry SLOs.** `tool.retry_count > 0` sliced by `tool.name` — the quiet indicator that an integration is flaky under load.
4. **Step-count distribution.** `plan.step_count` histograms expose “runaway agents” that spiral into dozens of steps — your first line of defence on cost and latency.

A practical dashboard note: **sample smart, not uniform.** LLM calls are expensive to trace if you capture full payloads; keep 100% of spans (they’re cheap metadata) but sample *content* capture at 1–5%, always forcing capture on errors or when `cost > threshold`. This gives you full latency fidelity at a fraction of the storage.

—

## 5. Error Analysis: Using Traces to Debug Hallucination Drift in Multi-Step Tasks

“Hallucination drift” is the failure mode where each reasoning step is *locally plausible* but the chain drifts from ground truth — and it’s nearly impossible to debug from the final output alone. Traces let you localize the exact step where the chain left the rails. My debugging playbook:

**Bisect the trace, don’t guess at the output.** Take a failing `agent.run`, open the waterfall, and walk top-down comparing each `agent.observe` span’s captured tool result against the `agent.act` that requested it. Drift typically appears at one of three seams:

– **Plan drift** — the decomposition was wrong at step 1 (e.g., the planner asked for “invoices” when it needed “credit notes”). The whole chain is doomed; reflection never catches it because every subsequent step is individually valid. Signal: `reflect.verdict = “accept”` on every iteration *and* a wrong final answer. That combination is the fingerprint of upstream plan drift.
– **Observation drift** — a tool returned correct data, but the synthesizer mis-read a column or an entity. Signal: `observation.byte_size` is normal, but a downstream `llm.call` produces a claim that contradicts the tool payload. With content capture sampled on, you diff prompt-in vs. output.
– **Reflection failure** — the critic approved a flawed step. Signal: `reflect.confidence` high while the answer is wrong; your verification prompt is too lenient. This is where you tune the critic, and the traces tell you *which* critiques to harden.

**Instrument the loop counter and the “stuck” signal explicitly.** Add attributes for `loop.iterations` and an event `replan` with the reason. Runaway loops and thrashing re-plans — the two most expensive drift symptoms — become trivial `WHERE loop.iterations > N` queries.

**Close the loop with evals.** Every trace you flag as drifted should become a labelled eval case. The trace is the reproduction; the eval is the regression test. Wire your OTel pipeline so a `drift.confirmed=true` tag exports the trace ID into the eval suite automatically. Over time this turns “the agent got weird last Tuesday” into a versioned, reproducible incident with a fix and a guard test.

The net effect on the team: mean time to *diagnosis* collapses from “hours of log spelunking across three services” to “open the trace, find the seam, ship a targeted fix.” That’s the difference between operating agents and merely running them.

—

## 6. The OnlineCloud Protocol: Audited, Observable Agent Delivery in 6–10 Weeks

All of the above is the architecture I’d build for any serious agent deployment — and it’s exactly what we deliver at **OnlineCloud**. We’re a US-based engineering studio (no offshore handoffs) focused on Model Context Protocol, autonomous agents, and applied ML, and our delivery model is built around the same discipline this post describes.

**Why observability is a first-class deliverable, not an afterthought.** We ship *evaluated, observable systems*. Every agent we build comes wired to an OTel pipeline from day one, because retrofitting tracing onto a live agent chain is a nightmare and because our clients — often in regulated, Oracle/MS SQL-backed environments — need audit-ready causal chains, not vibes. Plan/act/reflect loops ship with cost ceilings, human checkpoints, and full interaction logging out of the box.

**The 6–10 week protocol:**

– **Week 1 — Discovery sprint (fixed scope).** We map your data, pick the model class, and — critically — write the eval that defines “done.” No eval, no build.
– **Weeks 2–7 — Vertical slices.** One usable slice every 1–2 weeks against real data and real users: typed MCP tools over your actual schema, a live trace pipeline, a Jaeger/Honeycomb dashboard, and a cost/latency SLO. You demo working code, not status decks.
– **Weeks 8–10 — Operate & harden.** We hand over dashboards, retraining cadence, rollback paths, and on-call runbooks. The same architect who scoped it is the engineer who pushed the commits.

Concretely, a finished OnlineCloud engagement leaves you with: stateless MCP servers with auth, rate limits, and audit logs; streamable HTTP/SSE transports; connectors to Oracle, MS SQL, Postgres, and Snowflake with row-level policies; an OpenTelemetry-based tracing layer with reasoning-phase spans; and dashboards that measure outcomes in accuracy, latency, and dollars-per-request.

We instrument outcomes and share the numbers. That’s the whole point of tracing — if you can’t see it, you can’t ship it.

—

## Ready to Make Your Agents Observable?

If your multi-agent system works in the demo and terrifies you in production, let’s talk. We’ll map your reasoning chains, wire up end-to-end tracing, and ship an audited, observable agent — to production in **6–10 weeks, not quarters**.

**[→ Book a free initial consultation](https://www.onlinecloud.dev/contact)** — US-based senior engineers, response within 24 hours.

Have a project in mind? Email **contact@onlinecloud.dev** or visit **[onlinecloud.dev](https://www.onlinecloud.dev/)** to see how our team brings observability to agentic AI.

*Xlent Info tech. LLC — Agentic AI & MCP Engineering Studio. MCP-native architecture, audited end-to-end.*

# Scaling Inference for 10k+ Concurrent Agents on EKS: HPA, vLLM, and Spot Optimization


Running a fleet of autonomous agents is not the same problem as running a chatbot. A chatbot waits for a human to type. An agent fan-out does not. A single orchestration request can expand into twenty tool calls, a dozen retrieval hops, and a reflection loop that re-prompts the model three times before it decides it is done. Ten thousand concurrent agents is therefore not ten thousand concurrent requests — it is closer to two hundred thousand model invocations per minute, each with different token profiles, different latency budgets, and different preemption risk. This is the resource-density challenge, and it is where most inference platforms quietly fall over.

This deep dive is the architecture we use at [OnlineCloud](https://www.onlinecloud.dev/) when we provision GPU inference for agentic workloads on Amazon EKS. We will cover cluster scaling with Karpenter, inference engines (vLLM and TensorRT-LLM), request-per-token autoscaling with KEDA and HPA, a fault-tolerant spot strategy that cut compute spend by roughly 70%, and the benchmark data that tells you when to trade latency for throughput.

## 1. The Resource-Density Challenge of Large Agent Fleets

An agent fleet has a fundamentally different load shape from a REST API. Three properties dominate capacity planning.

**Burst amplification.** A single plan step can multiply into N tool calls. If each tool call requires an LLM judgment, your steady-state QPS is a poor predictor of peak GPU demand. We typically see a 6×–15× amplification factor between ingress requests and actual inference calls at p99.

**Variable token economics.** An extraction call returns 80 tokens; a summarization call returns 900. Autoscaling on request count alone will either under-provision the long-context pods or waste capacity on the short ones. You need a metric that reflects *work*, not just *arrivals*.

**Cold-start intolerance.** Agents hold state — a conversation, a scratchpad, a partially completed tool chain. When a node dies mid-loop, the agent must be resumable, or the user sees a broken experience.

The density math is unforgiving. An H100 80GB running a 70B model in FP8 with a healthy KV-cache budget can hold on the order of 100–200 concurrent sequences at 4k context. To reach 10k concurrent sessions you are provisioning 60–100 GPU nodes at peak — which is exactly why the spot question is not optional. At on-demand prices that fleet is a five-figure monthly line item; the same fleet on spot is roughly 30% of that.

## 2. Scaling H100/A100 Clusters with EKS and Karpenter

We standardize on EKS with Karpenter for node provisioning. The managed control plane gives us the API surface, IAM integration for IRSA, and cluster autoscaler primitives we do not want to maintain. Karpenter gives us something the legacy Cluster Autoscaler cannot: sub-minute node provisioning driven by actual pod scheduling pressure, and heterogeneous instance selection across GPU families.

The core pattern is **NodePool diversity**. We never pin a workload to a single instance type. Instead we define NodePools with `requirements` that accept a set of compatible GPU SKUs (p5, p4d, p4de, and A100-backed g5 families) and let Karpenter pick the cheapest available capacity that satisfies the pod’s resource request.

“`yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: gpu-inference
spec:
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 5m
limits:
cpu: “2000”
nvidia.com/gpu: “256”
template:
spec:
requirements:
– key: karpenter.sh/capacity-type
operator: In
values: [“spot”, “on-demand”]
– key: node.kubernetes.io/instance-type
operator: In
values: [“p5.48xlarge”, “p4d.24xlarge”, “p4de.24xlarge”, “g5.48xlarge”]
taints:
– key: nvidia.com/gpu
effect: NoSchedule
“`

Three things matter here. First, **weigh pods by real GPU footprint** — a 70B model needs a whole node; a 7B router model can bin-pack. Second, **taint GPU nodes** so only GPU-aware workloads land there; system daemonsets should never consume a full-precision slot. Third, **enable consolidation** with `WhenEmptyOrUnderutilized`. Agent traffic is bursty; without consolidation you pay for idle H100s between waves.

For AWS for AI workloads specifically, we also split NodePools by latency class: an interactive pool with `consolidateAfter: 30m` (cheap to lose, expensive to wait for) and a batch/offline pool with `consolidateAfter: 1m`. Mixing the two in one pool forces you to choose the wrong consolidation policy for half your traffic.

## 3. Inference Engines: vLLM and TensorRT-LLM for Multi-Agent Throughput

The engine choice is the single largest lever on cost-per-token.

**vLLM** is our default for multi-tenant agent serving. PagedAttention eliminates the KV-cache fragmentation that kills naive implementations, and continuous batching means a decoding step is never blocked waiting for the slowest sequence. For agent fleets this matters because a reflection loop and a classification call can share a batch efficiently. Prefix caching is a second-order win: agent system prompts are long and identical across sessions, so we cache the shared prefix and pay prefill once instead of ten thousand times. Typical gain on a 2k-token system prompt: 40–60% reduction in prefill compute.

**TensorRT-LLM** is our choice when a model is fixed and latency is the binding constraint. Compiled kernels, in-flight batching, and FP8/INT4 quantization on H100 give us 20–40% lower time-to-first-token than vLLM for the same model at high concurrency. The trade-off is rigidity: recompilation on model or shape change, longer cold starts, and a heavier image. We run TensorRT-LLM behind a **router** that sends latency-critical traffic (the user-facing first token) to TRT-LLM pods and everything else — tool calls, batch summarization, evaluator passes — to vLLM pods.

“`python
# Router: latency-critical traffic to TRT-LLM, everything else to vLLM
ROUTES = {
“agent.first_token”: “trtllm-pool”,
“agent.tool_call”: “vllm-pool”,
“agent.evaluator”: “vllm-pool”,
“batch.summarize”: “vllm-pool”,
}
“`

Both engines are exposed through a single OpenAI-compatible gateway so agent code never hardcodes an engine. We front them with an L7 proxy that does token-aware load balancing rather than round-robin — routing by estimated prompt+max-tokens so a long-context request does not land on a pod already at its KV-cache ceiling.

For multi-agent throughput, the practical configuration that scaled us to 10k sessions was: vLLM with `–max-num-seqs 256`, chunked prefill enabled, and a **per-pod KV-cache headroom target of 15%**. Headroom is not waste — it is the buffer that absorbs the burst amplification described earlier without triggering preemption.

## 4. Auto-Scaling with KEDA and HPA on Request-per-Token Metrics

Standard HPA scales on CPU. For GPU inference that is close to useless: an H100 at 100% SM utilization and one at 20% may both report the same pod CPU, because the CPU is just feeding the GPU. We scale on **work**, using a custom metric we call *tokens in flight* — the sum of prefill and decode tokens outstanding across a pod’s active sequences.

The pipeline is: engine exposes Prometheus metrics (`vllm:num_requests_running`, `vllm:gpu_cache_usage_perc`, plus a derived token counter) → Prometheus scrapes → **KEDA’s prometheus scaler** or the Prometheus Adapter exposes them to the HPA → HPA drives the deployment.

“`yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-agents
spec:
scaleTargetRef:
name: vllm-deployment
minReplicaCount: 4
maxReplicaCount: 96
cooldownPeriod: 180
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
– type: Percent
value: 100
periodSeconds: 30
scaleDown:
stabilizationWindowSeconds: 600
policies:
– type: Pods
value: 4
periodSeconds: 60
triggers:
– type: prometheus
metadata:
serverAddress: http://prometheus.monitor:9090
query: |
sum(rate(vllm:tokens_in_flight[1m]))
threshold: “180000” # tokens/sec of aggregate capacity per replica
“`

Two behaviors make this work at fleet scale. **Asymmetric scaling windows**: scale up aggressively (no stabilization, 100%/30s) because an under-provisioned fleet drops requests; scale down slowly (600s window, 4 pods/60s) because thrashing a GPU node costs more than keeping it warm. **Threshold expressed in tokens per second per replica**, tuned to the model. A 70B model at 4k context sustains a very different token rate than a 7B router, so we run separate ScaledObjects per model pool rather than one global rule.

We pair this with **predictive pre-warming** for known diurnal patterns — KEDA’s cron trigger scales the floor up before the morning wave — and a queue-depth guardrail: when the KEDA query is unreliable, a fallback trigger on the gateway’s pending-request count prevents a silent undercount.

## 5. Spot Instances: A Fault-Tolerant Agent Loop for ~70% Cost Reduction

Spot is where the economics change. H100 spot capacity in us-east-1 has historically run 60–75% below on-demand, and A100 families similar. On a 60–100 node fleet that is the difference between a viable product and a broken unit economics story. The catch is the two-minute interruption notice.

We built fault tolerance into the **agent loop**, not bolted onto the infrastructure. The principles:

1. **Checkpoint every step.** Agent state — the plan, completed tool results, scratchpad — is persisted to Redis (or DynamoDB for durability) after each step, keyed by session. A preemption costs at most one step of rework.
2. **Make steps idempotent.** Every tool call carries an idempotency key, so a replayed step does not double-charge a payment API or re-issue a ticket.
3. **Drain, do not die.** A DaemonSet watches the instance metadata endpoint for the spot interruption notice and immediately cordons the node and flips the pod’s readiness probe. The gateway stops routing new sessions; in-flight sequences finish their current step and checkpoint.
4. **Spread the risk.** We configure `topologySpreadConstraints` so no single interruption event can take more than a fraction of a model pool, and we keep a **20% on-demand floor** per pool with spot as the burst capacity.

“`yaml
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
topologySpreadConstraints:
– maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels: { app: vllm }
containers:
– name: interruption-watcher
image: onlinecloud/spot-drain:1.4
env:
– name: DRAIN_GRACE_SECONDS
value: “90”
“`

With checkpointing, idempotent tools, and graceful drain in place, preemption became a non-event: our measured p99 session abort rate dropped below 0.05%, which is lower than the abort rate from ordinary pod OOM kills. Combined with Karpenter’s bin-packing and the consolidation policy, the blended compute cost landed at roughly **30% of the pure on-demand baseline — a 70% reduction**.

## 6. Benchmarking: Latency vs. Throughput at 10k Concurrent Sessions

We benchmark every pool before it ships, and we report three numbers, not one: **TTFT p50/p99** (time to first token), **inter-token latency p99** (the metric users feel while streaming), and **tokens/sec/GPU** (the cost metric). Optimizing one in isolation is how teams ship a fast demo and an expensive product.

At 10k concurrent sessions across the mixed fleet, our reference results were:

| Configuration | TTFT p99 | ITL p99 | Tokens/s/GPU | Relative $/1M tok |
|—|—|—|—|—|
| vLLM, no batching, on-demand | 180 ms | 42 ms | 1,900 | 1.00× |
| vLLM + continuous batching | 240 ms | 38 ms | 4,600 | 0.41× |
| vLLM + batching + prefix cache | 210 ms | 38 ms | 6,400 | 0.30× |
| TensorRT-LLM FP8 (latency path) | 140 ms | 19 ms | 3,800 | 0.33× |
| Blended (TRT-LLM + vLLM) on spot | 165 ms | 26 ms | 6,100 | 0.11× |

The headline is the trade curve. Adding continuous batching raises TTFT slightly at p99 — more work queued ahead of you — while cutting cost per token by more than half. Prefix caching claws most of the TTFT back because the shared agent system prompt no longer re-prefills. TensorRT-LLM wins outright on latency but carries a lower throughput ceiling per GPU, which is exactly why the blended, routed architecture beats either engine run alone. Spot does not affect latency at all once drain and checkpointing are correct; it only moves the cost column.

Two saturation signals tell you the fleet is about to degrade: `gpu_cache_usage_perc` climbing past ~85% (KV-cache pressure → preemption storms) and inter-token latency p99 diverging from p50 by more than 3× (batching is starving interactive sequences). We alert on both, and the KEDA threshold above is tuned so the fleet scales before either fires.

## The Takeaway

Ten thousand concurrent agents is a density problem before it is a scaling problem. Answer it with Karpenter’s heterogeneous, consolidating NodePools; serve it with vLLM for throughput and TensorRT-LLM for latency behind a single routed gateway; scale it on tokens-in-flight rather than CPU; and fund it with a spot strategy that treats preemption as an expected event rather than an incident. Done right, the fleet holds p99 latency under 200 ms while paying roughly 11 cents per million tokens instead of a dollar.

—

**Building an agent fleet and watching the GPU bill climb?** OnlineCloud designs and operates the full stack — EKS GPU capacity planning, vLLM/TensorRT-LLM serving, KEDA autoscaling, and the spot economics that make agentic AI affordable at scale. We are a US-based senior engineering studio, and the architect who scopes your workload is the engineer who ships it. [Book a free 30-minute architecture review](https://www.onlinecloud.dev/#contact) and we will map your concurrency targets to a costed cluster design in a single call.

# Building a Custom Evaluator for Code-Generation Agents: The Sandbox Pattern


*By the engineering team at [OnlineCloud](https://www.onlinecloud.dev/) — we build evaluated, observable agent systems that ship to production in 6–10 weeks.*

—

## Introduction: Why ROUGE and BLEU Are the Wrong Ruler for Code

If you have ever shipped an LLM feature that generates code, you have probably felt the same quiet panic: the model looks correct, the demo is smooth, and yet you have no idea whether the thing actually *works*. So teams reach for the metrics they already know — ROUGE, BLEU, METEOR, maybe BERTScore if they are feeling fancy. These are lexical overlap metrics from machine translation and summarization. They measure how similar a generated string is to a reference string. Code is not a string. Code is a specification that either executes correctly or does not.

Consider a concrete example. A user asks for a function that returns the nth Fibonacci number. The reference implementation uses an iterative loop with two accumulator variables. Your agent returns a memoized recursive version. The two functions are functionally identical — same inputs, same outputs, same complexity class in practice — but they share almost no surface tokens. ROUGE-L would score that pair somewhere in the 0.3–0.4 range and tell you your model is mediocre. It is not.

Now invert the example. The agent returns the reference implementation character-for-character, but with one token changed: `i <= n` becomes `i < n`. ROUGE-L is near 1.0 — a "perfect" score. The function now silently off-by-one's on every input. Your metric gave the broken code a gold medal. This is the fundamental failure mode: **lexical metrics are invariant to the only property that matters, which is behavior.** They reward paraphrasing failure and punish correct alternate solutions. For code generation they are worse than useless — they are actively misleading, because they create false confidence in CI dashboards. There are three deeper problems: 1. **The reference-bias problem.** ROUGE/BLEU assume a single canonical answer. Real code has an enormous space of correct solutions — different algorithms, different style conventions, different library choices. A metric that privileges one reference is measuring "did you copy the answer key," not "did you solve the problem." 2. **The non-compositional problem.** Meaning in code is compositional and scoped. A metric that sums token overlap cannot see that a renamed variable is harmless while a flipped comparison operator is fatal. It has no grammar, no scope, no types. 3. **The non-executable problem.** The metric never runs anything. Bugs are exactly the behaviors that survive static similarity checks. Off-by-one errors, null dereferences, resource leaks, race conditions — all of these produce near-perfect string similarity and catastrophic runtime behavior. The fix is not a better similarity metric. The fix is to stop comparing text and start **executing behavior**. That means a custom evaluator that treats the model's output as a candidate program, runs it in an isolated environment against a suite of tests, and scores it on what actually happened. This post is the engineering blueprint for that evaluator — the **Sandbox Pattern**. --- ## The Sandbox Pattern: Secure, Ephemeral Execution Environments The core idea of the Sandbox Pattern is deceptively simple: **never evaluate generated code on a machine you care about.** Instead, for every candidate program, spin up a disposable, resource-bounded, network-isolated execution environment, run the tests, harvest the results, and destroy the environment. The evaluator itself never trusts the artifact it is scoring. ### Why you cannot just use a subprocess The naive approach — `subprocess.run(["python", candidate.py])` on your eval worker — is the single most common mistake we see. It fails on every axis that matters: - **Filesystem escape.** Generated code runs with your worker's user privileges and can read `~/.aws/credentials`, `.env` files, or write a persistence hook into your CI runner. - **Network egress.** It can exfiltrate environment variables, call home, or scan your internal VPC. - **Resource exhaustion.** An infinite loop or a fork bomb takes down the worker, not the candidate. - **Resource leakage.** Leftover processes, open sockets, and temp files accumulate until the evaluator itself becomes unstable. - **Non-determinism.** The candidate mutates its own environment between runs, so scores are not reproducible. A container alone improves things but does not solve them. Docker shares the host kernel, and a kernel exploit in generated code — or a cleverly crafted `syscall` — escapes the container entirely. Containers provide isolation for trusted workloads; generated code is definitionally untrusted. ### Kernel-level isolation: gVisor and Firecracker To evaluate untrusted code, you need a real isolation boundary. There are two production-grade options, and they use different philosophies. **gVisor** inserts a user-space kernel between the untrusted program and the host. When the candidate program makes a syscall, gVisor's `Sentry` intercepts it and re-implements the syscall in memory, brokering only a tiny, vetted surface down to the real host kernel. The effect: the code *thinks* it is talking to Linux, but the host kernel sees only a narrow, audited set of operations. gVisor gives you strong syscall-level defense with startup times measured in tens of milliseconds — fast enough to run per-test-case if you need to. **Firecracker** takes a different route: microVMs. Each evaluation runs inside a lightweight KVM-based virtual machine with its own guest kernel, booted in roughly 125 milliseconds and consuming a few megabytes of memory. Because the guest has its own kernel, the isolation boundary is hardware-virtualization-grade — the same primitive AWS Lambda uses under the hood. Firecracker is the right choice when you need the strongest possible boundary, when candidates may be multi-language or need a full OS surface, or when regulatory constraints demand VM-level separation. A practical architecture uses both: **gVisor for fast, high-volume inner-loop scoring** during development, and **Firecracker microVMs for the outer, untrusted boundary** in CI and production. Some teams nest them — a Firecracker microVM running a gVisor sandbox — for defense in depth. ### The ephemeral lifecycle Whatever the isolation primitive, the lifecycle is always the same five phases: 1. **Provision.** Create a fresh, minimal root filesystem from a pre-built image. No secrets, no host mounts, no persistent volumes. 2. **Inject.** Write the candidate program and the test harness into an in-memory or tmpfs location. Mount read-only where possible. 3. **Constrain.** Apply hard limits *before* execution: CPU quota (e.g., 2 vCPU-seconds), memory ceiling (e.g., 512 MB), wall-clock timeout (e.g., 5s), process/thread cap, file-descriptor cap, and a blocked network namespace. 4. **Execute.** Run the tests, stream stdout/stderr to a bounded buffer, and collect exit codes. 5. **Destroy.** Tear down the environment unconditionally, discarding the entire filesystem. Nothing persists. The next candidate starts from a pristine image. The word that carries the most weight here is **ephemeral**. Cold-start-per-evaluation is a feature, not a cost, because it eliminates cross-contamination: candidate A cannot poison the environment that scores candidate B, and a flaky candidate cannot leave residue that corrupts future scores. ### Implementation sketch A minimal evaluator loop in Python, using a sandbox abstraction that maps to gVisor or Firecracker at runtime: ```python from dataclasses import dataclass from sandbox import Sandbox, Limits # gVisor/Firecracker-backed @dataclass class EvalResult: passed: int total: int timed_out: bool stderr: str def evaluate(candidate_src: str, tests: list[TestCase]) -> EvalResult:
limits = Limits(
cpu_seconds=2.0,
memory_mb=512,
wall_timeout_s=5.0,
max_procs=64,
network=”none”, # network namespace fully isolated
filesystem=”ephemeral”, # discarded after run
)
with Sandbox(image=”python:3.12-slim”, limits=limits) as sb:
sb.write(“/work/solution.py”, candidate_src)
sb.write(“/work/test_solution.py”, render_tests(tests))
proc = sb.run(
[“pytest”, “-q”, “/work/test_solution.py”, “–timeout=3”],
capture=True,
)
return EvalResult(
passed=proc.parsed.passed,
total=len(tests),
timed_out=proc.exit_code == 124,
stderr=proc.stderr[:4096], # bounded, never unbounded
)
“`

The `with` block guarantees teardown. The `Limits` object is the security contract. The `network=”none”` and `filesystem=”ephemeral”` flags are the two that prevent the most damage.

—

## Unit-Test Scoring: Measuring Behavior, Not Text

With a sandbox in place, the scoring function becomes almost trivial — and dramatically more honest. Instead of comparing strings, you run tests and count passes.

### The scoring formula

A practical composite score weights several signals:

“`python
def functional_correctness(result: EvalResult) -> float:
if result.total == 0:
return 0.0
base = result.passed / result.total # test-pass rate
if result.timed_out:
base *= 0.5 # penalize non-termination
return round(base, 4)
“`

The **test-pass rate** — `passed / total` — becomes your primary functional metric. It is monotonic in the right direction: fixing a bug increases it, breaking a feature decreases it. It is invariant to how the code is written and sensitive to what the code does. That is exactly the property ROUGE lacks.

### Designing the test suite

The test suite is now the most important artifact in your evaluator, because it *is* your specification. Three tiers work well:

– **Canonical cases.** Straightforward inputs and expected outputs. These catch the obvious failures and anchor the score.
– **Edge cases.** Empty inputs, single elements, maximum sizes, negative numbers, unicode, nulls. These are where LLMs fail most, and where lexical metrics are blindest.
– **Property-based cases.** Instead of fixed input/output pairs, assert invariants: sorting is idempotent, encode/decode round-trips, output length is bounded, a search function returns an element that compares equal to the target. Property tests generalize far beyond the examples you write, which makes them excellent at catching overfitting.

### Partial credit and weighting

Not all test cases are equal. A schema-validation failure is less severe than a crash on the happy path. Weight cases accordingly — core functionality at 1.0, edge cases at 1.5, performance-sensitive cases with their own budgets — so the score reflects *severity* as well as *frequency*.

### Why this resists gaming

A concern you should anticipate: can the model game the tests? Only if the tests are visible to the model and shallow. Mitigations:

1. **Hold out tests.** Score against a private suite the model never sees; keep a public subset for the model to self-check against.
2. **Randomize inputs.** Generate test parameters at evaluation time from a seed, so memorized answers do not survive.
3. **Adversarial cases.** Include inputs specifically designed to break shortcuts and hardcoded lookups.
4. **Cross-check properties.** Where possible, verify with an independent oracle rather than a fixed constant.

—

## Security: Preventing Escape from the Evaluator Sandbox

The uncomfortable truth about evaluating an adversary’s code — and a code-generation model can absolutely produce adversarial output, either through a hostile prompt or a data-poisoning attack — is that your evaluator is now an attack surface. Treat it as one.

### The threat model

Assume generated code will attempt, at minimum:

– **Credential theft** — reading environment variables, `~/.ssh`, cloud metadata endpoints.
– **Network exfiltration** — POSTing data to an attacker-controlled host.
– **Cryptojacking** — running a miner on your GPU or CPU budget.
– **Resource exhaustion** — infinite loops, fork bombs, memory bombs, zip bombs.
– **Sandbox escape** — syscall abuse, kernel exploits, container breakouts.
– **Supply-chain tampering** — `pip install` of a malicious package, or modifying the test harness itself.

### Defense-in-depth controls

**Network.** Default to no network at all. Run the candidate in a network namespace with no interfaces. If a task genuinely needs a package, pre-populate a local wheelhouse and serve it read-only *outside* the sandbox’s reachable surface — never give the candidate open egress. Block cloud metadata IPs (`169.254.169.254`) explicitly even when networking is nominally “on.”

**Filesystem.** Mount the root read-only. Give the candidate a single writable tmpfs mount, size-bounded, discarded on teardown. Never mount the Docker socket, the host `/proc`, or any credential path. Strip all secrets from the environment before execution — a sandbox with `AWS_ACCESS_KEY_ID` in its env is not a sandbox.

**Resources.** Enforce limits with the kernel, not with convention: cgroups v2 for CPU and memory, `RLIMIT_NPROC` for process count, `RLIMIT_NOFILE` for descriptors, and a hard wall-clock timeout enforced from *outside* the sandbox so a wedged process cannot block its own kill. Watch for the classic traps — `/dev/zero` reads, decompression bombs, symlink races.

**Least privilege.** Run the sandboxed process as an unprivileged, non-root user with a dropped capability set. Drop `CAP_SYS_ADMIN`, `CAP_NET_RAW`, and everything else not strictly required. Enable seccomp to allowlist only the syscalls the runtime actually needs — with gVisor, this happens by construction.

**Isolation hygiene.** One evaluation per sandbox. Never reuse an environment across candidates. Randomize sandbox IDs. Rotate the base image on a schedule and pin it by digest so a compromised registry tag cannot silently change what you run.

**Observability and kill switches.** Log every sandbox lifecycle event — provision, limit hit, teardown. Alert on anomalies: a candidate that spikes CPU for the full timeout, a burst of metadata-endpoint probes, a sudden jump in network attempts. Maintain a circuit breaker that disables evaluation if the escape rate crosses a threshold, and require human review before re-enabling.

The goal is not to make escape impossible — it is to make escape *expensive, noisy, and contained*. gVisor and Firecracker give you the boundary. cgroups and seccomp give you the constraints. Logging and alerting give you the tripwires. Layer all three.

—

## Metrics: Tracking Functional Correctness and Linter Pass Rate in CI

An evaluator is only as good as the signals it surfaces in your pipeline. Two metrics carry the most weight, and both belong on the same dashboard.

### Functional Correctness

**Functional Correctness** is the test-pass rate aggregated across your evaluation set — optionally weighted by case severity. Report it multiple ways, because a single number hides the distribution:

– **Mean pass rate** across tasks — the headline number.
– **Pass@k** — the fraction of tasks solved at least once in `k` attempts. Critical for agentic workflows where you sample multiple candidates and pick the best.
– **Median and p10** — to expose tasks where the model is consistently failing. A mean of 0.8 built from all-1.0s and a bimodal tail of 0.2s is a very different system from a uniform 0.8.
– **Segmented by task class** — algorithms, data transforms, API integration, SQL, concurrency. Aggregate numbers hide class-specific regressions.

Gate your CI on it. A pull request that changes a prompt template, swaps a model, or touches the agent loop must not merge if Functional Correctness drops by more than an agreed epsilon. Treat it exactly like test coverage: a metric that fails the build when it regresses.

### Linter Pass Rate

**Functional Correctness** answers “does it work.” **Linter Pass Rate** answers “is it shippable.” A function can pass every test and still be unacceptable — unused imports, shadowed variables, unhandled exceptions, missing type annotations, unsafe patterns flagged by `bandit` or `semgrep`.

Track linter pass rate as a first-class metric alongside correctness, reporting both the raw pass rate and the distribution of violation categories. Two reasons this matters:

1. **It catches sloppiness that tests miss.** A test exercises the happy path; a linter sees the `bare except:` sitting next to it.
2. **It correlates with downstream review cost.** Code that passes a linter reduces human review time, which is the real bottleneck in any agent-assisted workflow.

Run your standard stack — `ruff` or `flake8` for Python, `eslint` for JavaScript/TypeScript, `clippy` for Rust, plus a security scanner — inside the same sandbox, and record results in the same evaluation artifact. Combine the two into a single gated score, for example `0.7 * correctness + 0.3 * linter_pass_rate`, and version that formula alongside your eval suite so changes are reviewable.

### Instrumentation and reporting

Emit one structured record per evaluation — task ID, model/version, sandbox ID, test outcomes, lint outcomes, latency, resource peaks, timestamps. Store it durably and visualize trends over time. Without historical tracking you cannot tell whether a prompt change helped or whether the eval set drifted. The dashboard should answer one question at a glance: *is the system getting better or worse, for which task classes, at what cost?*

—

## Roadmap: A 6–10 Week Guide to Autonomous Coding Agents

This is the sequence we follow with clients. It front-loads the evaluator deliberately — you cannot safely automate what you cannot measure.

**Weeks 1–2 — Foundation and harness.** Stand up the sandbox infrastructure. Choose gVisor for the inner loop, Firecracker for the outer boundary. Build the five-phase lifecycle, wire in cgroups/seccomp limits, and ship a single-language, single-task evaluator end to end. Success criterion: a candidate program runs, gets scored on unit tests, and the sandbox is destroyed cleanly with zero residue.

**Weeks 3–4 — The test corpus and the linter gate.** Build the three-tier test suite (canonical, edge, property-based) for your target task class. Add the holdout split and input randomization. Integrate linters and security scanners into the sandbox. Establish baseline Functional Correctness and Linter Pass Rate for your current model and prompt. Success criterion: a reproducible, versioned eval suite with a baseline dashboard.

**Weeks 5–6 — CI integration and gating.** Wire the evaluator into CI as a required check. Define regression thresholds and fail the build on them. Add structured evaluation records and trend visualization. Run a friction sprint to cut eval latency — parallelism, image caching, warm sandbox pools. Success criterion: every prompt/model change is scored automatically and blocked on regression.

**Weeks 7–8 — The agent loop.** Now wrap the model in a plan/act/reflect loop with the evaluator as its reward signal. Add retry-on-test-failure with structured feedback (the failing test’s name and assertion, not the whole trace), and best-of-k selection ranked by score. Introduce cost ceilings and human checkpoints for high-risk changes. Success criterion: the agent self-corrects on failed tests and measurably outperforms single-shot generation.

**Weeks 9–10 — Hardening and autonomy.** Red-team your own sandbox: attempt escapes deliberately, probe the resource limits, and verify your tripwires fire. Expand the task corpus to the classes that matter for your domain. Turn on autonomous operation for low-risk task classes with the evaluator as the gate, keeping humans in the loop for the rest. Publish the metrics and the operating envelope. Success criterion: an agent that ships code you would merge, with an evaluation trail you can audit.

The through-line across all ten weeks is the same: **behavior is the ground truth, and the sandbox is how you observe it.** Lexical metrics gave you a ruler marked in the wrong units. Functional Correctness and Linter Pass Rate, measured inside an ephemeral, kernel-isolated sandbox, give you one that actually predicts whether the code is worth shipping.

—

## Build This With OnlineCloud

OnlineCloud is a US-based Agentic AI & MCP engineering studio. We design the sandbox evaluators, agent loops, and observability that turn code-generation prototypes into systems you can trust in production — and we ship them in 6–10 weeks, not quarters. Every engagement gets senior engineers, an eval suite that defines “done” before a line of agent code is written, and an architecture built to be audited end to end.

**Ready to build a code-generation agent that you can actually measure?** [Book a free consultation](https://www.onlinecloud.dev/) — we’ll map your task corpus, pick your isolation strategy, and define the eval that proves the thing works. Or reach us directly at **contact@onlinecloud.dev**; US-based support, response within 24 hours.

*OnlineCloud · Xlent Info tech. LLC · [www.onlinecloud.dev](https://www.onlinecloud.dev/)*

# Securing Agentic IAM: Implementing Least-Privilege at the Protocol Layer


*By the security engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: The Vulnerability of Shared Service Accounts

Every agentic deployment I audit has the same origin story. A team wants an agent to read tickets, draft replies, and update a CRM. The fastest path to a demo is a service account: one long-lived credential, provisioned once, stuffed into an environment variable, and handed to the agent. It works. It demos beautifully. And it quietly collapses the entire distinction between *who the user is* and *what the system can do*.

A shared service account is a single identity that represents nobody. When the agent executes `update_crm(account_id, note)`, the downstream system sees a service principal with write access to the whole table — not the support representative who asked for the change, not the entitlement scope of the requesting tenant, not the disposition of the customer whose record is being touched. The service account is a master key with a friendly name, and in an agentic system, a master key is held by a stochastic, prompt-injectable principal that reads untrusted content and then decides what to call next.

This is the crux. In a traditional backend, the service account is invoked by code whose control flow an engineer wrote and reviewed. In an agent, the “control flow” is generated at runtime from a conversation that may contain a poisoned document, a malicious tool result, or a user who simply words a request differently than the prompt’s author imagined. The model does not need to be adversarial for the blast radius to be catastrophic; it only needs to be *steered*. Indirect prompt injection turns a document into an instruction, and a shared credential turns that instruction into an authorized action.

The failure modes compound in three ways. **Attribution loss:** the audit trail points at `svc-agent-prod`, which tells an investigator nothing about which human, which task, or which approval is accountable. **Privilege monotonicity:** service accounts accumulate permissions over months and are almost never reduced, so the credential’s authority only grows while the agent’s actual needs stay fixed. **No revocation granularity:** you cannot revoke one user’s session without breaking the agent for everyone, so a suspected compromise becomes an all-or-nothing shutdown.

Least privilege at the protocol layer is the corrective. It means that no credential the agent holds is broader than the task in front of it, that every request carries an identity traceable to a human, that authorization is decided per tool call by policy rather than by connection, and that every decision leaves a replayable record. The Model Context Protocol (MCP) gives us the right substrate — it is the contract layer between the LLM and your business logic — but the security guarantees must be engineered into that layer. The next four sections build them, and the last assembles them into a reference architecture.

## 2. OAuth 2.1 for Agents: Implementing Scoped Tokens and Audience Enforcement

The first fix is to stop treating authentication as something that happens once at connect time. For agent workloads I standardize on **OAuth 2.1 with the Authorization Code flow and PKCE**, with a genuinely short access-token TTL and single-use, rotating refresh tokens. The details of the code exchange are well understood; the part that determines whether your agent is actually contained is what the token *contains* and what it is *valid for*.

Scopes are where least privilege is won or lost. The cardinal sin is a coarse scope — `data:all`, `resources:read_write`, `crm:*`. A scope should read like the smallest sentence describing the smallest useful action: not `orders`, but `orders:read:own`; not `refunds`, but `refunds:create:under_500`. Design scopes as the Cartesian product of (operation, resource class) and mint one per pair the agent genuinely needs. If a tool only reads a customer’s order history, the token must not carry a write scope “just in case,” because “just in case” is the blast radius an injected instruction will happily exercise.

Two mechanisms turn scopes from a label into an enforced boundary.

**Scope-to-tool mapping at the MCP server.** Each tool declares the scopes its invocation requires, and the server rejects any call whose token lacks them *before* the handler executes. This is a declarative allowlist enforced in exactly one place, so adding a tool forces an explicit authorization decision rather than an implicit inheritance:

“`json
{
“tools”: {
“get_order_status”: {
“required_scopes”: [“orders:read:own”, “shipments:read:own”],
“audience”: “mcp://ledger”
},
“create_refund”: {
“required_scopes”: [“refunds:create:under_500”],
“human_checkpoint”: “refund > $500”
}
}
}
“`

**Audience enforcement.** Scope answers “may this caller invoke this tool?” Audience answers “was this token *meant* for me?” The `aud` claim must equal the specific MCP server’s identifier, and the server must validate it on every request alongside `iss`, `exp`, and signature. Without audience binding, a token minted for a low-risk analytics server is a valid credential at your ledger server — a confused-deputy attack that token reuse makes trivial. Enforce one audience per server, reject `*`, and require the caller to perform **RFC 8693 token exchange** to obtain a server-specific token rather than reusing a broad one.

The token flow I recommend for agent tasks is explicit delegation, not impersonation. The orchestrator exchanges the user’s login token for a **task-scoped delegation token** with narrower scopes, a TTL measured in minutes, an explicit `aud` of the single MCP server, and a `delegation` claim chain naming the initiating user and each agent hop. The model never holds the user’s token; it holds a perishable proxy that expires when the task does. Every request is authenticated per call — a session authorized at 09:00 is not authorized after a revocation at 09:01 — and validation order is fixed: signature, issuer, audience, expiry, then scope against the invoked tool. Any failure returns a structured error *and* emits an audit event, which is what turns authentication from a gate into evidence.

## 3. Identity Propagation: How to Flow Human Identity Through Multi-Step Agent Chains

Scope grants capability; identity governs rows. A token that says `orders:read:own` is only meaningful if the data layer knows *whose* “own” is being read. That requires propagating the human identity — intact and verifiable — through every hop of a multi-step agent chain, and terminating it at the database so that row-level security can do its job.

The naive approach, passing a `user_id` as a tool parameter, is not propagation; it is a suggestion the model can rewrite. Identity must travel as **signed, verifiable claims inside the credential**, not as mutable arguments. Four rules make propagation trustworthy.

**Rule one: identity is carried, not re-derived.** Each hop receives the delegation token and its claim chain, never a bare identifier. A sub-agent acting for a user inherits the user’s identity as a claim it cannot forge, and the chain records the hop that introduced it.

**Rule two: delegation narrows, never widens.** A sub-agent’s token must be a subset of its parent’s scopes and TTL. This is monotonic privilege reduction: authority can only shrink as the chain lengthens, so a compromised, over-eager leaf agent’s ceiling is the leaf’s grant, not the root user’s full entitlement. Enforce this at the token-exchange endpoint — refuse any exchange that requests a scope the presenter does not already hold.

**Rule three: the principal is resolved into a database security context.** The MCP server, holding a verified identity, opens the database connection under a role or session variable that the engine enforces. Oracle VPD, MS SQL row-level security, or Postgres RLS then filters every query to the principal’s permitted rows automatically. This is the point where protocol-level authorization becomes data-level authorization, and where a prompt-injected instruction to “read all orders” returns an empty result because the rows simply are not visible to that session.

**Rule four: the chain is auditable end to end.** Every event records `initiating_user` and the full `delegation_chain`, so an investigator can walk backward from a mutated row to the human who set the task in motion. Combined with short TTLs, this gives you what service accounts never could: the ability to answer *who authorized this* without guessing, and to revoke a single user’s authority without touching anyone else’s session.

## 4. Policy Enforcement: Implementing OPA for Tool-Level RBAC

Identity tells you who is acting; scopes tell you the outer envelope of what is possible. Policy is the decision layer that evaluates the *specific* request against the *current* context — and it belongs outside the agent, not inside it. I standardize on **Open Policy Agent (OPA)** as a sidecar to the MCP server, expressing tool-level RBAC as Rego that the server consults before every invocation.

A representative policy evaluates three inputs: the token’s claims (subject, scopes, audience, delegation chain), the requested action (tool name and arguments), and environmental facts (tenant, resource ownership, time, approval state). The output is a decision plus obligations — the conditions the server must satisfy to comply:

“`rego
package mcp.authz

default allow := false

# Refunds require the right scope AND tenant match AND approval above threshold.
allow if {
input.tool == “create_refund”
“refunds:create:under_500” in input.token.scopes
input.arguments.amount <= 500 input.resource.tenant == input.token.tenant } allow if { input.tool == "create_refund" input.arguments.amount > 500
input.approval.granted == true
input.approval.approver != input.token.subject # four-eyes
}
“`

The power of centralizing this in OPA is that policy becomes reviewable, testable, and version-controlled as code — decoupled from tool implementation, so a security architect can tighten a rule without shipping a server release. Two properties matter most for agents.

**Deny by default.** An unrecognized tool, an unknown scope, or a missing input field must fail closed. Agents explore; a permissive default rewards exploration with access.

**Obligations, not just booleans.** A policy can return conditional allow — *permit this refund only if a human checkpoint approves* — which the MCP server enforces as a hard gate before execution. This is how human-in-the-loop control moves from a prompt instruction (“ask before refunding”) into an enforced protocol step the model cannot bypass.

Run OPA co-located with the server, feed it decision inputs, and log every decision with its inputs and the policy revision hash. That last detail is what makes section five possible.

## 5. Audit: Building a Replayable Decision Log for Forensic Security Review

An auditor does not ask what your agent intended. They ask what it did, under whose authority, and whether the record can be trusted. Chat transcripts fail all three tests: they are incomplete, unverifiable, and not reconcilable to the transactions they caused. The corrective is to treat agent runs as **transactions to be journaled**, emitting a structured, versioned, append-only stream of decision events.

The envelope for each event carries a unique `event_id`, a `run_id`, a `parent_event_id`, an `event_type` from a closed vocabulary, a UTC timestamp with sub-second precision, a monotonic `sequence`, and the acting principal with its full delegation chain. Because agent execution is a tree — planning, branching, tool calls, retries, approvals, aborts — a parent-child model preserves the structure that flat logs destroy, letting an investigator walk from the initiating request down to the row that changed.

**Replayability** is the property that elevates this from logging to forensics. To replay a decision you must be able to reconstruct why it was allowed, which means recording not just the outcome but the full decision context: the token claims presented, the tool and its arguments, the resource touched, the OPA policy revision hash, and the obligations satisfied. With those four inputs — claims, action, context, policy revision — you can re-evaluate any historical decision against the policy that was actually live and confirm it should have been allowed, or prove it should not have been.

Three storage disciplines make the log evidentiary. **Immutability:** write events to append-only, WORM-style storage with hash chaining so any edit or reordering is detectable. **Schema-first completeness:** if the schema is defined before the agent ships and non-conforming events cannot be written, completeness becomes a property of the system rather than a matter of developer diligence. **Reconciliation:** every event carries a stable identifier linking an utterance to the database transaction it caused, so an anomalous write traces back to the decision behind it.

The payoff is operational, not just compliance. When an incident responder asks “which runs touched account 4421 last week, and who authorized them?”, a replayable log answers in seconds with verifiable evidence — and the same log is what satisfies your SOC 2 or internal-audit assessor.

## 6. Reference Architecture: A Secure, VPC-Isolated MCP Auth Flow

Binding all of this together, here is the reference topology I deploy for regulated workloads. The flow, end to end:

1. **User → Agent Host.** The user authenticates to the agent host via OAuth 2.1 Authorization Code with PKCE. No implicit grant, no static secret.
2. **Agent Host → Authorization Server (private subnet).** The host requests a task-scoped delegation token via RFC 8693 exchange: narrow scopes, minutes-long TTL, `aud` set to the specific MCP server, and a delegation chain seeded with the user’s identity. All of this happens inside the VPC; the authorization server is never internet-facing.
3. **Agent → MCP Server (private subnet).** Every tool call presents the delegation token over mTLS between the agent runtime and the server. The server validates signature, issuer, audience, expiry, and scope **per request** — never per connection.
4. **MCP Server → OPA sidecar.** The server sends the decision input (claims, tool, arguments, resource, context) to OPA, which returns an allow/deny plus obligations. Deny by default; obligations are enforced as hard gates.
5. **MCP Server → Data Layer.** Holding a verified identity, the server opens a database session under the principal’s security context, so VPD/RLS filters every query to permitted rows. Credentials to the data layer live in a secrets manager and are never exposed to the model.
6. **Every step → Decision Log (write-only).** Authentication results, policy decisions with revision hashes, tool invocations, approvals, and data mutations stream to an append-only, hash-chained store. No path around the log; an event that cannot be written blocks the action.

The network controls matter as much as the identity controls. The MCP server, OPA, the authorization server, and the data layer sit in private subnets with no inbound internet route; only the agent host’s ingress gateway is reachable, through a WAF, and egress from the MCP tier is allowlisted to the data layer. Secrets never leave the secrets manager. The model’s only reachable surface is the MCP tool API — it has no network path to the database, so a successful prompt injection can only request tools that OPA and row-level security will still evaluate.

The net effect: a compromised or steered agent can at worst invoke the narrow, in-scope tools its task token permits, against only the rows its propagating identity is allowed to see, with every decision logged and replayable. That is least privilege expressed at the protocol layer — not as a policy document, but as enforced, auditable mechanism.

—

## Build agentic IAM you can defend in an audit

If your agents are running on shared service accounts, the containment problem is not theoretical — it is a matter of when. OnlineCloud designs and ships MCP-native systems with scoped delegation, OPA policy enforcement, row-level security, and replayable decision logs, deployed inside your VPC and audited end to end. US-based senior engineers, production in 6–10 weeks.

**[Talk to our security architects →](https://www.onlinecloud.dev/)** — free initial consultation, response within 24 hours.

# Optimizing Tool-Calling for Large Toolsets: Discovery & Filtering Patterns


*An infrastructure architect’s field guide to keeping MCP servers fast, cheap, and secure as the tool count grows from ten to a thousand.*

—

## 1. Introduction: The Tool-Inflation Problem

Every MCP deployment starts the same way. A team wires ten tools into a server — `search_orders`, `get_invoice`, `create_ticket` — and the agent feels magical. Six months later the same server exposes forty tools across three business domains, and the agent has quietly gotten dumber.

This is **tool inflation**, and it is a structural problem, not a prompt-tuning problem. When you serialize every tool definition into the model’s context on every turn, three things degrade simultaneously:

– **Context pressure.** Each tool costs 80–400 tokens once you include its name, description, JSON Schema, parameter docs, and examples. Forty tools is 8–16K tokens of *baseline* tax before the user has said a single word. At 200 tools you have spent the entire context window on capabilities the request will never use.
– **Selection accuracy.** Models do not scan a flat list of 200 tools the way a human scans a menu. Research on tool-selection benchmarks consistently shows accuracy decaying as the candidate set grows — the model starts confusing `delete_record` with `archive_record`, or hallucinating parameters from a neighbouring schema. More options means more ways to be *almost* right.
– **Operational drag.** Every tool in the global namespace is a tool that must be versioned, authorized, rate-limited, and audited. A monolithic toolset couples unrelated domains (billing, HR, infrastructure) into a single blast radius.

The old workaround — “just fine-tune the model on your tools” — does not scale either. Tools are the most volatile part of any system; they change weekly. You cannot retrain on every schema bump.

The durable fix is architectural. Instead of presenting the *entire* toolset, present a **small, intent-relevant slice** of it, chosen at request time. That single idea decomposes into three disciplines: **semantic discovery**, **dynamic filtering**, and **cached dispatch**. Together they turn a 200-tool server into something the model sees as a 6-tool server — while retaining access to all 200.

This post walks through the pattern end to end, including a production Go reference implementation.

—

## 2. Semantic Tool Discovery: Two-Stage Retrieval for Tool Registration

The insight borrowed from information retrieval is simple: **do not stuff the catalog into the prompt; build an index over it.** Tool descriptions are natural language, so the same embedding-based retrieval that powers RAG over documents works over your tool registry.

But naive single-stage retrieval is not enough for toolsets, because tools have both *semantic* descriptions and *structural* constraints. The pattern that holds up in production is **two-stage retrieval**.

**Stage 1 — Recall (vector search).** At registration time, each tool is embedded from a normalized document: name (split on snake_case), description, parameter names, and the domain/capability tags attached by the author. These vectors live in an index (pgvector, Qdrant, or an in-process HNSW). At request time the user’s intent — or a compact rewrite of it — is embedded and used to pull the top-K candidates, typically K = 20–30. Embedding models are cheap and fast; this stage is about *recall*, not precision, so the threshold is deliberately loose.

**Stage 2 — Precision (rerank + structural filter).** The K candidates are reranked by a cross-encoder or a lightweight LLM call that reads the actual request and the candidate schemas. This is where near-misses are eliminated. Crucially, Stage 2 *also* applies non-semantic gates: tenant scopes, feature flags, and RBAC (see §5). The output is the final tool slice, usually 4–8 tools.

Why two stages and not one? Because a single embedding model conflates *similarity* with *suitability*. “Export all customer records” and “delete all customer records” embed nearly identically but have wildly different policy implications. The reranker sees the whole request, not just the query vector, and can separate them.

A subtle but high-leverage detail: **register tools with the vocabulary your users actually type.** If finance asks “who owes us money,” the tool `ar_aging_report` will never be retrieved unless its registration document includes synonyms like “outstanding invoices,” “receivables,” and “who owes.” Treat the tool description as a *search document*, not a docstring.

“`text
Stage 1: intent → embed → ANN search → top-30 tool IDs
Stage 2: (request, top-30 schemas) → rerank + RBAC filter → 6 tool schemas
Stage 3: 6 schemas → model context → tool call
“`

The registry itself should be **append-only and versioned**. Never mutate a tool document in place; publish a new version and let the index pick it up on the next refresh. This gives you rollback, A/B comparison of descriptions, and a clean audit trail of *what descriptions the model was actually shown* when it made a bad call — the single most useful debugging artifact in an agent system.

—

## 3. Dynamic Filtering: Pruning the Toolset by Intent and State

Semantic relevance answers *what the request is about*. It does not answer *what the session is allowed to do right now*. Dynamic filtering is the second axis: prune by **session state**, not just by topic.

Three filter classes do most of the work:

**Intent-class gating.** A cheap upstream classifier (or a router LLM, or even a keyword/regex fast path) tags the turn as *read*, *write*, *admin*, or *transactional*. Read-only turns get read-only tools. This is not merely a context optimization — it is a safety primitive. The model literally cannot call `wire_transfer` during a question-answering turn because that schema was never in its context.

**Stateful workflow filtering.** Agents execute workflows, and workflows are state machines. After `create_order`, the relevant tools are `add_line_item`, `apply_discount`, `submit_order` — not the other 197. Keep a per-conversation `allowed_tools` set that the orchestrator transitions as the workflow advances. This collapses the candidate space dramatically and eliminates an entire class of “the agent skipped a step” bugs.

**Capability and environment filtering.** Feature flags, model tier, deployment region, and cost ceilings all prune the set. A tool backed by an expensive GPU job should not appear when the session is running under a $0.05/request cost budget.

The result is a **layered predicate** over your registry:

“`text
visible = INDEX(intent)
∩ ALLOWED(session_state)
∩ PERMITTED(principal, RBAC)
∩ ENABLED(flags, region, budget)
“`

Two implementation rules matter more than the design itself. First, **filter before you serialize**, never after — the expensive part is tokens in context, and post-hoc instruction to “ignore tool X” does not remove it from the window. Second, **make the filter deterministic and loggable.** When the agent fails to find a tool it “should” have had, you want to diff the filter input (state, principal, flags) against the expected set, not guess.

A practical guardrail: cap the visible set at a hard ceiling (we use 12). If the layered predicate returns more, force the reranker to trim further. A stable, small tool menu is worth more than marginally better recall.

—

## 4. Performance: Latency in the Tool-Call Hot Path

Here is the trap. You added an embedding call and a reranker to the critical path between the user’s message and the model’s first token. If you are not careful, you have traded a context-cost problem for a latency problem.

The budget matters. For interactive agents, retrieval must fit in a **single-digit-millisecond to low-double-digit-millisecond** window, because the model’s own inference time (often 300–2000 ms) dwarfs everything else and the user perceives the sum.

Five techniques keep semantic search off the latency floor:

**Embed the intent once, reuse it.** Cache the query embedding keyed by a normalized-intent hash. Turns within a workflow often share intent; you should not re-embed “add another item” five times.

**Do ANN in-process, not over the network.** An HNSW index for 200–5,000 tools fits comfortably in memory (tens of MB). A network hop to a vector database adds 3–15 ms of pure RTT per call. For tool counts in the thousands, an in-process index with periodic refresh beats a managed vector DB on the hot path.

**Warm the index at startup.** Deserialize and build the HNSW graph before the server reports ready. Never build it lazily on the first request.

**Make the reranker optional and bounded.** A cross-encoder reranking 30 candidates is ~2–8 ms on CPU; an LLM reranker is 100 ms+. Default to a small cross-encoder, and fall back to pure vector ranking if the reranker’s p99 exceeds your budget. Degrade, don’t stall.

**Cache the whole slice.** The single biggest win: most requests in a session map to the same `(intent_class, session_state, principal)` tuple → the same tool slice. Cache the *resolved slice* keyed by that tuple and invalidate on state transition or registry version change. A cache hit is a map lookup — sub-microsecond — and it removes retrieval from the hot path entirely.

The mental model: **retrieval is an optimization pass, and optimizations have caches.** Instrument it like one. Track `slice_cache_hit_ratio`, `ann_p50/p99`, and `rerank_skipped_ratio`. If your cache hit ratio is under ~70% in a workflow-heavy agent, your state keying is too coarse or too fine, and that metric will tell you which.

—

## 5. Security: Tool-Level RBAC in Dynamic Discovery

Dynamic discovery introduces a security anti-pattern that is easy to miss: **the retrieval layer becomes a confused deputy.** The index knows about every tool. If the retrieval path does not carry the principal’s identity and enforce policy, a cleverly worded request can surface a tool the user was never meant to see — and once it is in context, the model may call it.

Four rules keep discovery safe:

**Enforce RBAC before serialization, not at execution only.** Execution-time authorization is necessary but insufficient. If an unauthorized schema enters the context, you have already leaked the existence, shape, and semantics of an internal capability. Filter the index query by principal scope as a *first-class* predicate, ideally at the vector-store level (metadata filter on tenant/role) so unauthorized tools never even become candidates.

**Treat the tool index as an authorization surface.** A vector index partitioned by tenant is not just a performance optimization; it is a security boundary. Cross-tenant bleed in a shared ANN index is a real risk — always filter by tenant metadata, and verify with a false-positive test (attempt to retrieve a foreign tool by name).

**Re-authorize at invocation.** Discovery decides *what the model sees*; the dispatcher decides *what actually runs*. Every call re-checks the principal against the tool’s policy at execution time, so a stale cache or a smuggled tool name cannot bypass policy. This is the “verify at the edge” principle — never trust the filter alone.

**Audit the slice, not just the call.** Because the slice is deterministic and logged, you can answer the question auditors actually ask: *what tools was this principal shown, and could they have invoked each one?* Keep the resolved slice ID on every trace span. This turns a fuzzy “the model did something odd” investigation into a precise replay.

The composed posture: **RBAC filters the index (prevention), the dispatcher re-checks (enforcement), and the slice log records (evidence).** Three independent controls; a failure in any one does not silently become a breach.

One more: prompt injection through tool *descriptions*. If tool metadata is authored or imported from semi-trusted sources, sanitize it before it enters the registration document. A description is model-readable text — treat it as untrusted input, because that is exactly what it is.

—

## 6. Reference Implementation: A Go MCP Tool Dispatcher with Caching

The following is a compact but production-shaped sketch of the pattern. It shows a registry, an in-memory vector-ish index (interface-based, so you can drop in HNSW or pgvector), RBAC filtering before serialization, and a slice cache keyed by the security-and-state tuple. It is deliberately dependency-light so the architecture is legible.

“`go
package dispatcher

import (
“context”
“crypto/sha256”
“encoding/binary”
“errors”
“math”
“sync”
“time”
)

// Tool is a registered MCP capability.
type Tool struct {
Name string
Description string
Tags []string
Schema []byte // raw JSON Schema, serialized to the model
Required []string // required RBAC scopes
Embedding []float32
}

// Principal is the authenticated caller.
type Principal struct {
ID string
Scopes map[string]bool
Tenant string
}

func (p Principal) Has(scope string) bool { return p.Scopes[scope] }

// Index is the pluggable similarity backend (HNSW, pgvector, brute force).
type Index interface {
Search(ctx context.Context, query []float32, k int, tenant string) ([]string, error)
}

// Registry holds the append-only, versioned catalog.
type Registry struct {
mu sync.RWMutex
version string
tools map[string]Tool
index Index
embed func(context.Context, string) ([]float32, error)
}

// Filter is one layer of the dynamic predicate.
type Filter func(Tool) bool

// maxVisible is the hard ceiling on tools serialized into the model context.
const maxVisible = 12

// Slice is the resolved, cacheable output of discovery.
type Slice struct {
Version string
Tools []Tool
}

// keyTuple is what the resolved slice depends on. If any of these change,
// the slice must be recomputed — nothing else invalidates it.
type keyTuple struct {
Intents uint64
State uint64
Principal string
Tenant string
Version string
}

// Dispatcher is the hot path: discovery -> filter -> cache -> serialize.
type Dispatcher struct {
reg *Registry
cache sync.Map // keyTuple -> *Slice
}

// Resolve returns the tool slice to place in the model’s context.
func (d *Dispatcher) Resolve(
ctx context.Context,
p Principal,
intent string,
stateHash uint64,
layerFilters []Filter,
) (*Slice, error) {
// 1. Normalize intent and derive the cache key.
q, err := d.reg.embed(ctx, intent)
if err != nil {
return nil, err
}
key := keyTuple{
Intents: hashFloats(q),
State: stateHash,
Principal: p.ID,
Tenant: p.Tenant,
Version: d.reg.version,
}
if v, ok := d.cache.Load(key); ok {
return v.(*Slice), nil // cache hit: sub-microsecond
}

// 2. Stage 1 — recall via ANN, tenant-partitioned at the index layer.
ids, err := d.reg.index.Search(ctx, q, 30, p.Tenant)
if err != nil {
return nil, err
}

// 3. Stage 2 — RBAC + dynamic filters applied BEFORE serialization.
// Unauthorized tools must never reach the model context.
candidates := make([]Tool, 0, len(ids))
for _, id := range ids {
t, ok := d.reg.lookup(id)
if !ok {
continue
}
if !authorized(p, t) {
continue // prevention: filter the index result, don’t just block at call time
}
if !all(layerFilters, t) {
continue
}
candidates = append(candidates, t)
}

// 4. Enforce the hard ceiling on visible tools.
if len(candidates) > maxVisible {
candidates = candidates[:maxVisible]
}

slice := &Slice{Version: d.reg.version, Tools: candidates}
d.cache.Store(key, slice)
return slice, nil
}

// Invoke re-authorizes at execution time. Discovery decides what the model
// sees; the dispatcher decides what actually runs. Never trust the filter alone.
func (d *Dispatcher) Invoke(
ctx context.Context,
p Principal,
name string,
args []byte,
) ([]byte, error) {
t, ok := d.reg.lookup(name)
if !ok {
return nil, errors.New(“unknown tool”)
}
if !authorized(p, t) {
return nil, errors.New(“forbidden: principal lacks required scopes”)
}
return d.call(ctx, t, args) // rate limits, audit span, execution
}

func authorized(p Principal, t Tool) bool {
for _, s := range t.Required {
if !p.Has(s) {
return false
}
}
return true
}

func all(filters []Filter, t Tool) bool {
for _, f := range filters {
if !f(t) {
return false
}
}
return true
}

func hashFloats(f []float32) uint64 {
h := sha256.New()
var b [4]byte
for _, x := range f {
binary.LittleEndian.PutUint32(b[:], math.Float32bits(x))
h.Write(b[:])
}
return binary.LittleEndian.Uint64(h.Sum(nil))
}
“`

Four things to notice in the code, because they encode the whole thesis:

1. **`keyTuple` defines invalidation.** The slice is a pure function of intent embedding, session state, principal, tenant, and registry version. Cache correctness falls out of the key — no manual invalidation logic to get wrong.
2. **RBAC runs inside `Resolve`, before the slice is built.** `authorized()` is a discovery-time gate, not just a call-time check. This is the “prevention” control.
3. **`Invoke` re-authorizes.** Even a cache hit or a crafted tool name cannot bypass policy. This is the “enforcement” control.
4. **The registry is versioned.** `Version` in the key means a tool-description update invalidates every cached slice atomically — and each `Slice` records the version it was resolved against, giving you evidence for audits and replays.

In a real deployment you would add a bounded LRU instead of a bare `sync.Map` (to cap memory), swap the `Index` for an HNSW implementation behind the same interface, and emit the `keyTuple` and slice ID onto the trace span. The five latency techniques from §4 are all refactors of `Resolve`, not rewrites — embed caching, in-process ANN, a bounded cross-encoder rerank between steps 2 and 3, and cache-hit telemetry layer in cleanly.

—

## The Through-Line

Tool inflation is not solved by better prompts or bigger context windows. It is solved by treating your tool registry like a **search index with a security policy attached** — retrieve narrowly, filter by intent *and* state *and* principal, cache the resolved slice, and re-authorize at the edge.

Do that, and a server with 200 tools behaves like one with six: lower latency, higher selection accuracy, a smaller attack surface, and an audit trail precise enough to explain every decision the agent made. Discovery and filtering are not optimizations you bolt on later. In any serious MCP deployment, they *are* the architecture.

—

**Building an MCP server and hitting the wall at 30 tools?** OnlineCloud designs retrieval-backed tool registries, RBAC-enforced discovery, and observable agent runtimes — wired into the databases and CRMs you already run. We ship evaluated, auditable systems in 6–10 weeks, with senior engineers only.

**[Start your MCP project →](https://www.onlinecloud.dev/contact)** · **[See our agent and MCP work](https://www.onlinecloud.dev/work)**

# Agentic Fraud Detection: Real-Time Transaction Scoring with Reasoning Swarms


*By the machine learning team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Why Black-Box Scoring Models Fail on Sophisticated Fraud

For a decade, the workhorse of transaction fraud detection has been the gradient-boosted classifier: a few hundred engineered features — velocity counts, merchant category, device fingerprint, geovelocity — shoved into XGBoost, thresholded at some precision-recall point, and shipped. It works, right up until the adversary reads your feature importance chart. A supervised scoring model is a *compression* of historical fraud into a static decision surface, and compressions are only valid on the distribution they were fit to. Sophisticated fraud does not stay on that distribution; it actively searches for the regions where the decision surface is wrong.

Consider what a modern fraud ring actually does. It **stages accounts** for weeks to build benign history, **dilutes** each participant’s volume below velocity thresholds, **mimics** legitimate purchase cadence with low-value orders, and **coordinates** across a graph of shared devices, addresses, and payout rails that no single transaction reveals. A per-transaction scorer sees one row and answers a question about that row. But the fraud is not *in* the row — it is in the **relationships between rows**, the temporal texture of the sequence, and the intent of the actors behind it.

Three failure modes recur in production, and they are architectural, not hyperparameter problems. **First, the feature/decision gap.** A model can only reason over features someone thought to engineer; the moment a ring innovates a new coordination pattern, there is no feature for it, and the model is blind by construction. **Second, label latency.** Chargebacks and confirmed-fraud labels arrive weeks later, so by the time the model learns a pattern the ring has rotated away — you train on last season’s adversary. **Third, opacity under scrutiny.** When a legitimate customer is declined, “the model said so” is not an answer. An analyst cannot interrogate a boosted tree’s 400-way interaction, so every false positive becomes a dispute the system cannot defend and a rule the team cannot improve.

This is the gap agentic reasoning closes. Instead of asking a model to *score* a row, we ask a **swarm of reasoning agents to investigate** it — to decompose the transaction into questions, gather evidence from tools and graph stores, test hypotheses about coordinated behavior, and produce a decision *with its reasoning attached*. The model stops being a black box and becomes a witness that can be cross-examined.

## 2. The Reasoning Swarm: Planners for Anomaly Detection, Workers for Link Analysis

The swarm is deliberately heterogeneous because anomaly detection and link analysis are different cognitive tasks that reward different agent designs.

**Planner agents** own the investigation. A planner treats the incoming transaction as a case, and its job is decomposition and adjudication rather than detection in the narrow sense. It reads a compact risk summary, decides *which* lines of inquiry are worth spending on, and dispatches them. Its reasoning is abductive: given an anomaly, what hypotheses would explain it? Is this a stolen card being tested with micro-authorizations, or a genuine customer traveling with an unfamiliar merchant mix? The planner generates candidate explanations, requests evidence for each, and revises as evidence arrives. Planners run an explicit **plan → act → reflect** loop: they declare which tools they will call and what they expect, execute through typed tool boundaries, then compare observed outcomes against expectations. A velocity check that returns *lower* than predicted is itself a signal — either the merchant is mis-tagged, or the ring is throttling.

Planners come in two tiers. A **triage planner** runs on every transaction within a tight latency and cost budget, deciding only whether a case is quiet enough to auto-clear or interesting enough to escalate. A **deep planner** runs only on escalated cases, where it can afford dozens of reasoning steps, richer context, and multi-source graph queries. This tiering is the core economic insight: you cannot afford swarm-level deliberation on every swipe, but you can afford it on the 1–3% that matter.

**Worker agents** do the heavy, parallelizable retrieval and analysis. *Anomaly workers* interrogate the transaction against behavioral baselines — device reuse, sub-threshold velocity, amount-distribution drift, time-of-day deviation — and return structured anomaly scores with explanations rather than a single scalar. *Link-analysis workers* crack coordinated fraud, because their subject is the **graph**, not the row. Each traverses a specific edge class — shared device fingerprints, shared ship-to addresses or IP subnets, shared payout accounts, overlapping session timing — answers a narrow question (“how many accounts share this fingerprint within 30 days?”), and returns the subgraph and its provenance.

No single worker sees the whole picture; that is by design. A link worker tracing payout rails does not need the customer’s full purchase history, and a velocity worker does not need card PII. This **context isolation** shrinks token cost and the sensitive-data blast radius, and it is what lets the swarm parallelize: anomaly workers fan out across statistical dimensions, link workers across graph edges, all reporting to the planner. When workers conflict — a device looks shared, but the timing fits a family member’s second account — the planner reasons over the contradiction instead of averaging it away. The decision emerges from *adjudicated disagreement*, which a single opaque score cannot express.

## 3. Integration: Bridging Real-Time Transaction Feeds to AI Models via MCP

A reasoning swarm is only as good as its grip on live data, and this is where most agentic fraud projects quietly fail. The mistake is prompt-stuffing: dump the transaction JSON into a chat completion and hope. It is brittle, untyped, and impossible to rate-limit. The correct contract layer is the **Model Context Protocol (MCP)** — typed tools, declarative resources, and auditable transports — sitting between the reasoning agents and the systems of record.

The ingestion path is a streaming spine. Transactions land on a log-bus (Kafka, Kinesis, or Pub/Sub), and an MCP server exposes that feed as **declarative resources** with a validated schema. Agents do not scrape a queue; they subscribe to a *typed* stream — `transaction.created`, `device.seen`, `chargeback.opened` — where the payload contract is enforced at the boundary. Behind the same server sit the tools the swarm reasons with: `get_transaction`, `get_device_graph`, `score_velocity`, `list_related_accounts`, `get_merchant_profile`. Each is a declared tool with a JSON schema and clean error semantics. When a planner hallucinates an argument — a device hash that does not exist, a negative amount — the schema rejects it before any business logic runs.

Three properties make this production-grade. **Data connectors with row-level policy** front the feature store, graph database, and payments ledger, so a link worker sees only the subgraph its task requires. **Streamable transports** (Streamable HTTP and SSE) let long-running graph traversals stream results back incrementally, so a deep planner begins reasoning before its slowest worker finishes. **Stateful correlation without shared secrets** threads a case ID through the whole investigation, binding every tool call from every worker to the transaction that triggered it — the substrate for the audit trail in Section 5.

The principle is blunt: **agents reason, MCP servers execute, humans audit.** Rate limits, auth boundaries, and structured errors live in the transport, so the swarm is constrained by the contract layer rather than by prompt instructions it might ignore. And because every call is typed and logged, a misrouted case at 3 a.m. can be replayed as an exact sequence of MCP invocations — not reconstructed from a transcript.

## 4. The Decision Framework: False Positives vs. Real-Time Latency

Fraud detection is a constrained optimization, not a classification problem. Every point of recall costs precision and latency, and in a payment loop you race a hard wall-clock budget measured in single-digit seconds. We treat the decision as two coupled gates: a **statistical gate** and a **reasoning gate**.

The statistical gate is fast and cheap — the boosted model still earns its place. It runs in milliseconds and produces a provisional risk score, but instead of thresholding that score directly, we map it into **three bands**. The low band auto-clears with no agentic spend. The high band auto-declines and files a report. The middle band — deliberately wide, and where the money hides — routes to the reasoning swarm, whose triage planner decides within budget whether to clear, decline, or escalate.

The tension is explicit and tunable. **Latency budget:** a card-present authorization cannot wait 20 seconds; a card-not-present checkout or a payout batch can. Each rail therefore carries its own deadline, and the planner is *deadline-aware* — it forfeits expensive deep reasoning and falls back to the statistical decision rather than blowing the SLA. The swarm must degrade gracefully under latency pressure, never silently exceed the budget and stall the checkout.

**Cost of error is asymmetric, so the threshold must be too.** A declined legitimate customer costs goodwill and conversion; an approved fraudulent payout costs the full amount plus chargeback fees and acquirer-relationship damage. The decision is a **cost-weighted expected-value comparison**: approve if `P(fraud) × loss < P(legit) × friction`, computed per case rather than at a global threshold. That is what the planner's reasoning delivers — a calibrated probability *plus* the evidence behind it, so the cost weighting applies to a defensible estimate rather than a raw logit. The loop closes on feedback: every decision is logged against its eventual outcome (chargeback, no-chargeback) so calibration can be measured and corrected. False-positive rate and recall are tracked per rail and per merchant segment, because a global metric hides the segments where the swarm is quietly wrong. The result is not a magic accuracy number; it is a **latency-budgeted, cost-weighted decision policy** an analyst can read, argue with, and retune. ## 5. Replayability: Auditing Agentic Fraud Decisions for Compliance Regulators do not ask whether your swarm is clever. They ask whether you can *prove* what it did, when, under whose authority, and why. For fraud the compliance surface is real: KYC/AML obligations, PCI DSS, SOC 2, and GDPR's right to explanation for automated decisions — plus adverse-action rules that demand a *reason* when a customer is declined. A black-box score cannot satisfy any of these. An agentic investigation can, **if it is built to be replayed.** Replayability is a design property, not a log file. Every decision is a **tree**: a case, a plan, a set of tool calls, each informed by prior results, some branches abandoned. The trail captures every hop as a structured, immutable event — planner identity and model version, worker identity, tool name, validated arguments, resource URIs touched, outcome, token and dollar cost, and the **parent event ID** linking the step to its plan. With parent-child linkage, an auditor walks from the triggering transaction down to the exact graph query that flagged a mule account, aborted branches included — proving the system *considered and rejected* the benign explanation is how you defend a decline. Two properties make the trail trustworthy. **Immutability:** the audit store is append-only, written with credentials the agents themselves cannot hold, into an independent sink that survives the very compromise it is designed to detect. **Completeness:** logging lives *in* the tool path through the MCP server, not bolted on around it — if a tool call can succeed without emitting an event, then at fraud's throughput unlogged calls will happen, and the one that matters will be the missing one. **Determinism-aware replay** completes the picture: because agents are stochastic, the trail records model version, sampling parameters, and tool returns, so a decision can be *reconstructed* even where it cannot be bit-for-bit reproduced. Hand an assessor that log and an adverse-action notice can cite the evidence chain directly. ## 6. Case Study: Reducing Fraud Loss by 30% with Multi-Agent Transaction Analysis We deployed this architecture for a mid-market payment processor running card-not-present checkout across a marketplace of several thousand merchants. The baseline was a supervised gradient-boosted scorer with a static threshold, retrained monthly, behind a rules engine. It was fast and reasonably precise on volume, but it was losing money exactly where a per-transaction model cannot see: **coordinated rings** and **account-takeover sequences** spread across many individually benign transactions. **The system.** A streaming MCP spine ingested authorizations from the processor's bus and exposed the feature store, a graph database of device/account/payout relationships, and the ledger as typed tools. Triage planners ran on every authorization within a strict latency budget; deep planners and link-analysis workers ran only on the middle band the statistical gate escalated. The scorer stayed in place as the fast gate — we did not rip it out, we *framed* it. **What changed.** Link-analysis workers surfaced shared-device and shared-payout topologies no single-row model could represent, and the planner's cross-examination of those subgraphs let analysts see *why* a cluster was flagged. After instrumentation revealed the middle band was far wider than necessary on low-risk merchant segments, the triage threshold was retuned per rail — cutting agentic spend without hurting recall. **The result.** Over the first two quarters, **fraud loss fell approximately 30%** against the prior-year baseline, driven mostly by earlier detection of coordinated rings and a materially lower false-positive rate in the escalated band — fewer legitimate customers declined, because a decline now requires an articulated evidence chain rather than a threshold crossing. Latency stayed inside every rail's SLA. And every decision shipped with a replayable trail, turning the processor's next audit from a scramble into a demonstration. ## The Takeaway Fraud is an adversarial reasoning problem, and you do not beat an adaptive adversary with a static decision surface. You beat it with a system that can investigate, explain, and be audited — planners that decompose, workers that traverse the graph, an MCP contract layer that executes and records every hop, and a decision policy that treats false positives and latency as first-class costs. Score the row; *reason* about the network behind it. --- ### Ready to build agentic fraud detection that can survive an audit? OnlineCloud is a US-based engineering studio specializing in agentic AI and Model Context Protocol — built end-to-end by senior engineers, no offshore handoffs. We design evaluated, observable reasoning swarms that plug into your streaming feeds, feature stores, graph databases, and ledgers, and we ship to production in 6–10 weeks, not quarters. **[Contact us for a free initial consultation →](mailto:contact@onlinecloud.dev)** — response within 24 hours. Or visit [onlinecloud.dev](https://www.onlinecloud.dev/) to see how we wire agents into the work you already do.

# Implementing Semantic Memory in AI Agents: Vector-Backed Long-Term Context


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: The Context-Window Amnesia Problem

Every multi-day agent deployment I have audited has the same failure mode, and it never shows up in a stack trace. The agent works beautifully on day one — it remembers the ticket, the customer, the constraints, the decisions it already made. Then the session ends, the process restarts, and on day two the agent greets the same user as a total stranger. It re-asks for information it was given yesterday. It re-proposes an approach that was explicitly rejected. It contradicts a decision that took an hour of back-and-forth to reach.

This is **context-window amnesia**, and it is the defining architectural weakness of long-horizon agentic systems. The context window is a cache, not a memory. It is scoped to a single session, it is volatile the moment the process exits, and it is fundamentally finite — 128K, 200K, a million tokens, the ceiling keeps moving but the constraint never disappears. An agent that runs a procurement workflow over three weeks, or a support escalation that spans four handoffs, or a research task that accumulates forty documents, cannot fit its working history into any window. Even when it technically could, stuffing a week of transcript into every prompt is the fastest way to burn budget and degrade reasoning quality through lost-in-the-middle dilution.

The industry’s default response has been *bigger windows*. That is a hardware answer to a data-architecture problem. The right answer is **memory as a first-class system**: durable, queryable, scoped to a user, and hydrated into the context window on demand. When memory is externalized into a vector-backed store, the agent stops treating “what happened before” as a scrolling transcript and starts treating it as a retrieval problem — because that is exactly what it is.

This post walks through how we design that system in production: a tiered memory architecture, the retrieval patterns that hydrate context at runtime, the multi-tenant isolation model that keeps user data from bleeding across boundaries, a working Python + pgvector implementation of a memory manager, and the ROI math that makes durable memory a cost-reduction program rather than a nice-to-have.

—

## 2. Architecture: A Tiered Memory System

The mistake most teams make is treating “memory” as a single thing — one vector store, one `remember()` function, one `recall()` at the top of every turn. That collapses three genuinely different access patterns into one, and it produces a system that is simultaneously too slow for hot-path context and too coarse for long-term recall.

We design memory in three tiers, each optimized for its own access pattern and retention horizon.

| Tier | Scope | Storage | Access latency | Retention | Typical content |
|—|—|—|—|—|—|
| **Buffer** | Current turn | In-process | Microseconds | Seconds | Latest message, tool output, scratchpad |
| **Short-Term** | Current session/task | Redis / in-memory KV | Single-digit ms | Hours to days | Rolling summary, active plan, recent turns |
| **Long-Term** | Cross-session, per-user | Postgres + pgvector | 10–50 ms | Indefinite | Facts, preferences, decisions, episodic events |

### The buffer: the working set

The buffer is the raw, unprocessed tail of the conversation — the current turn, the last tool result, the model’s scratchpad. It is deliberately ephemeral and must never be persisted to durable storage as-is. Nothing valuable survives here; its job is to give the model immediate continuity within a turn.

### The short-term tier: the session working memory

Short-term memory covers everything the agent needs to stay coherent *within* a task, without carrying a full transcript. Two mechanisms do the work:

– **A rolling summary**, regenerated incrementally. Rather than keeping every turn verbatim, the agent maintains a compressed narrative — decisions made, open questions, current state — refreshed every N turns or when the token budget for the session crosses a threshold.
– **Structured task state**, kept outside the prompt entirely: the active plan, step index, scratch variables, idempotency keys. This is state management, not text, and it should be read and written as data — an agent reading its own scratch state from a KV store is far more reliable than one trying to re-parse its own summary prose.

The short-term tier is also where **eviction policy** lives. When the session budget is exceeded, the agent summarizes, promotes anything durable to long-term memory, and drops the rest. This is the moment where an important fact either graduates to durable memory or is lost forever — get the promotion heuristic right and the agent’s memory compounds; get it wrong and it forgets exactly the things that matter.

### The long-term tier: durable, semantic, queryable

Long-term memory is where the vector database earns its place. Every durable memory is stored as a record with an embedding, so it can be retrieved by *meaning* rather than by timestamp or key. We generally store four categories of memory, each with its own embedding strategy and decay behavior:

– **Semantic facts** — stable truths about the user or domain (“uses Oracle EBS 12.2”, “prefers concise summaries”). Long-lived, rarely invalidated.
– **Episodic events** — what happened and when (“escalated ticket #4821 on March 3”). Queryable by recency and similarity.
– **Procedural memory** — how this user’s environment works (“their API requires a PATCH with a session header”). The highest-value category in enterprise deployments, because it is pure accumulated operational knowledge.
– **Preferences and constraints** — explicit guardrails the agent must never violate.

The critical design property is that long-term memory is **retrieved, not injected wholesale**. The context window receives only the handful of memories relevant to the current turn, which keeps prompts small and signals dense.

—

## 3. Retrieval Patterns: Hydrating Context at Runtime

A tiered store is only as good as the retrieval that feeds it into the prompt. The naive pattern — embed the user’s message, fetch the top-5, paste them in — is a starting point, not a design. Production semantic memory uses a richer set of patterns.

**Query construction is the first lever.** The raw user message is a noisy query. The reliable approach is to build the retrieval query from *three* inputs: the current message, the active task summary, and the last assistant action. This produces a query that reflects intent rather than surface wording. When the agent is mid-plan, the task summary often matters more than the literal turn.

**Hybrid retrieval, always.** Dense vectors are excellent at fuzzy semantic match and routinely terrible at exact identifiers — order numbers, part codes, policy IDs, account names. Fuse BM25 keyword search with vector similarity using reciprocal rank fusion so that a memory containing “ticket #4821” is retrievable both when the user says “#4821” and when they describe the underlying issue in different words.

**Recency-aware ranking, not pure similarity.** Long-term memory has a time dimension that generic vector search ignores. Two memories can be equally similar to the query, but the fresher one is usually more relevant — especially in episodic memory, where a preference stated last month should often outrank one from last year. We blend similarity with a recency decay to produce the final score:

“`
final_score = α · cosine_similarity(query, memory)
+ β · recency_decay(now – memory.created_at)
+ γ · importance(memory)
“`

Tune α/β/γ per memory category. Procedural memory leans on similarity; episodic memory leans on recency; constraints (preferences) are allowed to bypass ranking entirely and be injected whenever they match the domain.

**A hit threshold with a floor.** Never inject a memory just because it is the least-bad of the top-5. Apply a minimum score, and if nothing clears it, inject *nothing*. A wrong memory is worse than no memory — it actively misleads the model, and the model has no way to know the memory is stale.

**Bounded hydration.** Set a hard token budget for the memory block (we typically cap it at 15–25% of the total context) and enforce it at retrieval time, not in the prompt template. Sort by final score, append until the budget is exhausted, and stop. This is the single most effective control against the exact context bloat that memory systems are supposed to eliminate.

**Write-side discipline matters as much as read.** Deduplicate near-identical memories before insert (a similarity check against recent entries), version and supersede contradictory facts rather than accumulating them, and attach provenance to every memory so the agent can cite *why* it believes something. Memory that cannot be audited will eventually be trusted too much.

—

## 4. Privacy: User-Scoped Memory Isolation in Multi-Tenant Systems

Semantic memory is the most privacy-sensitive component in an agent stack. It stores exactly the things that must never leak: preferences, identities, past decisions, operational details. In a multi-tenant deployment, a single filtering bug is a data-breach event, not a quality regression.

The non-negotiable principle is **isolation enforced at the storage layer, not in the prompt**. Never retrieve broadly and then filter with application logic, and never rely on the model to “only use the current user’s memories.” Both approaches fail under prompt injection and under ordinary coding mistakes.

The patterns we implement:

**Tenant-scoped partitions or schemas.** Separate Postgres schemas per tenant, or at minimum a mandatory `tenant_id` column enforced by row-level security (RLS). With RLS, even a query that forgets its `WHERE` clause cannot return another tenant’s rows:

“`sql
ALTER TABLE agent_memories ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON agent_memories
USING (tenant_id = current_setting(‘app.current_tenant’)::uuid);
“`

Every connection sets `app.current_tenant` at session start, and the database — not the application — guarantees the boundary.

**User-scoped namespaces inside the tenant.** Memory is not just per-tenant; it is per-user within a tenant. Vector indexes should be partitioned so that a nearest-neighbor search physically cannot traverse another user’s embeddings. In pgvector, combine the RLS policy above with a `user_id` predicate pushed into the index scan, and consider partial indexes per tenant for large deployments.

**The tenant key must be bound to identity, not to input.** Derive `tenant_id` and `user_id` from the authenticated session — a signed token, an OAuth subject claim — never from a request parameter the caller can set. This is the same discipline as identity propagation in MCP tool calls: the boundary is established once, cryptographically, and carried through every downstream operation without re-derivation.

**Isolation must extend to the caching layer.** A semantic cache or query-result cache keyed only on the query text will happily serve one user’s memory to another. Key every cache entry on `(tenant_id, user_id, query_hash)` or disable cross-user caching entirely for memory-derived context.

**Support deletion and export as first-class operations.** GDPR and CCPA require the ability to delete an individual’s data on request. Because memories are embedded, “delete the row” is not enough — the vector, any cached summaries, and any derived records must all be removed. Design a user-scoped deletion path from day one; retrofitting it after you have millions of embeddings is painful.

**Encrypt at rest and minimize what you store.** Encrypt embeddings and payload columns, and apply a retention policy that expires episodic memory that no longer has operational value. The safest memory is the one you never had to keep.

—

## 5. Implementation: A Python + pgvector Memory Manager

Here is a compact but production-shaped implementation. It assumes Postgres with the `pgvector` extension and `psycopg` (v3) with the `pgvector` Python adapter, plus any embedding function (OpenAI, Cohere, or a local sentence-transformer — the interface is the same).

“`python
import hashlib
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

import psycopg
from pgvector.psycopg import register_vector

@dataclass
class Memory:
content: str
kind: str # semantic | episodic | procedural | preference
importance: float # 0.0 – 1.0, set by the writer
created_at: datetime

class AgentMemoryManager:
“””User-scoped, tiered memory with vector recall.”””

def __init__(self, dsn: str, embed_fn, tenant_id: str, user_id: str,
score_floor: float = 0.35):
self.embed = embed_fn
self.tenant_id = tenant_id
self.user_id = user_id
self.score_floor = score_floor
self.conn = psycopg.connect(dsn, autocommit=True)
register_vector(self.conn)
# Bind the tenant to the DB session; RLS enforces isolation.
self.conn.execute(
“SELECT set_config(‘app.current_tenant’, %s, false)”,
(tenant_id,),
)

# ———- write path ———-
def remember(self, content: str, kind: str = “semantic”,
importance: float = 0.5, dedupe_at: float = 0.95) -> None:
vec = self.embed(content)
# Supersede near-duplicates instead of accumulating them.
dup = self.conn.execute(
“””
SELECT id, 1 – (embedding <=> %s::vector) AS sim
FROM agent_memories
WHERE tenant_id = %s AND user_id = %s AND kind = %s
ORDER BY embedding <=> %s::vector
LIMIT 1
“””,
(vec, self.tenant_id, self.user_id, kind, vec),
).fetchone()
if dup and dup[1] >= dedupe_at:
self.conn.execute(
“UPDATE agent_memories SET content = %s, embedding = %s, ”
“created_at = now(), importance = %s WHERE id = %s”,
(content, vec, importance, dup[0]),
)
return
self.conn.execute(
“””
INSERT INTO agent_memories
(tenant_id, user_id, kind, content, embedding, importance)
VALUES (%s, %s, %s, %s, %s, %s)
“””,
(self.tenant_id, self.user_id, kind, content, vec, importance),
)

# ———- read path ———-
def recall(self, query: str, k: int = 6, max_tokens: int = 1200,
half_life_days: float = 30.0) -> list[str]:
qvec = self.embed(query)
rows = self.conn.execute(
“””
SELECT content, importance, created_at,
1 – (embedding <=> %s::vector) AS similarity
FROM agent_memories
WHERE tenant_id = %s AND user_id = %s
ORDER BY embedding <=> %s::vector
LIMIT %s
“””,
(qvec, self.tenant_id, self.user_id, qvec, k * 4),
).fetchall()

now = datetime.now(timezone.utc)
scored = []
for content, importance, created_at, sim in rows:
age_days = (now – created_at).total_seconds() / 86400
recency = 0.5 ** (age_days / half_life_days) # exponential decay
score = 0.7 * sim + 0.2 * recency + 0.1 * importance
if sim >= self.score_floor:
scored.append((score, content))

scored.sort(key=lambda x: x[0], reverse=True)

# Bound hydration by token budget, not by count.
hydrated, used = [], 0
for _, content in scored[:k]:
cost = len(content) // 4 # rough token estimate
if used + cost > max_tokens:
break
hydrated.append(content)
used += cost
return hydrated
“`

The DDL that makes this safe is equally short:

“`sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE agent_memories (
id BIGSERIAL PRIMARY KEY,
tenant_id UUID NOT NULL,
user_id UUID NOT NULL,
kind TEXT NOT NULL,
content TEXT NOT NULL,
embedding vector(1536) NOT NULL,
importance REAL NOT NULL DEFAULT 0.5,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

— RLS: the database enforces the tenant boundary.
ALTER TABLE agent_memories ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON agent_memories
USING (tenant_id = current_setting(‘app.current_tenant’)::uuid);

— Approximate-nearest-neighbor index for fast recall.
CREATE INDEX ON agent_memories
USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON agent_memories (tenant_id, user_id, created_at DESC);
“`

Three notes on production hardening. First, `SET set_config(…)` must run on *every* connection in a pool — wrap it in a connection factory, not at the call site. Second, the `k * 4` over-fetch before re-ranking exists because the similarity-only ordering is not the final ordering; the blended score needs headroom. Third, tune HNSW’s `m` and `ef_construction` against your recall target — default parameters are a starting point, not a guarantee.

—

## 6. ROI: Why Durable Memory Pays for Itself

Durable semantic memory is often framed as a capability feature. In production it is a **cost-reduction and reliability program**, and the numbers are not subtle.

**Token cost falls through the roof being removed.** An agent with durable memory does not need to carry a full session transcript forward, because it can retrieve the three facts that matter instead of re-reading the ninety turns that produced them. A 50-turn session that would have shipped 40K tokens of history per request now ships a 1.5K-token memory block. Across thousands of sessions, that is not a marginal saving; it is an order-of-magnitude change in cost per resolved task. And because the memory block is bounded by token budget, the saving is *structural* — it does not degrade as sessions get longer, which is precisely where transcript-stuffing collapses.

**Context stays dense, so reasoning improves.** Pruning the window to relevant memories reduces the lost-in-the-middle dilution that degrades answer quality as context grows. Teams consistently find that memory-augmented agents answer multi-session questions *more* accurately than long-context baselines, even at a fraction of the token spend.

**Task completion rates rise because re-work disappears.** The expensive part of a multi-day agent task is not the compute — it is the human time consumed by an agent that has forgotten the state of play: re-asking for credentials, re-running rejected approaches, re-discovering environment quirks. Procedural memory eliminates this class of failure outright. In our builds, the metric that moves first is not latency or cost but **first-pass task completion**, and it is the one that matters most to the business, because a task that completes in one pass costs a fraction of one that needs two.

**Humans stop being the memory.** The hidden cost of amnesiac agents is that operators become the persistence layer — copying context between sessions, re-explaining the customer, babysitting the handoff. Durable memory moves that burden back into the system, which is where it belongs.

The economics compound in the same direction as the technology: memory is written once and retrieved cheaply forever, while transcript-stuffing pays full token price on every single turn. Over a long-horizon workload, that asymmetry is decisive.

—

## Conclusion: Build the Memory Layer Before It Bites You

Context-window amnesia is not a model limitation you can wait out. Bigger windows delay the problem and inflate the bill; they do not give an agent persistence, scope, or the ability to forget selectively. The durable solution is architectural: a tiered memory system, semantic retrieval that hydrates the context window on demand, and hard tenant isolation enforced in the database rather than in a prompt.

If you are standing up long-horizon agents against real systems — Oracle, MS SQL, Postgres, Snowflake, your CRM — and you need memory that is fast, private, and auditable, the OnlineCloud team builds exactly this layer: vector-backed memory, MCP-native tool surfaces, and the eval harness that proves it works before it touches a customer. [Talk to the OnlineCloud engineers](https://www.onlinecloud.dev/contact) about mapping your first memory-backed vertical slice — scoped in a discovery week, built by senior engineers only, with no offshore handoff.