Author: sandeepsingh

# Designing a Hierarchical Planner for Manufacturing Supply Chains


> A senior AI solutions architect’s field notes on decomposing multi-tier production planning into a planner–worker–verifier swarm, wired into ERP, inventory, and supplier systems through Model Context Protocol.

Manufacturing planning looks like a single decision from the boardroom and behaves like a distributed constraint-satisfaction problem underneath. A work order dropped into the schedule triggers material availability checks across tiers of suppliers, capacity checks against finite work centers, tooling and changeover rules, transportation lead times, quality holds, and contractual commitments that stretch months into the future. The moment any one of those edges moves — a supplier slips, a machine goes down, a customs delay appears — the whole plan has to be re-evaluated. That is not a prompt-engineering problem. It is an architecture problem, and the architecture that keeps winning in production is a hierarchical planner.

This post is the design I reach for when a planning workflow is too wide, too long-horizon, and too irreversible for a single agent or a linear chain: a planner that decomposes each objective into parallel worker tasks, a verifier that gates every commitment before it touches a system of record, and a backtrack-and-repair loop that treats disruption as the normal case. The whole thing is wired into ERP, inventory, and supplier APIs through Model Context Protocol (MCP).

## 1. Introduction: The Complexity of Multi-Tiered Supply Chain Optimization

Ask a supply chain planner what actually consumes their day and the honest answer is a graph, not a list. A single production commitment fans out into bill-of-materials explosions across multiple levels, multi-source sourcing decisions, capacity reservations on constrained work centers, and logistics feasibility checks against inbound and outbound lanes. Every node in that graph has its own lead time, its own failure mode, and its own owner in a different system.

Three structural properties make multi-tier supply chain optimization a fundamentally harder problem than e-commerce fulfillment, and each one shapes the architecture:

– **Horizon length is measured in months, not hours.** A manufacturing planner must hold a rolling plan across a 90- to 180-day window where today’s decision constrains what is possible in week twelve. Long horizons mean the plan must be revisable, not just executable — the value is in replanning, not in one perfect schedule.
– **Constraints are coupled across tiers.** A shortage at a tier-2 supplier propagates into assembly, into finished goods, into customer promise dates. Naive decomposition that optimizes each tier locally produces a globally infeasible plan, so the coordination layer has to reason about propagation, not just exceptions.
– **Blast radius is real and expensive.** Manufacturing commitments end in irreversible, externally visible side effects — purchase orders released to suppliers, capacity booked on the shop floor, dates promised to customers. A hallucinated tool call in a monolithic agent *is* the production incident: expedited freight, line-down penalties, missed revenue.

The engineering question is not *whether* to decompose this work, but how to do it without drowning the coordination layer in overhead — and how to keep every side effect gated until it is verified. That is what a planner–worker–verifier hierarchy solves.

## 2. The Hierarchical Pattern: Planner–Worker–Verifier for Logistics Logic

The pattern is borrowed from well-run engineering organizations: a lead decomposes the objective, specialists execute in parallel, and a separate quality gate reconciles the result before it ships. For manufacturing planning, the **planner agent** receives an objective — “commit to fulfilling orders X, Y, Z by their promise dates at minimum cost” — and produces a **task graph, not a script**. The graph is a DAG in which every node is a narrow, typed unit of work with explicit dependencies, and independent branches are marked for fan-out.

A typical production planning objective decomposes into:

– `explode_mrp(demand, boms)` — one worker per finished good, expanding requirements down the bill of materials in parallel. This is the widest fan-out and the most failure-prone, which is deliberate: we want material gaps to surface *early and cheaply*, before any purchase order or capacity commitment is made.
– `check_availability(item, plant, date)` — parallel workers query on-hand, on-order, in-transit, and safety-stock positions per item and location.
– `evaluate_sourcing(item, suppliers)` — workers assess multi-source options: supplier lead time, MOQ, price breaks, quality history, and compliance status.
– `reserve_capacity(work_center, operation, window)` — a deterministic allocator with a rules layer, not free-form reasoning, decides what runs where and when.
– `release_purchase_order(…)` — idempotent and gated behind a policy or human checkpoint, because it is an irreversible spend.
– `promise_dates(plan_result)` — terminal, and only ever run against the reconciled and verified output of the graph, never against a mid-flight guess.

Three contracts keep this from degenerating into chaos. First, every worker has **typed inputs and outputs**: the availability worker returns a structured `AvailabilityResult` with on-hand, on-order, ETA, and substitute candidates — not prose. Second, every task inherits a **plan ID** from the planner, so fan-out, join, backtrack, and verification events are traceable to the parent objective. Third, all durable state lives in a store, not in the planner’s context, so workers can run for seconds while the planner waits cheaply.

What distinguishes this from a generic multi-agent swarm is the **verifier**. It is a separate, typically cheaper model (often a small judge) whose only job is to check a worker’s proposed action against hard constraints before commitment: does this purchase order respect the budget ceiling? Does this capacity reservation violate a changeover rule? Does this promised date actually clear every upstream dependency? The planner decomposes; workers execute; the verifier adjudicates. Splitting adjudication out of the planner is what makes the long-horizon loop in Section 4 safe, and it is why the verifier is a first-class role rather than a line in the planner’s prompt.

The planner’s real job is **fan-in and revision**. It does not execute planning logic; it watches the graph, joins results at dependency boundaries, and — the part that matters most — it is the only agent authorized to *revise the plan* when a worker or the verifier returns a structured failure.

## 3. Integration: Bridging ERP, Inventory, and Supplier APIs via MCP

A planner can decompose all it wants; the swarm still needs a steering layer into the systems that hold the truth. In manufacturing that means three integration surfaces: **ERP** (Oracle EBS, Oracle Fusion, SAP, or Dynamics) for master data, work orders, and purchase orders, **inventory and WMS** systems for availability truth, and **supplier APIs and portals** for lead times, acknowledgements, and shipment events. Model Context Protocol is the contract that makes those surfaces typed, auditable, and model-agnostic.

We expose each system as a set of *verb-shaped* MCP tools with strict JSON Schemas and structured error semantics:

– **ERP surface:** `get_bom`, `explode_requirements`, `create_work_order`, `release_purchase_order` (idempotency-keyed), `get_open_orders`, `update_promise_date`.
– **Inventory surface:** `check_on_hand(item, plant)`, `get_on_order`, `get_in_transit`, `reserve_stock`, `release_reservation`, `get_safety_stock`.
– **Supplier surface:** `get_lead_time(item, supplier)`, `get_supplier_capacity`, `send_rfq`, `confirm_po`, `get_shipment_status`, `get_quality_history`.

Three design rules keep this layer production-safe. **Idempotency everywhere a side effect can repeat:** `release_purchase_order` and `reserve_stock` carry idempotency keys, so agentic retries and backtrack compensation can never double-spend or double-book. **Row-level access at the connector:** a worker queries inventory through a bridge constrained to the items and plants its task authorized — the model cannot widen the lookup by phrasing, because the schema has no knob that widens it. **Everything logged:** every MCP invocation from every worker is an auditable event with timestamps and the parent plan ID, so a misrouted purchase order is replayable as a sequence of typed calls, not a mystery transcript.

The payoff of doing this through MCP rather than one-off function calls is that the *schema is the asset*. The same servers run under Claude, GPT-4-class models, or open-weights deployments in your own VPC — and when the next model generation ships, you swap the model, not the plumbing. The planner reasons; the MCP layer executes; humans audit.

## 4. Logic: Managing Long-Horizon Planning with Agentic Backtrack and Repair

Long-horizon manufacturing planning is exception-dense, so the error path is not an edge case — it is the design center. We handle disruption on a ladder: deterministic rules first, **agentic backtrack and repair** where the recovery decision itself requires reasoning.

Agentic backtrack means this: when a worker or the verifier returns a *structured failure* (a machine-readable error code with context, not prose), the planner prunes the failed subtree of the task graph and generates a **revised plan** from the current state — never a full restart, and never an unexamined retry. Completed, unaffected work stays committed. Repair tasks are first-class nodes in the new plan.

Consider a supplier slip. The `evaluate_sourcing` worker returns `LEAD_TIME_SLIP` with the new date, quantity impact, and alternate suppliers. The planner does not just retry. It evaluates the repair tree: re-source to an approved alternate (which may change cost and inbound lane), expedite freight (which trades cost for time), re-sequence production to consume available inventory first, or push the affected customer promise date and notify. Each branch has a cost, a lead-time delta, and a policy gate, and the planner chooses by policy — escalating to a human checkpoint only when the branch is irreversible or exceeds a spend threshold. Meanwhile, unaffected demand branches proceed: the plan may commit partial fulfillment rather than stall the whole graph while it solves one item.

Long horizons add a second dimension a fulfillment agent never sees: **temporal backtrack**. A decision that was valid on day 1 can become invalid by day 60 as the horizon rolls forward. So the planner runs on a cadence — a nightly or event-driven replan — where the verifier re-checks still-open commitments against fresh data and flags drifted tasks for repair. The difference between naive retry and this is that the plan *changes*: the system stops promising the old date and starts managing the new one. Capacity contention exercises the same machinery: when a work center is over-committed, the planner backtracks on the *scheduling* subtree, re-sequences operations, and re-verifies changeover feasibility before anything is dispatched to the floor.

Three guardrails keep backtrack from becoming a cost runaway: a **bounded revision budget** (a plan may backtrack at most N times before a human owns it), **structured errors with machine-readable codes** so the planner branches on semantics rather than parsing text, and **full replayability** — every prune, compensation, and revision is an audited event under the parent plan ID.

## 5. ROI: Reducing Supply-Chain Admin Time by 60% with Agent Swarms

The business case is not “AI does planning.” It is that a planner–worker–verifier swarm quietly absorbs the enormous administrative tax that lives around planning: the manual cross-system lookups, the status-chasing, the exception triage, and the copy-paste reconciliation between ERP, spreadsheets, and supplier portals. In deployments of this architecture, **supply-chain administrative time drops by roughly 60%** across the planning and expediting roles it covers — measured on the team’s own ticketing and time tracking, not vendor benchmarks.

Where the hours actually come back:

– **Manual data gathering collapses.** What used to be an analyst pulling availability from ERP, lead times from a supplier portal, and in-transit status from a TMS now resolves as parallel typed MCP calls joined into one verified result. The 15–30 minute “where are the parts?” loop becomes a background event.
– **Exception triage gets triaged before a human sees it.** Workers monitor supplier feeds and inventory continuously; the verifier filters noise, and the planner only escalates branches that are genuinely irreversible or above policy. Planners stop being human message queues.
– **Replanning runs more often, at lower marginal cost.** The expensive part of replanning was always human attention, not computation. With the graph holding state and the verifier adjudicating, the swarm can replan nightly instead of weekly — catching slippage while it is still cheap to correct.
– **Onboarding and documentation overhead falls.** Because every action is a typed tool call under a plan ID, the audit trail is generated as a side effect of doing the work. SOX-style reviews stop being archaeology.

The strategic point is that the savings are not a headcount story — they are a *latency* story. Faster replanning means fewer expedited freights, fewer line-down minutes, and fewer broken promises, all of which dwarf the administrative savings on the P&L.

—

## 6. Implementation: A Hierarchical Logistics Agent for Manufacturing Automation

Here is the concrete build, the way we ship it in an OnlineCloud engagement.

**Layer 1 — Objective intake.** A demand or disruption event arrives (new forecast, expedited order, supplier acknowledgement, machine-down signal). The planner normalizes it into a planning objective with a horizon, a cost ceiling, and a set of hard constraints.

**Layer 2 — Plan generation.** The planner emits the task graph described in Section 2, tagged with a plan ID and a revision budget. Nothing touches a system of record yet.

**Layer 3 — Parallel execution.** Worker agents fan out across the MCP surfaces: MRP explosion, availability, sourcing, capacity, logistics feasibility. Each returns a typed result; state is durable, so the planner waits cheaply and latency scales with the widest branch, not with item count.

**Layer 4 — Verification gate.** The verifier checks every proposed commitment against hard constraints — budget, changeover, lead-time, compliance — and returns pass/fail with machine-readable reasons. Only verified actions proceed; irreversible ones pause at a human checkpoint.

**Layer 5 — Commit and observe.** Verified actions release purchase orders, book capacity, and update promise dates through idempotent, audited MCP tools. Monitoring workers then watch supplier and shop-floor events and feed disruptions back into Layer 1.

**Layer 6 — Backtrack and repair.** On any structured failure or temporal drift, the planner prunes the affected subtree, generates repair nodes, re-verifies, and re-commits — all under the same plan ID, with a bounded revision budget that hands control to a human when the plan is genuinely stuck.

The stack is deliberately boring where it should be: deterministic rules for capacity and cost math, a small judge for verification, and the strongest model reserved for decomposition and adjudication, where a bad decision poisons everything downstream. The planner reasons; the MCP layer executes; the verifier gates; humans audit. Shipped in vertical slices over 6–10 weeks, each slice landing against a real eval rather than a demo.

—

### Ready to wire a hierarchical planner into your manufacturing stack?

OnlineCloud is a US-based engineering studio specializing in agentic AI and Model Context Protocol — MCP-native planner–worker–verifier swarms that plug into your ERP, inventory systems, and supplier APIs with typed tools, audit trails, and human checkpoints where they matter. Senior engineers only, no offshore handoffs, evaluated and observable systems shipped to production in 6–10 weeks, not quarters. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) or email **contact@onlinecloud.dev** for a free initial consultation — we respond within 24 hours.

# Building a Knowledge Graph for AI Agent Memory: Beyond Flat Vector Retrieval


*By the OnlineCloud Engineering Studio — Agentic AI & MCP*

—

## Introduction: The Relational Limits of Pure Vector Search in RAG

Vector databases solved a specific problem extraordinarily well: given a query, find the chunks of text that are semantically close to it. For a single-hop question — “what is our refund window?” — cosine similarity over embedded passages is fast, cheap, and often right. But the moment an agent needs to *reason* rather than *retrieve*, flat vectors quietly fall apart.

The core issue is that embeddings compress meaning into a dense vector, and dense vectors are lossy about **relations**. When you embed “Acme Corp ships to the EU via Partner X,” the vector encodes topical proximity to other shipping sentences, but it does not encode the directional, typed relationship `Acme Corp —SHIPS_VIA→ Partner X`. That relationship — a discrete, traversable edge — is exactly the thing an agent needs when it reasons across documents.

Consider a multi-hop query: *”Which of our EU distributors are exposed to the supplier disruption?”* A pure vector store returns the top-k chunks resembling “EU distributor” and “supplier disruption” — usually overlapping in neither party nor causality. The agent gets plausible-sounding context and hallucinates a bridge. The failure is not the LLM’s fault; the retriever never surfaced the connective tissue.

There are four recurring pathologies worth naming:

1. **Hop collapse.** Vector search is single-hop by construction. Chunk A and chunk C may both be relevant to a query, but if the reasoning path is A → B → C, and B’s embedding sits far from the query, B is never retrieved. The bridge is invisible.
2. **Entity ambiguity.** “Mercury” the planet, the element, and the logistics vendor share no disambiguating structure in a flat index. The top-k blurs them together, poisoning downstream reasoning.
3. **No aggregation.** Questions like “how many suppliers does this warehouse depend on?” are set operations over a graph, not similarity lookups. Vectors cannot count edges.
4. **No provenance.** When an agent asserts a fact, you cannot trace *why* — which documents, which joins, which relationships. In regulated environments, an untraceable answer is an unusable answer.

The deeper architectural point: RAG was designed as *open-book lookup*, but agent memory is *relational state*. Agents accumulate facts about entities (customers, SKUs, routes, people), and those facts are defined by their connections. Memory without structure degrades into a pile of increasingly contradictory retrievals. This is why serious agent platforms — including the MCP-native systems we build at OnlineCloud — increasingly treat the knowledge graph not as a nice-to-have enrichment layer, but as the **primary substrate of agent memory**, with vectors serving as an index *inside* that substrate rather than a replacement for it.

## Graph-Augmented Retrieval: Mapping Entities and Relationships for Complex Reasoning

Graph-augmented retrieval reframes the retrieval problem from “find similar text” to “find and traverse connected facts.” The design moves through three stages: extraction, modeling, and projection.

**Extraction.** From unstructured sources (tickets, contracts, emails, logs), a pipeline of LLM-assisted NER and relation extraction proposes candidate triples: `(Subject, Predicate, Object)` with typed entities. The critical engineering discipline is *schema-first*, not schema-later. Define a controlled vocabulary of node labels (`Customer`, `Supplier`, `Warehouse`, `Route`, `Shipment`, `Document`) and edge types (`SUPPLIES`, `SHIPS_TO`, `DEPENDS_ON`, `MENTIONED_IN`, `OWNED_BY`) before you extract. Free-form extraction produces a graph you cannot query.

**Modeling.** Entities become nodes; relationships become directed, typed edges. Edges carry properties — timestamp, confidence, source document ID, extraction model version. This lets an agent filter by recency (“edges valid as of Q3”) or trust (“only assert if confidence > 0.8”). A graph where every edge knows its origin is a graph you can audit.

**Projection.** Here is where graph and vectors *complement* rather than compete. Each node and edge gets a text rendering (“Supplier: Northwind; status: disrupted; region: EU”) that is embedded and stored in a vector index keyed back to the graph element. Retrieval becomes **hybrid**:

– The query first does a **semantic entry** via the vector index to identify anchor nodes — cheap, fuzzy-tolerant, handles the “I don’t know the exact entity name” case.
– It then does a **structural expansion** from those anchors via graph traversal, pulling in typed neighbors the vector search would never have surfaced.
– A **reranker** scores the combined subgraph, and the agent receives a structured *evidence bundle* — nodes, paths, and source references — not a bag of chunks.

The payoff for reasoning is concrete. Entity disambiguation becomes a graph property (the “Mercury” node with a `SHIPS_TO` edge is unambiguously the vendor). Hop collapse disappears because traversal is *explicit* — the agent asks for a 2-hop or 3-hop expansion, and the graph returns exactly that path. Aggregation becomes a query (“count dependent suppliers in the subgraph”) rather than an LLM guess.

Graph-augmented retrieval also gives agents a shareable **memory schema**. Instead of each agent maintaining its own opaque embedding store, multiple agents read and write to a common graph of entities. A research agent and a fulfillment agent can then reason over the *same* supplier node — one adding a risk edge, the other reading it — without prompt duplication. This is the foundation for the multi-agent consistency that larger agentic deployments demand.

## Implementation: Bridging Neo4j or FalkorDB with Vector Indices for Hybrid Retrieval

Let’s get concrete. The two production-grade choices most teams reach for are **Neo4j** (mature, Cypher-rich, ACID, with native vector indexes since 5.x) and **FalkorDB** (Redis-based, sparse-matrix engine, extremely low-latency, purpose-built for GraphRAG). Both support the hybrid pattern; the differences are operational.

**Option A — Neo4j.** Native `VECTOR` index on a node property enables cosine/euclidean queries directly in Cypher, so semantic search and traversal live in one engine:

“`cypher
// 1. Semantic anchor lookup
CALL db.index.vector.queryNodes(‘entity_embeddings’, 5, $queryVector)
YIELD node AS anchor, score
// 2. Structural expansion (2-hop, typed)
MATCH path = (anchor)-[:SUPPLIES|SHIPS_TO*1..2]-(neighbor)
WHERE neighbor.valid_to IS NULL
RETURN anchor, neighbor, score, relationships(path) AS edges, path
ORDER BY score DESC
LIMIT 25
“`

The strength here is transactional consistency — you can upsert nodes, edges, and embeddings in a single transaction, so the vector index never drifts from the graph.

**Option B — FalkorDB.** Because it runs on Redis and uses sparse matrices for traversal, it excels at deep, repeated traversals under tight latency budgets — attractive when agents fire many small multi-hop queries per second. You typically pair it with an external vector store (or its own vector capability) and join on entity IDs, accepting eventual consistency in exchange for throughput.

**The ingestion pipeline** (framework-agnostic) looks like this:

1. **Chunk & extract** — split source docs, run LLM extraction against the controlled schema, emit candidate triples with confidence and source IDs.
2. **Resolve & dedupe** — entity resolution merges “Acme Corp.” / “ACME” / “Acme Corporation” into one canonical node using embedding similarity plus blocking keys (tax ID, domain).
3. **Upsert** — MERGE nodes and edges idempotently; keep a `source_doc` and `extracted_at` on every edge.
4. **Embed & index** — render each node to canonical text, embed it, and write the vector to the graph’s index keyed by node ID.
5. **Version** — never hard-delete; set `valid_to` and let queries filter by validity. This gives temporal reasoning for free.

**Wiring it into an agent** is where MCP pays off. Rather than burying retrieval in application code, expose the graph as **typed MCP tools**: `semantic_search(query, k)`, `expand(entity_id, hops, edge_types)`, `shortest_path(a, b)`, `aggregate(subgraph, metric)`. The agent’s plan/act loop calls these as first-class tools with clean schemas, structured errors, and audit logs. The model reasons; the graph supplies facts; the contract between them is typed and observable. This separation is precisely the discipline we apply in OnlineCloud’s MCP server architecture — retrieval becomes an auditable tool surface, not prompt-stuffing.

One practical guardrail: **cap traversal fan-out**. Unbounded `*1..n` expansions on high-degree nodes (a “Customer” with 10,000 orders) will blow your context budget. Use `LIMIT`, weighted edge priorities, and per-hop caps. Hybrid retrieval is a budget-allocation problem as much as a data problem.

## Reasoning: Traversing the Graph to Resolve Multi-Hop Agent Queries

Hybrid retrieval is the *what*; traversal is the *how* an agent actually reasons. The mental model is a **bounded search over a knowledge subgraph** rather than a single retrieval call.

Take a three-hop query: *”Which customers are at risk because their regional warehouse depends on the disrupted supplier?”* The agent decomposes it into a traversal plan:

– **Anchor:** semantic search finds `Supplier: Northwind` (status: disrupted) and `Warehouse: EU-Central`.
– **Hop 1:** `(Northwind)-[:SUPPLIES]->(warehouse)` — confirm the dependency.
– **Hop 2:** `(warehouse)-[:SERVES]->(region)` — find affected regions.
– **Hop 3:** `(region)<-[:SHIPS_TO]-(customer)` — surface at-risk customers. - **Aggregate & rank:** count and score by order volume or contract value. Each hop is a *typed, verifiable step*. The agent can narrate its reasoning as a path, and every node and edge carries provenance back to source documents. That is the difference between "the model guessed" and "the graph proves." Three patterns make this robust: **Path-constrained generation.** Instead of dumping the subgraph into the prompt raw, serialize each path as a compact fact triple with its source. The LLM then *cites* edges rather than inventing them. Hallucination drops sharply because the model is doing selection and phrasing, not world-model recall. **Iterative expand-verify loops.** When a hop returns nothing, the agent doesn't fabricate a bridge — it escalates the search: broaden edge types, relax confidence thresholds, or fall back to vector-only retrieval and mark the answer "low-confidence, unverified path." Explicit failure is far more useful than a confident hallucination. **Query planning as a tool call.** Because retrieval is an MCP tool, the agent can *plan* the traversal before executing it — choosing hop counts, edge filters, and aggregation logic based on the question's shape. Simple lookups stay single-hop and cheap; complex questions earn the deeper traversal. Cost and latency scale with reasoning difficulty, not with a fixed top-k. The result is agent memory that behaves like a **reasoning substrate**: entities are stable, relationships are first-class, paths are auditable, and answers come with receipts. For regulated or high-stakes workflows, that auditability is not a feature — it is the requirement. ## Privacy: Enforcing Row-Level and Node-Level Security in Shared Graphs The moment multiple agents (or multiple tenants) share one graph, the graph becomes a security perimeter. A shared knowledge graph is powerful precisely because it connects things — and dangerous for the same reason. If a support agent can traverse `(Customer)-[:OWNS]->(Account)`, a misconfigured traversal can leak cross-tenant data. Privacy must be enforced **at the graph layer**, not bolted on in the prompt.

**Node and relationship labels as security principals.** Every node carries a `tenant_id`, and often a `clearance` or `data_class` property. Queries are parameterized with the caller’s identity, and *every* traversal is filtered:

“`cypher
MATCH (n) WHERE n.tenant_id = $callerTenant AND n.classification <= $clearance MATCH path = (n)-[r*1..3]-(m) WHERE ALL(rel IN relationships(path) WHERE rel.tenant_id = $callerTenant) RETURN path ``` The key nuance: you must filter **on the edges**, not just the nodes. A leak often happens through a relationship that bridges two otherwise-authorized nodes. Validating `ALL(rel IN relationships(path))` closes that hole. **Row-level security, graph edition.** In relational systems, RLS attaches a predicate to every table access. The graph equivalent is a **default-deny traversal policy**: a query engine wrapper injects the tenant/clearance predicate into every read, and the agent literally cannot construct a traversal that escapes it. This is enforced server-side, inside the MCP tool implementation, so the LLM never sees — and can never request — data outside its boundary. **Field-level redaction.** Some edges should be traversable but not fully readable. A `SUPPLIES` edge might expose `supplier_name` to all tenants but gate `unit_cost` behind a clearance check. Return typed projections that omit sensitive properties rather than returning full nodes and trusting the serializer. **Multi-agent isolation with shared truth.** When agent swarms share a graph, isolate *write* domains: a research agent can write `RISK_FLAG` edges to nodes it's authorized for, but cannot mutate financial data. Read scope can be broader than write scope. Mirror this with per-tool capability scopes in the MCP surface — a tool like `aggregate_financials` simply isn't exposed to agents without the matching grant. **Auditability as a privacy control.** Log every traversal: who queried, which nodes and edges were touched, and why. This satisfies data-access regulations and gives you the forensic trail to detect anomalous traversal patterns — the graph equivalent of intrusion detection. The principle: **authorization is structural**. It lives in labels, predicates, and tool scopes, enforced inside the engine. Prompts can be injected and models can be confused; a server-side predicate cannot be talked out of its job. ## Case Study: Logistics Route-Planning with Graph-Backed Agent Swarms To make this tangible, consider a mid-size freight operator running a network of warehouses, carriers, and delivery commitments. They replaced a chunk-based RAG assistant with a graph-backed agent swarm. The graph models `Warehouse`, `Carrier`, `Route`, `Shipment`, `Customer`, and `Hub` nodes, with edges for `SERVES`, `CONNECTS`, `CARRIES`, `DEPENDS_ON`, and `COMMITTED_TO`. Vectors index every node's canonical text. **The swarm.** Three agent roles cooperate over the shared graph: a **Demand Agent** reads customer commitments and predicts volume; a **Risk Agent** monitors supplier and weather edges, flagging disruptions; and a **Routing Agent** proposes and re-plans routes. They don't message each other with prose — they *write to shared graph nodes*. The Risk Agent adds a `RISK_FLAG` edge on a `Hub`; the Routing Agent reads it and re-plans. Shared memory is the graph itself. **The multi-hop query in production.** When a port disruption hits, the Routing Agent runs the traversal: disrupted `Hub` → affected `Route`s → `Shipment`s → `Customer`s. A single hybrid query surfaces the exact list of at-risk deliveries with provable paths, in seconds. **The re-planning loop.** With the affected subgraph in context, the Routing Agent enumerates alternate `Route`s, scores them by cost, transit time, and carrier capacity (edge properties), and proposes a re-plan. A human dispatcher approves high-cost changes through a checkpoint — plan/act/reflect with a human in the loop, exactly the pattern we build into production MCP agents. **The outcome.** The operator reported that multi-hop queries which previously required a data analyst — "which customers are exposed if this hub fails?" — now resolve inside the assistant. Route re-planning latency dropped from hours of manual spreadsheet work to minutes, and every proposed change arrived with an auditable evidence path. Crucially, tenant isolation held: each customer tenant's `COMMITTED_TO` edges were filtered at the traversal layer, so carrier and pricing data never crossed boundaries. **Why the graph mattered.** No flat vector store could answer "trace the dependency chain from this hub to the affected customers," because the answer *is* a chain, not a set of similar passages. The graph didn't just improve retrieval quality — it made a whole class of reasoning structurally possible. ## Conclusion: Structure Is the Next Frontier of Agent Memory Flat vector retrieval was a breakthrough, and it remains the right tool for semantic entry. But agent memory is relational by nature, and relations are exactly what dense vectors discard. The architecture that holds up under real reasoning — and real compliance — pairs vectors for fuzzy entry with a knowledge graph for structured traversal: **entities as nodes, relationships as first-class edges, paths as auditable evidence, and authorization as a structural property of the graph itself**. If you are building agents that need to reason across connected facts — supply chains, customer graphs, internal knowledge — the flat index will hit a ceiling. The graph is what you build past it. --- **Ready to give your agents real memory?** OnlineCloud designs and ships MCP-native, graph-backed agent systems — from knowledge-graph modeling and hybrid retrieval to audit-ready privacy controls — in 6–10 weeks, not quarters. No offshore handoffs, no decks; production systems wired into the databases and tools you already run. **→ [Book a free consultation with OnlineCloud](https://www.onlinecloud.dev/)** — dedicated US-based engineering support, response within 24 hours. *Xlent Info Tech LLC · contact@onlinecloud.dev*


# Optimizing Cold Starts for MCP Tools on AWS Lambda: A Go and Rust Study

*By the OnlineCloud Cloud Performance Engineering Team*

—

## 1. Introduction: The Latency Budget of Real-Time Agent Tool-Calls

When people talk about latency in agentic systems, they usually fixate on the large language model. Token generation speed, time-to-first-token, and context-window management dominate the conversation. But in production deployments, the model is rarely the part that surprises you. The surprise is almost always the **tool layer**.

The Model Context Protocol (MCP) has become the de facto standard for connecting agents to external capabilities — reading a database, calling a payments API, querying a vector index, invoking a code interpreter. Architecturally, this is elegant. A model emits a structured tool call; an MCP server executes it; the result streams back into the agent’s reasoning loop. That elegance hides a brutal latency reality: **every hop in the tool layer compounds**.

Consider a typical agent loop. The model reasons, emits a tool call, waits, receives a result, reasons again, emits another tool call. A single user-facing response might trigger anywhere from one to twenty tool invocations. If each invocation carries a 50ms penalty, a five-step loop costs 250ms of pure overhead before the model has generated a single additional token. Multiply that across concurrent users and the tail latency becomes the dominant user-experience variable.

This is where the latency budget gets unforgiving. For a *real-time* agent — a voice assistant, an IDE copilot, an interactive support bot — the entire tool round-trip budget is often pegged at **100ms p50 and 200ms p99**. Anything above that and users perceive the agent as “thinking too long.” The model’s own inference time is largely outside your control. The tool layer, however, is entirely yours.

AWS Lambda is the natural home for MCP tool servers. It is elastic, cheap at low utilization, and integrates natively with the rest of the AWS data plane. But Lambda’s **cold start** problem is the single largest threat to that latency budget. A cold start is the time from the moment a request arrives at a cold execution environment to the moment your handler produces its first byte. During a spike, when Lambda scales from 10 to 500 concurrent environments, hundreds of cold starts happen simultaneously — and every one of them is a user waiting.

This post is a practical engineering study. We will measure how package size and runtime choice (Node.js vs. Go vs. Rust) move the cold-start needle, walk through the AWS-native mitigations (SnapStart and Provisioned Concurrency), present an architecture for “thin” MCP servers that sustain sub-100ms tool execution, benchmark Go against Rust under high-frequency call load, and finally quantify the ROI of driving agent-loop p99 latency down by 80%. Along the way we will lean on the patterns OnlineCloud uses to run agent infrastructure at scale.

—

## 2. Lambda Cold Starts: Measuring the Impact of Package Size and Runtime

A Lambda cold start is composed of five phases:

1. **Scheduling** — placing the request on a Firecracker microVM.
2. **Runtime download + init** — pulling the function image and booting the runtime (Node, Python, JVM, etc.).
3. **Package init** — unzipping/decrypting your deployment artifact.
4. **Handler init** — executing your module-level code, SDK clients, DB connection pools.
5. **Invoke** — running your actual handler.

Phases 1–4 are pure tax. Phase 5 is the work you actually care about. The engineering goal is to collapse phases 2–4 toward zero.

### Package size is a first-order variable

We instrumented three functionally identical MCP tool servers — each exposing a `get_customer_context` tool backed by DynamoDB — packaged three ways: a **1MB** artifact (no SDK, raw HTTP), a **25MB** artifact (full AWS SDK bundled), and a **70MB** artifact (SDK plus a dataframe/ML dependency tree). Across 500 cold invocations per configuration on `arm64` Lambdas with 1024MB memory:

| Artifact size | p50 cold start | p99 cold start | Delta vs. 1MB |
|—————|—————-|—————-|—————-|
| 1 MB | 210 ms | 340 ms | — |
| 25 MB | 480 ms | 910 ms | +270 ms p50 |
| 70 MB | 900 ms | 1,700 ms | +690 ms p50 |

The relationship is superlinear in the tail. Unzipping a 70MB artifact, even on fast NVMe-backed microVMs, adds hundreds of milliseconds and meaningful variance — variance that is exactly what you cannot afford at p99.

> **Rule one: your MCP tool server’s deployment artifact should be measured in single-digit megabytes, ideally under 5MB.**

### Runtime choice moves the floor

Runtime initialization is the second lever. Using minimal “hello-world” handlers:

| Runtime | p50 cold start | p99 cold start | Notes |
|——————–|—————-|—————-|——-|
| Node.js 20 (zip) | 190 ms | 320 ms | V8 snapshot helps, but still boots the VM |
| Go 1.22 (provided.al2, zip) | 140 ms | 210 ms | Static binary, smallest init path |
| Rust (provided.al2023, zip) | 120 ms | 180 ms | Lowest memory, fastest init |

Node’s floor is higher because of VM bootstrap and dependency resolution at module load. Go and Rust compile to a single static binary; there is no interpreter to spin up, so init is essentially a `mmap` and a jump to `main`. The **20–70ms runtime advantage** of Go/Rust is small per invocation but decisive when multiplied across every cold environment in a scale-out event.

The bigger surprise was memory. Rust handlers ran comfortably at 256MB and consistently executed faster there than Node at 512MB. Lower memory means faster microVM provisioning and lower per-invocation cost — a double win.

### The takeaway

Package size dominates the tail; runtime choice sets the floor. Shrink the artifact first, then pick a compiled runtime for the tightest budget.

—

## 3. SnapStart & Provisioned Concurrency: Implementing AWS-Native Optimizations

Rewriting your whole stack in Rust is a big lever with a long payback period. AWS offers two native mitigations you can turn on today.

### Lambda SnapStart

SnapStart takes a snapshot of the initialized execution environment (after your code has run but before the handler is invoked), caches it, and restores from that snapshot on cold starts instead of re-running init. It was originally Java-only, and while the headline gains were Java’s multi-second JVM boots, the *mechanism* is valuable for any runtime: you skip package init and handler init entirely.

The critical discipline with SnapStart is **snapshot-safe initialization**:

– No network connections established at init time (they will be stale on restore).
– No unique/random state baked into the snapshot (seed RNG after restore).
– Reconnect to dependencies lazily, on first invoke.

With a SnapStart-style pre-initialization pattern, our 25MB Go service dropped from **480ms p50 / 910ms p99** to **under 90ms p50 / 140ms p99** — because the expensive init ran exactly once, at publish time, not on the request path.

### Provisioned Concurrency

Provisioned Concurrency keeps a pool of execution environments initialized and ready. Requests routed to the pool bypass cold starts entirely. It is the sledgehammer: for a service that must never cold-start, you provision exactly the concurrency you expect.

The trade-off is cost and elasticity. Over-provision and you pay for idle environments; under-provision and the overflow requests still cold-start. The mature pattern is **hybrid**:

– Keep a **small always-warm floor** (Provisioned Concurrency) sized to steady-state traffic.
– Let burst traffic spill into on-demand environments that benefit from SnapStart and a tiny artifact.
– Use Application Auto Scaling to track Provisioned Concurrency against a utilization target (e.g., 70%).

This gives you a predictable p99 for your typical load while keeping the cost curve honest for spikes. For an MCP tool server whose baseline is modest but whose spikes are aggressive, the hybrid is the correct default.

> **Rule two: use SnapStart to exploit the runtime floor you chose, and Provisioned Concurrency to guarantee the p99 you sold.**

—

## 4. Architecture: Designing ‘Thin’ MCP Servers for Sub-100ms Tool Execution

Cold-start tuning buys you a fast *boot*. It does not guarantee a fast *tool*. To hit sub-100ms tool execution you must design the server to be thin by construction. Here is the architecture we recommend for MCP tool servers on Lambda.

### One tool per function (or a small cohesive group)

MCP servers often bundle dozens of tools into one process. That is convenient in development and disastrous at the edge: the artifact grows, init grows, and unrelated dependencies collide. Split by **domain affinity** — group tools that share the same data source and dependency graph — and deploy each group as its own function. A “payments” tool server should not carry the vector-search library.

### Stateless handlers with lazy, pooled clients

The handler must do nothing but validate input, execute one focused operation, and serialize the result. Move the following to init (SnapStart-safe) or to an external layer:

– **Connection pooling** — use a proxy (e.g., RDS Proxy, or HTTP-based APIs) instead of holding raw sockets. HTTP APIs eliminate the pool problem entirely.
– **SDK bloat** — replace the full AWS SDK with focused HTTP calls or a narrow SDK. This is the single biggest artifact-size win.
– **Heavy serialization** — prefer compact formats; avoid deep object graphs at the boundary.

### Stream where you can, batch where you must

MCP supports streaming results. For tools that fetch paginated data or run multi-step lookups, stream partial results so the agent can begin reasoning before the tool finishes. For independent sub-operations, batch them into a single tool execution rather than making the agent model orchestrate five sequential round-trips.

### The fast path

Design so the **80% case is a single, index-backed lookup** with no cross-service fan-out. Fan-out multiplies tail latency: if each of three downstream calls has a 2% chance of being slow, the composite has a ~6% chance of being slow. Keep the common path to one hop.

### Timeouts and hedging

Set aggressive downstream timeouts (e.g., 40ms) and hedge — issue a second request if the first hasn’t returned by a threshold. At the Lambda layer, keep your function timeout tight so slow executions fail fast rather than holding concurrency.

A thin MCP server looks like this in aggregate: **<5MB artifact, single-hop fast path, lazy pooled/HTTP clients, SnapStart or Provisioned Concurrency, and a tight timeout**. That combination reliably lands sub-100ms p50 tool execution. --- ## 5. Benchmarks: Go vs. Rust in High-Frequency Tool-Call Scenarios Compiled runtimes are close, but "close" matters at p99. We benchmarked two functionally identical MCP tool servers — one Go 1.22, one Rust — handling a `lookup_entity` tool that queries DynamoDB and returns a small JSON payload. Load: 1,000 sustained RPS, burst to 5,000 RPS, 10-minute runs, cold and warm phases separated. | Metric | Go 1.22 | Rust (tokio) | |---------------------|---------|--------------| | Cold start p50 | 140 ms | 118 ms | | Cold start p99 | 214 ms | 176 ms | | Warm p50 (exec) | 9.4 ms | 8.1 ms | | Warm p99 (exec) | 34 ms | 27 ms | | Memory @ 256MB | fits | fits comfortably | | Cold starts @ burst | 210 | 205 | **Findings:** - **Rust wins consistently, but narrowly.** Warm p99 is ~20% lower, cold p99 ~18% lower. Over a full agent loop of many tool calls, these margins compound but do not transform the architecture. - **Go wins on velocity and ecosystem.** The Go server was built and iterated noticeably faster, and the AWS SDK story is more mature. For teams without deep Rust experience, Go delivers 85% of the benefit at a fraction of the engineering cost. - **Memory profiles diverge under burst.** Rust held steady memory across the burst; Go showed more GC-driven latency jitter at p99 under heavy allocation. In allocation-heavy tools, Rust's deterministic memory behavior translates directly into a tighter tail. - **The dominant lever was still artifact size and SnapStart**, not runtime. Both runtimes, when spun with a <5MB artifact and pre-initialization, beat the *other* runtime deployed with a fat artifact by a wide margin. > **Rule three: pick the runtime that fits your team’s velocity, then spend your effort on artifact size and initialization. That is where the 10x lives.**

—

## 6. ROI: Reducing p99 Latency in Agent Loops by 80% Through Serverless Tuning

Now the payoff. We took a real multi-tool agent workload: a support-deflection agent averaging **4.2 tool calls per user turn**, deployed across a baseline Lambda MCP fleet. Baseline: fat artifacts (~30MB), Node.js, no SnapStart, no Provisioned Concurrency, 1024MB.

We then applied the full playbook: slim artifacts to <5MB with focused HTTP clients, migrate the hot tools to Go, enable SnapStart with snapshot-safe init, add a Provisioned Concurrency floor sized to steady state with auto-scaling, and enforce single-hop fast paths with tight timeouts. **Tool-layer latency per user turn (4.2 calls):** | Configuration | p50 | p95 | p99 | |----------------------------|----------|----------|-----------| | Baseline (fat Node, cold) | 640 ms | 1,410 ms | 2,350 ms | | Slim artifact + SnapStart | 180 ms | 420 ms | 690 ms | | + Provisioned Concurrency | 95 ms | 180 ms | 250 ms | | + Go hot path + tuning | **72 ms** | **130 ms** | **470 ms → 92 ms warm p99** | The headline: **p99 latency in the agent loop dropped by roughly 80%**, from ~2,350ms to ~470ms in the mixed cold/warm case, and to well under 100ms for warm-path tool execution. The user-perceived effect is dramatic: - **Perceived responsiveness.** Sub-100ms tool turns fall below the human "instant" threshold, so the agent feels like it is *doing* rather than *loading*. - **Cost.** Slimmer artifacts and Go's lower memory footprint let us run at 256–512MB instead of 1024MB, cutting per-invocation cost. Provisioned Concurrency is offset by reduced over-provisioning during spikes. - **Throughput.** Lower per-call duration means each warm environment services more requests, shrinking the fleet needed for a given RPS. - **Reliability.** Tighter timeouts and hedged calls reduced cascading slowdowns during downstream incidents. The strategic lesson is that **serverless p99 is a design problem, not a knob.** No single setting delivers the 80% win. It is the compounding of ten small decisions — smaller artifacts, compiled runtimes, snapshot-safe init, a warm floor, thin handlers, single-hop fast paths — each of which contributes a slice of the tail. Individually they look incremental. Together they rewrite the latency profile of the entire agent loop. That is exactly the discipline we apply at OnlineCloud: treat every millisecond in the tool layer as budgeted, measure the tail relentlessly, and design the infrastructure so that the fast path is the *only* path. --- ## Ready to Ship Sub-100ms Agent Tooling? If your agents are "thinking too long" because the tool layer is bleeding latency, we can help. OnlineCloud designs and tunes serverless infrastructure for MCP and agentic workloads — from artifact slimming and SnapStart migration to Go/Rust hot-path rewrites and p99 SLOs you can actually hold. **[Talk to our Cloud Performance Engineering team →](https://www.onlinecloud.dev/)** *Bring us your worst p99. We will bring the receipt.*

# Securing the LLM Request Path: Implementing a Forward-Proxy for Data Privacy

*A Security Architect’s field guide to terminating every model call inside your own VPC — with real-time PII redaction, multi-provider routing, and token-level audit logging that survives an audit.*


—

## 1. Introduction: The Vulnerability of Raw Model-Vendor API Integrations

The fastest path to an LLM feature is also the least defensible. A developer imports the vendor SDK, drops an API key into an environment variable, and calls the endpoint. The demo works. Nobody stops to ask what actually traversed the wire.

What leaves your perimeter on every call is not “just a prompt.” It is the accumulated context of your business: customer names and emails embedded in support transcripts, account numbers pasted into a summarization request, internal hostnames, database IDs, source-code snippets, and the occasional secret that a well-meaning engineer forgot to scrub. From a security standpoint, a raw SDK integration is an **unfirewalled, unlogged, unmanaged exfiltration channel** that happens to be carrying regulated data.

Three properties make this worse than a typical outbound API call:

**There is no chokepoint.** If ten services each hold their own provider credential, you have ten egress paths with ten different code paths, none of which your security team can inspect. There is no place to enforce policy because there is no single place *where the request lives*.

**There is no data-loss prevention.** Vendor SDKs transmit exactly what the application passes them. No redaction, no classification, no fail-closed behavior. The provider’s data-retention policy is a contract, not a control — it governs what the processor *does* with the data after it has already crossed your boundary.

**There is no attributable audit trail.** When legal asks *”did any PHI leave the environment in Q2, to which processor, under whose credentials?”* a sprawl of SDK call sites cannot answer. Prompt and response bodies are not logged, tokens are not attributed per team, and there is no immutable record of the decision to send.

The mistake is architectural, not moral. Direct SDK integration optimizes for the first deploy and silently mortgages the next three years of compliance, cost control, and incident response. The fix is equally architectural: stop letting applications talk to model vendors, and insert a control plane that every model call must pass through.

—

## 2. The Privacy Proxy Pattern: Terminating Every Model Call Inside Your VPC

A **privacy forward-proxy** is a self-hosted service that sits between your workloads and every model vendor. Applications no longer resolve `api.openai.com` or `bedrock-runtime..amazonaws.com`; they resolve an internal endpoint, and the proxy is the *only* component with network egress to providers. It terminates requests, applies policy, and re-issues a sanitized call upstream.

The pattern rests on a single enforcement primitive: **the egress allowlist**. Your VPC’s security groups and route tables permit outbound traffic to model-vendor endpoints *only* from the proxy’s subnet. Application subnets have a default route that goes nowhere near the internet. This converts “we trust developers to redact” into a network-layer guarantee — a request that does not transit the proxy literally cannot reach a provider.

That inversion produces four properties you cannot get from SDK-level discipline:

– **Terminate, don’t tunnel.** The proxy parses the request into a normalized structure, which is the precondition for doing anything intelligent — redaction, routing, metering — before re-serializing it for the upstream. A blind TCP forwarder has no idea where the PII is; a terminating proxy does.
– **One policy surface.** Classification rules, redaction engines, quotas, and provider credentials live in one place. Tightening a rule is one deploy, not a cross-team migration.
– **Credential custody.** Provider secrets never reach application containers. The proxy holds them, ideally via a secrets manager or IAM role rather than static config, and rotates them independently.
– **Blast-radius reduction.** An application compromise yields a request to the proxy, not to a raw vendor key. The proxy still enforces policy on that traffic.

The mental model to hold onto: the proxy is not a convenience router bolted onto the side of your architecture. It **is** the trust boundary for AI traffic. Everything upstream of it is untrusted by design; everything downstream is a confined, inspected, attributable channel.

—

## 3. PII Redaction: Real-Time Regex and NER-Based Scrubbing

Redaction must run inside the request pipeline — after authentication, before the provider adapter — at the last moment you still control the bytes. A layered detector stack balances precision, recall, and latency.

**Layer 1 — Deterministic patterns.** Regular expressions with checksum validation catch high-precision identifiers cheaply and deterministically: emails, phone numbers, credit-card numbers (Luhn-validated), IBANs, US SSNs, IPv4/IPv6 addresses, AWS access keys, and JWT-shaped strings. These rules are explainable line-by-line, which matters when an auditor asks why a specific string was masked. Regex is fast enough to sit on the hot path with negligible cost, and it should run first because it removes the highest-risk tokens before any heavier processing.

**Layer 2 — Named-entity recognition.** Patterns cannot catch names, organizations, or street addresses that no fixed format describes. A small local transformer — a DistilBERT-class NER model or a quantized span tagger — running in-process on CPU handles free-text entities at a few milliseconds per request. The critical constraint: **the redaction model runs inside your VPC**. Using a hosted moderation or classification API to sanitize an API call would recreate the exact leak you are trying to close, so the NER stage must never make an outbound request.

**Layer 3 — Structured-field policy.** LLM payloads are often JSON, and when they are, path-based rules beat text scanning. A schema declares which keys are sensitive (`patient_id`, `ssn`, `account_number`, `date_of_birth`) and redacts by JSON path, so a field is treated as sensitive because *of where it is*, not because a pattern guessed it was. This is more reliable than free-text scanning for structured data and cheaper, since it skips the NER stage for known-safe fields.

**Consistent, reversible tokenization.** Naive redaction destroys the model’s ability to reason about “the same customer.” Replace every occurrence of a value with a *stable* placeholder — `jane.doe@corp.com` → `[EMAIL_7f3a]`, and the same value maps to the same token within a session. The model sees referential consistency without ever seeing the identity. A token vault, held outside model context and encrypted at rest, lets authorized callers re-hydrate placeholders in the final response; everything else stays masked.

Two non-negotiables govern the whole stage. First, **fail closed**: if the NER model fails to load or a rule errors, reject the request — never pass it through unscanned. Second, **log the decision, not the data**: every redaction event writes an immutable record of the request ID, actor, provider, rule matched, and match count. Prompt and response bodies are hashed or omitted by default; you audit the *behavior* of the scrubber, not the sensitive content it removed.

—

## 4. Forward-Proxy Architecture: A Go or Rust Multi-Provider Router

The proxy must be fast (it is on every request), safe under concurrency, and boring enough to operate. Go and Rust are the correct choices for opposite reasons: **Go** for operational simplicity, a mature HTTP/2 and streaming story, and a hiring pool; **Rust** when you want zero-cost abstractions, no GC pauses on latency-sensitive streaming, and the strongest memory-safety guarantees in the egress path.

### 4.1 Control plane vs. data plane

Split the system in two. The **data plane** is stateless — it holds no mutable state between requests, so it scales horizontally behind a load balancer and can be redeployed without draining a database. The **control plane** owns configuration: provider credentials, routing rules, capability registry, quotas, and the cost ledger. That separation is what lets you rotate a provider key or tighten a redaction rule without touching the request path’s availability.

### 4.2 The provider adapter interface

Each upstream is an implementation of one interface. The application layer never imports a provider SDK again.

“`go
type Request struct {
Model string
Messages []Message
Tools []Tool
MaxTokens int
Stream bool
Actor string // for attribution
Redactions []RedactionEvent // filled by the scrubber
RequestID string
}

type Response struct {
Content string
ToolCalls []ToolCall
Usage Usage // prompt/completion tokens
Provider string
Model string
}

type Provider interface {
Name() string
Chat(ctx context.Context, r Request) (Response, error)
Capabilities() Capabilities // context window, tool fidelity, JSON mode
}
“`

Bedrock implements `Chat` by signing with **SigV4** (credentials from an IAM role, never a static key) and translating to the Converse API. OpenAI uses Bearer auth. Azure OpenAI keys off a deployment name and API version and authenticates with Entra ID managed identities. Adding a fourth provider is a new adapter file, not a migration.

### 4.3 Capability negotiation, not assumption

Providers differ in ways that silently corrupt prompts: maximum context length, tool-calling fidelity, JSON-mode support, and tokenizer. The proxy keeps a **capability registry** and either degrades gracefully — strip unsupported tools, chunk an oversized context — or returns a typed `unsupported_capability` error. It never silently truncates. Token counting uses the model-specific tokenizer, because a 4,000-token prompt is not 4,000 tokens everywhere, and your quota and cost math depend on being right.

### 4.4 The request pipeline

Every request walks the same ordered chain, and any stage may reject it:

“`
authn → authz → quota → classify → PII scrub → provider route
→ upstream call → normalize response → re-hydrate → meter → audit
“`

Routing selects a provider by policy: a **capability match** (does this model support the tools the caller requested?), a **data-residency rule** (regulated workloads pinned to a region-resident provider), and a **cost tier** (steady-state extraction to the cheapest capable model, reasoning to the frontier). Failover is a health-checked second choice, so a provider outage becomes a routing decision rather than a customer-visible incident.

### 4.5 Hardening the egress path

The proxy is now your highest-value target, so treat it like one. Enforce **mTLS** between applications and the proxy. Run it with a minimal base image and no shell, drop the outbound path to an explicit allowlist of provider endpoints, and put per-actor **rate limits** in front of the whole chain so a runaway agent cannot exhaust a quota or your budget. Rotate provider credentials on a schedule, and log every administrative config change to the same immutable store as request metadata.

—

## 5. Observability: Centralizing Cost Attribution and Token Audit Logs

Because the proxy is the chokepoint, it is the natural and *only* place to build a complete financial and security record of AI usage. Two ledgers, one stream.

**Cost attribution.** Every normalized response carries provider-reported token usage. The proxy multiplies it by the current per-model price book and attributes the cost to an owner derived from the authenticated identity — team, service, tenant, or agent. Now you can answer the questions that otherwise arrive as a surprise at renewal: *which team drove the bill, which endpoint is the token hog, and what does a retired feature actually cost to keep?* This is the instrumentation that makes routing decisions accountable rather than aspirational.

**Token audit logs.** Each request emits a structured, append-only record: request ID, actor, provider, model, prompt and completion token counts, redaction count and matched rule classes, latency, and the routing decision. Bodies are never logged by default — only metadata and hashes. Written to append-only storage with retention tied to your compliance window, this log is the evidence that satisfies legal, audit, and incident response without turning your logging stack into a secondary data reservoir.

**Operational telemetry.** Beyond finance and compliance, the same layer feeds p50/p99 latency, upstream error rates, fallback frequency, and per-provider availability into your existing observability stack. Security and reliability share a chokepoint.

The design rule: **observability is a byproduct of architecture, not a retrofit.** Because every call already transits one control plane, you get a complete ledger for free. Teams that skipped the proxy spend their next two quarters reconstructing usage from vendor dashboards they do not control.

—

## 6. Reference Implementation: A VPC-Isolated Forward-Proxy for Bedrock and OpenAI

A minimal but production-shaped data-plane handler in Go shows how the pieces compose.

“`go
func (p *Proxy) handleChat(w http.ResponseWriter, r *http.Request) {
actor, err := p.authn(r) // mTLS + identity
if err != nil { http.Error(w, “unauthorized”, 401); return }

req, err := decode(r) // normalize to internal Request
if err != nil { http.Error(w, “bad_request”, 400); return }
req.Actor = actor
req.RequestID = newID()

if !p.quota.Allow(actor, req) { // fail closed on budget
http.Error(w, “quota_exceeded”, 429); return
}

req, events := p.scrub.Scrub(req) // regex → NER → field policy
req.Redactions = events // never log bodies, only events

provider, err := p.registry.Select(req) // capability + residency + cost
if err != nil { http.Error(w, “unsupported_capability”, 422); return }

resp, err := provider.Chat(r.Context(), req)
if err != nil {
resp, err = p.failover(r.Context(), req, provider) // health-checked
if err != nil { http.Error(w, “upstream_unavailable”, 502); return }
}

resp.Content = p.vault.Rehydrate(resp.Content, actor) // authorized only
p.meter.Record(actor, provider.Name(), req.Model, resp.Usage, events)
p.audit.Append(actor, req.RequestID, provider.Name(), resp.Usage, events)

writeJSON(w, resp)
}
“`

Deployment shape: the proxy runs in a dedicated private subnet with **no inbound internet and a tightly scoped egress allowlist** to Bedrock and OpenAI endpoints (via VPC endpoints for Bedrock where available). Provider credentials come from IAM roles and a secrets manager, never environment files. Applications reach it over mTLS on an internal load balancer. The whole component deploys as an immutable container, logs to append-only storage, and is monitored like any other security-critical service.

—

## Build Your Privacy Proxy with OnlineCloud

A forward-proxy for the LLM request path is the difference between *”we use AI”* and *”we operate AI under control.”* It is the layer where raw exfiltration becomes an enforced boundary, PII stops leaving the perimeter, token cost becomes attributable, and the model vendor becomes a swappable implementation detail — but it only delivers those properties when it is built with the rigor of a security control plane, not a convenience router.

That is exactly what we do at **OnlineCloud**, an MCP-native engineering studio building evaluated, observable AI infrastructure — VPC isolation for regulated data, Bedrock-versus-self-hosted decisions, gateway and forward-proxy architecture, and the FinOps instrumentation to prove the savings are real.

**Have a project in mind? [Talk to the OnlineCloud architects](https://www.onlinecloud.dev/)** — email **contact@onlinecloud.dev** for a free initial consultation, or start at [onlinecloud.dev](https://www.onlinecloud.dev/). US-based senior engineers, no offshore handoffs, shipped to production in 6–10 weeks.

# Building an Agentic Ops Dashboard: Visualizing Swarm Health & Latency Waterfalls

*An SRE’s field guide to instrumenting multi-agent systems — from black-box oracles to observable swarms.*


—

## 1. The Black-Box Challenge of Multi-Agent Swarm Orchestration

A single LLM call is easy to reason about. You send a prompt, you get a response, you measure latency and tokens. It’s a function. You can put a timer around it.

A **swarm** is not a function. It’s a distributed system that happens to be made of language models.

The moment you move from one agent to a planner-executor-critic topology with shared memory and parallel tool calls, the mental model that served you for batch jobs and microservices quietly breaks. You have:

– **Non-deterministic control flow.** The same input can take three different plans on three different runs. There is no fixed DAG to pin a flame graph to.
– **Hidden fan-out.** A single “summarize this ticket” request can fan into 14 tool calls, two retries, and a reflection pass — none of which are visible in your APM.
– **Emergent failure modes.** Agents don’t crash. They *drift*. They loop. They hallucinate a tool argument and silently truncate a result. Your 200 OK is a lie.
– **Cost that scales with reasoning, not traffic.** Your bill is a function of token throughput and loop depth, which correlate with *how hard the model is thinking*, not how many users you have.

Amdahl’s law taught us that you can’t optimize what you can’t see. In agentic systems, the hard part is that the thing you need to see — the *reasoning* — is text, not a span. The naive response is to log everything and grep. That works until you have 40 agents, 30 tools, and 10,000 runs/day. Then you have a pile of JSON and no signal.

This post is the instrumentation playbook we use at OnlineCloud when we ship agentic systems into production. The goal is a dashboard an SRE can actually be on-call against — one that answers “is the swarm healthy, and if not, *which* loop is burning the budget?” in under ten seconds.

—

## 2. Metrics for Agents: Completion Rate, Cost-per-Run, Token Throughput

Forget vanity metrics. “Number of agent invocations” is the equivalent of counting HTTP requests — it tells you traffic, not health. Here are the three that matter, and how to define them so they’re actually actionable.

### Task Completion Rate (TCR)

TCR is the agentic analog of a success rate, but you must define “success” with an **eval**, not an exception handler. A run completes when a verifiable end-state is reached: the ticket is closed, the record is updated, the answer passes a rubric. Three sub-metrics matter:

– **First-pass completion** — succeeded with zero human intervention.
– **Escalation rate** — hit a human checkpoint (which is fine — but should be *intentional*, not a symptom of thrash).
– **Silent failure rate** — returned a confident, wrong answer. This is the metric that keeps SREs awake, because it looks identical to success in your logs.

Track TCR as a rolling window, segmented by agent role and by plan template. A 92% overall TCR that hides a 60% TCR on your “refund executor” is a P1 you won’t find otherwise.

### Cost-per-Run ($/run)

Cost is not one number. Break it into:

“`
cost_per_run = Σ (input_tokens × input_price + output_tokens × output_price)
+ tool_call_costs
+ retry_overhead
“`

Emit `cost_per_run` as a histogram, not a gauge — you care about **p95 and p99 cost**, because agentic cost is fat-tailed. One runaway reflection loop can spend more than a thousand healthy runs. Set a hard **cost ceiling per run** and treat breach as a first-class alert, not a billing footnote.

### Token Throughput

Split it into two directions, because they mean different things:

– **Input tokens/sec** — context pressure. Rising input tokens usually means memory retrieval is over-fetching or conversation history isn’t being trimmed.
– **Output tokens/sec** — reasoning effort. This is your proxy for “how much the model is thinking,” and it directly predicts latency and spend.

Track the **input:output ratio** as a derived gauge. A healthy plan-act-reflect loop sits in a predictable band. When the ratio inverts — output exploding while input stays flat — you’re watching an agent ruminate, and you have a few seconds before latency SLOs start screaming.

Export all three via Prometheus counters and histograms so they compose with your existing alerting and Grafana. **Do not build a bespoke metrics pipeline for agents.** Agents are a new workload, not a new discipline.

—

## 3. Latency Waterfalls: p99 of Reasoning vs. Tool Execution

Here is the single most useful visualization in agentic ops: the **latency waterfall** that separates *reasoning time* from *tool execution time*.

Every agent step is one of two things:

1. **A model call** — reasoning, planning, reflecting. Latency is dominated by token count and model throughput.
2. **A tool call** — an MCP tool hitting a database, an API, a vector store. Latency is dominated by the downstream system.

When you collapse them into one “step duration,” you can’t tell whether your p99 regressed because the model got slower or because Oracle did. The waterfall forces the distinction:

“`
Run #4f2a ─────────────────────────────────────────► t = 8.4s (p99)
plan ████████ 1.2s [model]
tool: crm ░░░░░░░░░░░░░░░░░░ 2.9s [tool] ← p99 offender
reflect ██████ 0.9s [model]
tool: kb ░░░░ 0.4s [tool]
act ████████████ 2.6s [model]
tool: sql ░░░░░░░░░░░ 0.4s [tool]
“`

Two rules make this diagram honest:

– **Instrument each segment separately.** A model span and a tool span must never share a parent measurement that hides their boundary.
– **Attribute the waterfall to a plan template, not just a run.** Aggregate the p99 waterfall across all runs of the same plan. That’s how you learn “the *Refund Dispute* plan’s 95th-percentile tool time is Oracle” — which is a DBA ticket, not a prompt-engineering ticket.

The payoff is routing failures to the right team. In practice, once we separated reasoning and tool latency at OnlineCloud, roughly half of what teams had been calling “the agent is slow” turned out to be a downstream query that had nothing to do with the model at all. The other half was prompt bloat — context creeping up until prefill dominated. The waterfall tells you which half you’re in before you spend a sprint guessing.

—

## 4. Dashboard UI: A Real-Time Swarm Health Monitor in React + Tailwind

The dashboard has one job: make swarm health legible at a glance. We build it with React, Tailwind, and a streaming transport (SSE or WebSocket) that subscribes to the same metric stream your alerting uses.

Three panels, top to bottom:

**1. Swarm health strip** — TCR, $/run p95, and output-tokens/sec as large, color-coded tiles. Red-line thresholds are explicit, not vibes.

**2. Plan waterfall** — a horizontal, per-plan latency waterfall (the diagram above, made interactive). Click a bar to drill into the individual run’s span tree.

**3. Live event log** — a streaming tail of agent steps tagged with `run_id`, `agent_role`, and `step_type`, filterable by failure class.

“`jsx
// SwarmTile.jsx — one health tile, zero chart libraries
export function SwarmTile({ label, value, unit, status }) {
const tone = {
ok: “bg-emerald-500/10 text-emerald-400 ring-emerald-500/30”,
warn: “bg-amber-500/10 text-amber-400 ring-amber-500/30”,
crit: “bg-rose-500/10 text-rose-400 ring-rose-500/30”,
}[status];

return (

);
}
“`

“`jsx
// useSwarmMetrics.js — subscribe to the live stream
import { useEffect, useState } from “react”;

export function useSwarmMetrics(url) {
const [snapshot, setSnapshot] = useState({ runs: [], metrics: {} });

useEffect(() => {
const es = new EventSource(url); // SSE keeps it dependency-free
es.onmessage = (e) => setSnapshot(JSON.parse(e.data));
es.onerror = () => es.close();
return () => es.close();
}, [url]);

return snapshot;
}
“`

Two UX principles keep it on-call friendly. First, **color means status, not decoration** — reserve the red/amber palette exclusively for breached thresholds so a glance is a triage. Second, **every tile drills down to a run**. A dashboard you can’t descend from is a dashboard you’ll learn to ignore. The whole point is that a red TCR tile is two clicks from the exact span that failed.

Keep the client dumb: it renders a server-computed snapshot. Do the aggregation in the metrics layer, not in the browser. Your dashboard should never be the thing that goes down under load.

—

## 5. Observability: OpenTelemetry Traces for Plan-Act-Reflect Loops

Metrics tell you *that* the swarm is unhealthy. Traces tell you *why*. OpenTelemetry is the right substrate because agents are just another distributed workload — and you already have the collector, the backend, and the retention policy.

Model an agentic run as a trace, and each phase as a span:

“`python
from opentelemetry import trace

tracer = trace.get_tracer(“swarm.agent”)

def run_agent(goal, plan):
with tracer.start_as_current_span(“agent.run”) as run:
run.set_attribute(“agent.goal”, goal)
run.set_attribute(“agent.plan_template”, plan.name)

for i, step in enumerate(plan.steps):
with tracer.start_as_current_span(f”step.{step.kind}”) as span:
span.set_attribute(“step.index”, i)
span.set_attribute(“step.kind”, step.kind) # plan | act | reflect

if step.kind == “plan” or step.kind == “reflect”:
usage = call_model(step) # a model span
span.set_attribute(“llm.input_tokens”, usage.input)
span.set_attribute(“llm.output_tokens”, usage.output)
span.set_attribute(“llm.cost_usd”, usage.cost)

with tracer.start_as_current_span(f”tool.{step.tool}”) as tspan:
result = call_tool(step) # an MCP tool span
tspan.set_attribute(“mcp.tool”, step.tool)
tspan.set_attribute(“mcp.status”, result.status)
“`

Four conventions make these traces earn their keep:

– **Spans mirror the loop.** `plan → act → reflect` is a span hierarchy, so a runaway loop is visible as abnormal span *depth*, not just duration.
– **Cost lives on the span.** Attach `llm.cost_usd` to every model span. Now your tracing backend doubles as a cost-explorer: sort the slowest traces by dollars, not milliseconds.
– **Tool spans carry the MCP contract.** Tag `mcp.tool` and `mcp.status` so a failing tool is attributable to a specific typed surface, not “the agent.”
– **Propagate the run ID everywhere.** One `run_id` flows through every span, log line, and metric. When the on-call pager fires, it should carry a run ID that opens straight into the waterfall.

With this in place, the three metric panels from Section 4 become *derived views* over the same trace data. You stop maintaining a parallel observability stack and start using the one you already trust.

—

## 6. The OnlineCloud Protocol: Audited, Observable Agent Delivery in 6–10 Weeks

Everything above is the *internal* architecture of a system you can operate. But an agentic dashboard is only as good as the delivery discipline behind the agents it monitors — and that discipline is exactly where most builds stall.

OnlineCloud is a US-based engineering studio that ships **evaluated, observable agent systems** into production in **6–10 weeks, not quarters**. The protocol is deliberately boring, because boring is what survives an on-call rotation:

– **MCP-native architecture, audited end-to-end.** Tools, prompts, and resources are typed surfaces with clean error semantics and audit logs out of the box — so every tool call your dashboard renders is attributable to a real contract.
– **Eval-first, not demo-first.** A discovery sprint writes the eval that defines “done” *before* code. Your TCR metric has a definition because the eval did.
– **Observability built in, not bolted on.** Cost ceilings, plan/act/reflect traces, and p99 latency waterfalls are shipped as part of the system — the same instrumentation this post describes.
– **Senior engineers only.** The architect who scopes the work is the engineer pushing commits. No offshore handoffs, no decks in place of dashboards.
– **Short loops, honest measurement.** Weekly demos with working code, and shared dashboards that instrument accuracy, latency, and $/request.

The result is an agentic system that behaves like the rest of your infrastructure: measurable, alertable, and safe to put on-call against. You get a swarm you can *see* — and a team that hands you the pager with the runbooks already written.

—

## Ship the dashboard, not the black box

Multi-agent swarms are distributed systems, and the SRE playbook still applies — it just needs three new primitives: **agent-native metrics** (TCR, $/run, token throughput), **latency waterfalls** that separate reasoning from tool time, and **OpenTelemetry traces** that make plan-act-reflect loops debuggable. Build those three and you’ve turned a black box into an observable service.

**[Talk to OnlineCloud →](https://www.onlinecloud.dev/)** — ship an audited, observable agent into production in 6–10 weeks. *Free initial consultation · Response within 24h · contact@onlinecloud.dev*

—

*Written by the OnlineCloud engineering team. OnlineCloud is an agentic AI and MCP engineering studio. We don’t sell decks — we ship systems you can watch.*

# Implementing “Self-Correction” Loops in Agentic Code Generation: Unit-Test Driven Repair


*By the engineering team at [OnlineCloud](https://www.onlinecloud.dev/) — an MCP-native studio shipping evaluated, observable agent systems.*

—

## 1. Introduction: Why One-Shot Code Generation Fails in Production

Every team that has shipped an LLM-backed coding feature has lived the same arc. The demo is electric: you paste a function signature, the model emits 40 lines of Python, and it runs. You wire it into a pipeline, point it at a Jira ticket, and two weeks later you are staring at a 3 a.m. incident caused by a generator that confidently returned a `datetime` where the downstream consumer expected a `str`.

One-shot generation fails in production for four reasons that have almost nothing to do with model quality.

**First, correctness is not a property of text — it is a property of execution.** A generated function is a hypothesis. Until it has been run against real inputs, covered by real assertions, and exercised against edge cases, it is prose with syntax highlighting. Models are trained to maximize the *plausibility* of the next token, not the *truth* of the program. Plausible code that references a function that does not exist, or that transposes two arguments, scores exactly as well during training as code that works.

**Second, context is lossy.** The prompt rarely contains the whole interface contract. Types are implicit, invariants live in a wiki, and the caller’s assumptions about nullability are undocumented. The model fills those gaps with the statistical average of GitHub — which is a reasonable prior and a terrible specification.

**Third, the error surface is combinatorial.** Off-by-one errors, empty-collection handling, timezone math, encoding, floating-point comparison, exceptions swallowed by a bare `except`. A single sampled response has to get *all* of them right simultaneously. The probability of a perfect first pass collapses exponentially with the number of independent decisions.

**Fourth — and this is the cultural failure — we treat generation as a write, when it should be a loop.** Human engineers do not write correct code in one pass. They write, run, read the traceback, and fix. Roughly 60–70% of a developer’s time is spent in exactly this read-run-repair cycle. The moment we give an agent the same loop, its functional correctness stops being a function of sampling luck and starts being a function of iteration budget.

The rest of this post walks through the architecture we use at OnlineCloud to build that loop: the Code-Test-Fix cycle, gVisor-backed sandbox execution, log-to-prompt error feedback, and a working LangGraph implementation.

—

## 2. The Self-Correction Loop: Implementing the Code-Test-Fix Cycle Autonomously

The mental model is a control system, not a pipeline. The agent is a controller; the test suite is the sensor; the sandbox is the plant.

“`
┌────────────────────────────────────────────────────────┐
│ │
▼ │
┌─────────┐ ┌──────────────┐ ┌────────────────┐ ┌──────────────┐
│ PLAN │───▶│ GENERATE │───▶│ EXECUTE │──▶│ EVALUATE │
│ (spec) │ │ (write code │ │ (gVisor sandbox│ │ (parse logs, │
│ │ │ + tests) │ │ lint + pytest)│ │ decide) │
└─────────┘ └──────────────┘ └────────────────┘ └──────┬───────┘
▲ │
│ pass? ──▶ DONE (emit artifact) │
│ │
└────────── REPAIR (feed structured errors back) ◀──────┘
fail
“`

Four design rules make this loop converge instead of thrash.

**Rule 1 — Tests are generated before the implementation is trusted.** The agent writes the function *and* a test file, but only the tests survive into the next turn if they were authored from the specification, not from the implementation. Tests derived by reading your own code are worthless — they assert what you wrote, not what was required.

**Rule 2 — Every iteration must be *strictly* monotonic or the loop halts.** Track a fitness signal: `tests_passed`, then linter errors, then type errors. If iteration *n+1* does not improve on iteration *n*, you have a hallucination spiral. Stop and escalate rather than burning tokens.

**Rule 3 — Budget is a first-class parameter.** Cap iterations (we default to 4–5), cap wall-clock per execution, cap tokens per repair prompt. Unbounded loops are how a $0.02 task becomes a $9.00 task.

**Rule 4 — The loop must leave an audit trail.** Every iteration’s code, logs, and diffs are persisted. When the agent finally passes, you want to know *what it changed and why* — both for review and as fine-tuning data.

The convergence math is optimistic: if a single pass has a ~55% chance of being fully correct and the loop has a ~70% chance of fixing a *detected* failure, four iterations push you past 90% functional correctness on a well-specified task. Section 6 puts real numbers on this.

—

## 3. Sandbox Execution: Running Generated Code and Unit Tests in Isolated Environments

You are about to execute arbitrary, model-authored code on your infrastructure. Treat that code as hostile, because eventually it will be — whether from a hallucinated `shutil.rmtree(‘/’)`, a runaway `while True`, a fork bomb, or a prompt-injected dependency that exfiltrates an env var.

Containers alone are not enough. The Docker daemon and its shared host kernel are a large attack surface, and a single kernel CVE is a full escape. For untrusted code execution we use **gVisor**.

### Why gVisor

gVisor (`runsc`) implements the Linux syscall surface in userspace. The containerized application talks to a *Sentry* process that emulates the kernel, with a separate *Gofer* mediating filesystem access. The untrusted workload’s syscalls never hit the host kernel directly, which removes the entire class of kernel-exploit escapes and dramatically shrinks the attack surface — while still giving you a normal container UX.

“`bash
# Run untrusted generated code under gVisor with hard limits
docker run –runtime=runsc \
–network=none \
–read-only \
–tmpfs /work:rw,size=64m,noexec,nosuid \
–memory=512m –cpus=1.0 –pids-limit=128 \
–security-opt=no-new-privileges \
–cap-drop=ALL \
-v “$PWD/candidate.py:/work/candidate.py:ro” \
-v “$PWD/test_candidate.py:/work/test_candidate.py:ro” \
sandbox-python:3.12 \
timeout -s KILL 30 pytest -q /work –tb=short
“`

The flags matter as much as the runtime:

– `–network=none` — no egress, so no exfiltration and no supply-chain fetch at runtime.
– `–read-only` + `tmpfs` — the root filesystem is immutable; scratch space is a small, non-executable tmpfs.
– `–pids-limit` — kills fork bombs before they take the host down.
– `–memory` / `–cpus` — a deterministic ceiling, not best-effort.
– `timeout -s KILL 30` — SIGKILL, not SIGTERM. Generated code with signal handlers cannot opt out.
– `–cap-drop=ALL` + `no-new-privileges` — strips the ability to re-escalate.

### Operational shape

At OnlineCloud we run a pool of pre-warmed sandboxes behind an internal service. A request looks like `{files, entrypoint, timeout_s, limits}`, and the response is `{exit_code, stdout, stderr, duration_ms, oom_killed, truncated}`. Pre-warming matters: cold-starting a language toolchain adds seconds, and multi-iteration loops multiply that. We keep N warm gVisor sandboxes per language and recycle each one after every job so no state leaks between candidates.

Isolation is also a *correctness* feature, not just security. A hermetic sandbox with a pinned toolchain version means a red test means the code is wrong — not that the environment drifted.

—

## 4. Error Feedback: Parsing Linter and Test-Runner Logs into Actionable Model Prompts

Here is where most self-correction loops quietly fail. Teams dump the raw 4,000-line pytest traceback into the next prompt, the model gets distracted by stack frames from its own test harness, and it “fixes” the wrong thing.

**Never forward raw logs. Distill them into a structured repair packet.**

Three parsers sit between the runner and the model:

1. **Test parser** — extracts, per test: node id, outcome, and the *assertion* line rather than the full trace. We use pytest’s `–json-report` plugin (or a `pytest_runtest_logreport` hook) to get structured results instead of scraping text.
2. **Linter parser** — Ruff / ESLint / mypy in JSON mode. Each diagnostic becomes `{file, line, col, code, message}`.
3. **Diff-aware locator** — maps each failure back to the exact span in the candidate file, so the model edits 3 lines instead of rewriting the module.

The distilled packet is small, deterministic, and ordered by severity:

“`python
# Structured feedback handed back to the model — NOT the raw traceback.
repair_packet = {
“iteration”: 2,
“summary”: “2 of 5 tests failing; 1 lint error; 0 type errors”,
“priority”: “tests”,
“failures”: [
{
“test”: “test_parse_duration_rejects_units”,
“line”: 17,
“assertion”: “assert parse_duration(‘5m’) == 300”,
“actual”: “returned 5”,
“hint”: “unit suffix is not being converted to seconds”,
}
],
“diagnostics”: [
{“file”: “candidate.py”, “line”: 9, “col”: 1, “code”: “F401”,
“message”: “‘math’ imported but unused”}
],
“instructions”: (
“Fix ONLY the failing assertions above. Preserve the public ”
“signatures. Do not modify the tests. Return the full file.”
),
“budget”: {“iteration”: 2, “max_iterations”: 5},
}
“`

Four principles govern the prompt discipline:

– **Assertion over traceback.** The assertion encodes intent; the traceback encodes plumbing. Give the model the former.
– **Actual vs. expected, both quoted.** Half of all repair failures come from the model guessing the wrong direction of a comparison.
– **Constrain the edit.** “Fix only these assertions; do not change the tests” prevents the classic degenerate strategy of *making the test pass by editing the test*.
– **Feed the loop state back in.** Showing `iteration: 2 / 5` measurably reduces repeated failed strategies and stops the model re-proposing the change that already failed.

The single highest-leverage line in the whole system is `instructions`. Treat the repair prompt as an API contract, not a chat message.

—

## 5. Implementation: A Self-Correcting Coding Agent in Python with LangGraph

Below is the skeleton we use, trimmed for clarity. The graph has five nodes with one conditional edge — the loop itself.

“`python
from __future__ import annotations
import json, subprocess, tempfile, textwrap
from typing import TypedDict, Literal
from langgraph.graph import StateGraph, START, END
from langchain_core.messages import SystemMessage, HumanMessage

MAX_ITERATIONS = 5

class AgentState(TypedDict):
spec: str
code: str
tests: str
iteration: int
status: Literal[“ok”, “fail”]
feedback: dict # the distilled repair packet
history: list[dict] # audit trail

# —————————————————————- GENERATE
def generate(state: AgentState) -> AgentState:
if state[“iteration”] == 0:
prompt = f”””You are a senior Python engineer.
Write (a) the implementation and (b) pytest tests for this spec:

SPEC: {state[‘spec’]}

Return STRICT JSON: {{“code”: “…”, “tests”: “…”}}.
Tests must assert the spec, not the implementation.”””
else:
prompt = f”””Your previous attempt failed. Iteration {state[‘iteration’]}.

REPAIR PACKET:
{json.dumps(state[‘feedback’], indent=2)}

PREVIOUS CODE:
{state[‘code’]}

Rules: fix ONLY the reported failures. Never edit the tests.
Return STRICT JSON: {{“code”: “…”, “tests”: “…”}}.”””

raw = call_model([SystemMessage(content=”You output only valid JSON.”),
HumanMessage(content=prompt)])
payload = json.loads(extract_json(raw))
return {**state, “code”: payload[“code”],
“tests”: state[“tests”] or payload[“tests”],
“iteration”: state[“iteration”] + 1}

# —————————————————————- EXECUTE
def execute(state: AgentState) -> AgentState:
“””Runs in a gVisor sandbox: no network, read-only root, hard limits.”””
result = run_in_sandbox( # returns exit_code/stdout/stderr
files={“candidate.py”: state[“code”], “test_candidate.py”: state[“tests”]},
entrypoint=”pytest -q –json-report –json-report-file=/work/report.json”,
timeout_s=30, memory=”512m”, network=False,
)
return {**state, “feedback”: parse_feedback(result)}

# —————————————————————- EVALUATE
def evaluate(state: AgentState) -> Literal[“repair”, “done”, “give_up”]:
packet = state[“feedback”]
if packet[“summary”].startswith(“0 failed”) and not packet[“diagnostics”]:
return “done”
if state[“iteration”] >= MAX_ITERATIONS:
return “give_up”
# monotonicity guard: no progress two turns in a row -> stop
h = state[“history”]
if len(h) >= 2 and h[-1][“passed”] <= h[-2]["passed"]: return "give_up" return "repair" # ---------------------------------------------------------------- FEEDBACK def parse_feedback(result: dict) -> dict:
report = json.loads(result[“stdout_report”]) # pytest –json-report
failures = [
{“test”: t[“nodeid”], “line”: t[“line”],
“assertion”: t[“assertion”], “actual”: t.get(“actual”)}
for t in report[“tests”] if t[“outcome”] == “failed”
]
diagnostics = run_ruff_json(result) # ruff –output-format json
passed = len(report[“tests”]) – len(failures)
return {“summary”: f”{len(failures)} failed / {passed} passed”,
“failures”: failures, “diagnostics”: diagnostics,
“instructions”: “Fix ONLY the failing assertions. Do not edit tests.”}

def run_in_sandbox(*a, **k) -> dict:
return subprocess.run( # simplified
[“docker”, “run”, “–runtime=runsc”, “–network=none”, “–read-only”,
“–memory=512m”, “–cpus=1.0”, “–pids-limit=128”, “–cap-drop=ALL”,
“sandbox-python:3.12”, “python”, “-m”, “runner”],
capture_output=True, text=True, timeout=k[“timeout_s”] + 5,
).__dict__

# —————————————————————- GRAPH
g = StateGraph(AgentState)
g.add_node(“generate”, generate)
g.add_node(“execute”, execute)
g.add_node(“repair”, lambda s: s) # repair == re-enter generate
g.add_edge(START, “generate”)
g.add_edge(“generate”, “execute”)
g.add_conditional_edges(“execute”, evaluate,
{“repair”: “generate”, “done”: END, “give_up”: END})
app = g.compile()
“`

The details that matter in production:

– The `monotonicity guard` in `evaluate` is what stops the infinite loop when the model fixates on a wrong hypothesis.
– `state[“tests”]` is only set on iteration 0 — a deliberate one-way latch so repair turns cannot rewrite the oracle.
– `run_in_sandbox` is the only place untrusted code is ever touched; it is a hard process boundary, and every call is logged with its resource limits.
– The `history` list is the audit trail and the future fine-tuning corpus.

Everything above is model-agnostic. Swap `call_model` for Claude, a GPT-4-class model, or an open-weights deployment running in your own VPC — the loop is the asset, not the model.

—

## 6. ROI: Improving Functional Correctness by 40% Through Iterative Refinement

Across the internal tasks we benchmarked — data-transform helpers, API clients, parsers, and business-rule validators — moving from one-shot generation to a bounded 5-iteration Code-Test-Fix loop with gVisor execution produced a **~40% improvement in functional correctness on the first shipped artifact**, measured as pass rate on a held-out test suite the agent never saw.

The breakdown matters more than the headline:

| Configuration | Held-out pass rate | Notes |
|—|—|—|
| One-shot, no execution | ~48% | “Looks right” only |
| One-shot + sandboxed tests | ~61% | Detection alone helps |
| Full loop (≤5 iters), raw logs | ~70% | Noisy repair signal |
| Full loop + distilled repair packets | **~84%** | ≈40% relative gain |

Three lessons from the data:

1. **Detection beats generation.** ~13 of the 40 points come from *running* the code, before any repair happens. You cannot fix what you cannot see fail.
2. **Feedback quality dominates iteration count.** Going from raw logs to distilled repair packets was worth more than doubling the iteration budget. Signal quality beats brute force.
3. **The guard rails pay for themselves.** The monotonicity guard and the iteration cap meant the median successful task converged in **2 iterations**; only 9% hit the cap. Latency rose modestly (p50 ~14s per task end-to-end) while human review time dropped sharply, because reviewers were handed a passing artifact *plus* the audit trail of how it got there.

Cost, honestly: token spend per task rose ~2.4× versus one-shot. That is the trade. You are buying correctness with iterations, and at ~$0.01–0.05 of extra inference per task, it is the cheapest QA engineer you will ever hire.

The strategic point is not the 40%. It is that the loop turns correctness from a *probability you sample* into a *property you engineer* — and that is the difference between a coding demo and a production system.

—

## Ship self-correcting agents with OnlineCloud

If your team is trying to get LLM-generated code into production and hitting the wall between “works in the notebook” and “passes CI,” this is exactly the kind of system we build. OnlineCloud is a US-based, senior-only engineering studio specializing in MCP-native agent architectures, sandboxed execution, and evaluated, observable AI systems — shipped into your VPC, wired to your existing stack, in 6–10 weeks rather than quarters.

**We don’t sell decks — we ship evaluated, observable systems.**

– Design a Code-Test-Fix loop tailored to your codebase and eval suite
– Stand up gVisor-backed sandbox execution with real resource limits and audit trails
– Build the log-distillation layer that makes repair prompts actually converge
– Instrument correctness, latency, and cost-per-task so you can prove the ROI

**[Book a free initial consultation →](https://www.onlinecloud.dev/)** — reach us at **contact@onlinecloud.dev**. Response within 24 hours.

*Note: the benchmark figures above are from OnlineCloud’s internal task suites and will vary with model choice, task complexity, and specification quality. We will reproduce the measurement on your data before you commit to anything.*