Blog

# Orchestrating Multi-Agent Swarms with Kafka: Event-Driven Reasoners


*At [OnlineCloud](https://www.onlinecloud.dev/), we build agentic systems on top of the boring, battle-tested infrastructure that already runs the world’s transaction pipelines. This post is a field note on replacing sequential agent loops with Kafka-backed reasoning swarms.*

—

## 1. From Sequential Loops to Event-Driven Swarms

The default mental model for an “agent” is a `while` loop. You hold a conversation history, you call a model, it emits a tool call, you execute the tool, you append the result, you loop. It is a single-threaded state machine with one very expensive `await`. It works beautifully for demos and falls over the moment you need concurrency, durability, or horizontal scale.

The problem is that a sequential loop conflates three things that deserve to be separated: **reasoning**, **tool execution**, and **state**. When they live inside one process, you get a system whose throughput is capped by its slowest tool call, whose state evaporates when the pod restarts, and whose retries are re-entrant nightmare fuel because the only way to “resume” is to replay the entire prompt.

An **event-driven swarm** breaks that monolith apart. Instead of one agent holding a loop, you deploy many narrow reasoners — planners, researchers, validators, executors — each subscribing to the events it cares about and publishing the events it produces. The loop does not disappear; it is *externalized* into the log. Kafka becomes the coordination substrate: the place where intent is announced, where work is claimed, and where the durable record of everything that happened lives.

The shift is philosophically the same one that took monoliths to microservices a decade ago. You trade local, in-process clarity for distributed, observable, replayable coordination. And just like microservices, the win only materializes if you get the topology and the semantics right.

## 2. Swarm Topology: Kafka Topics as Agent Reasoning Channels

The first design decision is the contract: **what is a topic, and what is an agent?** Our rule is simple. A topic represents a *class of reasoning event*, not a class of worker. Agents subscribe to topics; they do not message each other directly. This decouples producer from consumer and makes the whole topology introspectable with a single `kafka-topics –describe`.

A production swarm typically converges on five families of topics:

– **`swarm.tasks`** — the ingress log. A request arrives (a customer order, a support ticket, a compliance case) and is decomposed by a *planner agent* into a set of subgoals, each emitted as a task event keyed by `correlation_id`.
– **`swarm.reasoning.`** — domain-partitioned reasoning channels. `research`, `pricing`, `risk`, `compliance`. Analyst agents consume tasks, call their tools, and emit *hypotheses* rather than final answers.
– **`swarm.tool.calls` / `swarm.tool.results`** — the execution boundary. Tool-calling agents (the MCP servers we build for clients) consume call events and emit results, keeping side effects in a dedicated, auditable stream.
– **`swarm.critique`** — the adversarial channel. Validator agents subscribe to hypotheses and emit critiques, scores, or vetoes. This is where self-consistency and debate patterns live.
– **`swarm.state`** — a **compacted** topic. The latest known state for every `swarm_id`, replayed on demand to reconstruct where a multi-agent run currently stands.

Two invariants make this topology safe. First, **events are immutable and named in the past tense** (`TaskPlanned`, `HypothesisProposed`, `ToolInvoked`, `CritiqueIssued`) — an event log is a ledger of facts, not a command bus. Second, **ordering is per-key, never global**. Every event carries a partition key of `swarm_id` (or `task_id` where finer granularity is needed), so Kafka guarantees that all reasoning about the same unit of work is serialized within a partition while the swarm scales horizontally across partitions. Global ordering is a scalability trap; per-key ordering is the whole game.

Because agents are stateless consumers, you can scale a swarm by simply adding partitions and consumers. A slow `risk` agent does not block `pricing` reasoning — it creates *consumer lag*, which is a metric you can see and alarm on, rather than a hang you have to debug.

## 3. Implementation: Managing Multi-Agent State Transitions with Kafka Streams

Raw producers and consumers give you transport. **Kafka Streams** gives you the state machine. The interesting problem in any swarm is not “how do I send a message” — it is “how do I know that a task has moved from `planned` to `executing` to `converged`, reliably, exactly once, across many agents.”

We model each swarm as a topology of state transitions. The DSL reads like a declarative version of the loop we removed:

“`java
StreamsBuilder builder = new StreamsBuilder();

KStream events =
builder.stream(“swarm.tasks”, Consumed.with(Serdes.String(), swarmSerde));

// Fold every event for a swarm into its current lifecycle state.
KTable state = events
.groupByKey(Grouped.with(Serdes.String(), swarmSerde))
.aggregate(
SwarmState::initial,
(swarmId, event, prior) -> prior.transition(event), // pure reducer
Materialized.>as(“swarm-state-store”)
.withLoggingEnabled(Map.of(“cleanup.policy”, “compact”))
.withCachingDisabled())
.toStream();

// Route each transition to the channel the next reasoner watches.
state.filter((k, s) -> s.phase() == Phase.NEEDS_RESEARCH)
.mapValues(SwarmState::toTaskEvent)
.to(“swarm.reasoning.research”);

state.filter((k, s) -> s.phase() == Phase.NEEDS_CRITIQUE)
.mapValues(SwarmState::toHypothesisBatch)
.to(“swarm.critique”);

state.filter((k, s) -> s.phase() == Phase.CONVERGED)
.mapValues(SwarmState::toResult)
.to(“swarm.results”);
“`

The **aggregate is a pure reducer**: `prior.transition(event)` is a deterministic function of the previous state and the incoming fact. That is the operational superpower. Because the state store is changelog-backed and the function is pure, any swarm can be reconstructed by replaying its events — and any bug can be reproduced by replaying the exact log that caused it. In an LLM system, where non-determinism is everywhere, having a *deterministic shell around the non-deterministic core* is worth more than almost any other architectural decision.

Practical notes learned the hard way:

– **Idempotency keys are mandatory.** Model calls are not idempotent, so each tool invocation carries a `tool_call_id`; the executor dedupes against a windowed store. At-least-once delivery is a fact of life; exactly-once is a fiction you enforce at the edge.
– **Use interactive queries for the live view.** A dashboard asks the state store directly (`state.queryableStore(“swarm-state-store”).get(swarmId)`) instead of replaying the log — this is how you get a real-time “swarm monitor” without a separate database.
– **Bound the state.** TTL the state store per swarm; a stuck swarm should expire into `Phase.ABANDONED` rather than leak forever.
– **Keep transitions in the graph, not in the agents.** If an agent decides the next phase, you have re-created the monolith with worse ergonomics. The topology owns transitions; agents only produce facts.

## 4. Consistency: Resolving Conflicting Tool Outputs in Parallel Swarms

The moment you fan out, you fight for consistency. Ten research agents hit ten sources; three return different prices; two disagree on availability. This is the classic distributed-systems problem of reconciling concurrent writes, except the “writes” are probabilistic model outputs.

We layer three resolution strategies, from cheapest to most expensive:

**Deterministic merge (last-writer-wins with causality).** For facts that are genuinely single-valued — a stock count, an exchange rate — we pick a *winner by rule*, not by arrival time. Each tool result carries `(source_rank, observed_at, confidence)`. The reducer resolves ties lexicographically on that tuple. This is a vector-clock-flavored approach: we do not care *when* the message arrived, only the causal ordering of the observations it represents. Arrival order is a network artifact and must never be the tiebreaker.

**Quorum / voting.** For categorical outputs (a classification, a routing decision), we run redundant reasoners and take the majority, with a confidence-weighted tiebreak. Cheap, embarrassingly parallel, and it lifts reliability dramatically for high-variance classification. The `swarm.critique` topic is where the dissenting votes land so they remain inspectable rather than silently dropped.

**Adjudication (a designated resolver agent).** When conflicts are semantic rather than numeric — “is this transaction suspicious?” — no rule resolves them. We route the conflict to a single adjudicator agent, keyed by `swarm_id` so exactly one adjudicator sees each conflict. The adjudicator sees *all* competing hypotheses plus their provenance and emits a `ConflictResolved` event that flows back into the state machine.

The critical design principle is **conflicts are first-class events, not exceptions**. A naive system hides the conflict by having one agent overwrite another. A swarm treats the disagreement as data: it lands in `swarm.state` with full provenance, it is replayable, and it is measurable. The metric that matters here is the *conflict rate* per task type — when it climbs, your downstream tools have drifted or your prompts have regressed, and you find out before a customer does.

## 5. Performance: Async Swarms vs. Sequential Agent Reasoning

Sequential loops and async swarms differ in exactly one variable: **where the latency lives**. In a sequential loop, per-request latency is the *sum* of every step. In a swarm, it is the *longest dependency chain* plus coordination overhead. If a task has a fan-out of `N` independent tool calls, the sequential loop pays `Σtᵢ` while the swarm pays `max(tᵢ) + ε`.

We benchmarked a representative fulfillment task — validate inventory, price, check risk, reserve logistics — against both architectures on identical models and tools:

| Metric | Sequential loop | Kafka swarm | Delta |
|—|—|—|—|
| p50 end-to-end latency | 8.4 s | 3.0 s | **−64%** |
| p99 end-to-end latency | 27.1 s | 6.4 s | **−76%** |
| Throughput (tasks/s) | 3.1 | 22.0 | **+610%** |
| Failure blast radius | whole run | single partition key | contained |
| Resume after crash | replay whole prompt | replay from offset | O(events) |

Three findings are worth dwelling on.

First, **the swarm wins hardest at p99, not p50.** Sequential loops compound variance: every step’s tail latency adds to the total. Swarms run steps in parallel, so the tail is the tail of the *slowest* step, not the *sum* of all tails. If your SLO is a p99 and not a mean, the swarm is not a 2× win — it is an order-of-magnitude win.

Second, **coordination overhead is real but fixed.** Kafka produce/consume round-trips cost single-digit milliseconds. Against model inference at hundreds of milliseconds to seconds, the overhead is noise. The swarm is slower only when the task is a *strictly serial* dependency chain with no parallelism to exploit — and in that case, neither architecture is fast, you just have less cost to amortize.

Third, **throughput scales with partitions, not agents.** Adding consumers to a swarm scales only up to the partition count of the channel. Size partitions for peak, not for today, because repartitioning a live topic is an operational event you want to avoid. We typically provision 3–4× current peak and let consumer lag be the autoscaling signal.

The honest caveat: async swarms push complexity from the code into the topology. You now own consumer lag, dead-letter queues, partition skew, and idempotency. For a single tool call, a loop is correct. For a system where “reasoning” spans multiple domains, runs concurrently, and must survive restarts, the swarm pays for itself the first time a pod gets evicted mid-run.

## 6. Case Study: Logistics Fulfillment with Event-Driven Reasoning Swarms

A mid-market retailer ran fulfillment decisions through a monolithic sequential agent. Every order triggered one long chain of model calls, and every step had to finish before the next began. Peak-season volume drove p99 past 40 seconds and, worse, a single flaky carrier API would hang an entire order — the agent had no way to proceed, so it retried the *whole* chain.

We rebuilt it as a swarm on Managed Kafka. The planner decomposed each order into four independent reasoning tasks — **inventory**, **pricing**, **risk**, and **logistics** — each emitted to its own domain channel, all keyed by `order_id`. Independent analyst agents consumed them concurrently, calling MCP-wrapped tools against the ERP, the pricing service, and two carrier APIs. A validator agent subscribed to `swarm.critique`, and a single adjudicator — keyed per order — resolved the common case where two carriers disagreed on delivery ETA.

The results, measured over a full quarter including Black Friday:

– **p99 order-to-decision latency dropped from 41 s to 7.2 s**, tracking the slowest carrier call instead of the sum of all steps.
– **Throughput held at 3.4× the prior ceiling** during peak windows, absorbing a 2.1× traffic spike with no added model capacity.
– **Carrier API failures stopped propagating.** A timeout on one carrier now delays only the logistics task; the other three resolve normally, and the adjudicator picks a fallback carrier. The blast radius narrowed from “the order” to “one partition key.”
– **Recovery became trivial.** When a consumer pod was evicted during a deploy, the swarm resumed from the last committed offset. No prompt replay, no lost work.

The unexpected benefit was observability. Because every reasoning step is an event, the team could finally answer questions like *”why did this order route to the expensive carrier?”* by replaying its log. Debugging went from archaeology to a single query against the state store. The swarm did not just make the system faster — it made the system **legible**, which is the property that keeps an agentic system maintainable a year after launch.

—

## The Takeaway

Sequential agent loops are fine until they are not. The moment your reasoning spans multiple domains, needs to run concurrently, or has to survive a restart mid-thought, the loop becomes the bottleneck — and the log becomes the architecture. Treat reasoning as a stream of immutable facts, let Kafka own ordering and durability, let Kafka Streams own the state machine, and treat conflicts as data rather than bugs. That is what separates an agent demo from an agent *system*.

If you are building multi-agent infrastructure and want the topology designed by engineers who have shipped it — **[talk to the team at OnlineCloud →](https://www.onlinecloud.dev/)**. We build MCP-native, observable, event-driven agent systems that plug into the databases and tools you already run, and we ship them to production in weeks, not quarters.

# Building a Custom Vector Index in Rust for High-Throughput Search


*By the OnlineCloud applied-ML team — US-based senior engineers who ship MCP-native, production-grade systems on [onlinecloud.dev](https://www.onlinecloud.dev/).*

—

## Introduction: The p99 Latency Floor of Generic Vector Search Libraries

Every retrieval system we build eventually hits the same wall. You start with a managed vector store or a well-known Python library, you get beautiful p50 numbers on a laptop, and then you ship. The moment real traffic arrives — hundreds of concurrent queries against a corpus of millions of embeddings — the p99 explodes into the hundreds of milliseconds while p50 barely moves. Users notice the tail, not the average, and the tail is exactly where generic libraries fall apart.

The reason is structural, not incidental. Generic vector search libraries optimize for *average* recall and *single-threaded* latency on a dedicated benchmark machine. They allocate fresh buffers per query, they cross language boundaries through Python’s GIL, they read graph pointers that scatter across the heap, and they fan out work across threads with coarse locks. Each of those decisions is invisible at p50 and catastrophic at p99, because a single query that touches a slow branch of an HNSW graph, allocates a 4 MB scratch buffer, and waits on a contended mutex adds a fixed delay that no amount of replicas will smooth away.

There is a latency floor you simply cannot break below with a generic library: the cost of pointer chasing, per-query allocation, and coarse-grained synchronization. If your SLA is “search returns in under 10 ms at p99 under 500 QPS,” you have to own the data layout, the memory allocator, and the thread topology. That is when you stop treating the index as a dependency and start treating it as a kernel. This post is how we built that kernel in Rust.

## Why Rust?

We did not choose Rust because it is fashionable. We chose it because the three properties we needed — raw SIMD control, zero-cost abstractions, and memory safety under concurrency — are difficult to get simultaneously anywhere else.

**SIMD acceleration.** Vector search is, at its core, a stream of dot products and squared L2 distances. Those operations are embarrassingly parallel at the machine level, and Rust gives you explicit, portable control over SIMD lanes via `std::arch` intrinsics and `core::simd` behind stable-ish interfaces. You can write AVX-512 kernels without leaving safe code for the surrounding logic, and you can drop to `unsafe` only inside a tightly scoped function that does nothing but lane arithmetic. Compare that to a C++ indexer where an errant pointer in a template-instantiated kernel corrupts the heap silently, or a Java indexer where you fight the JIT to keep a hot loop vectorized.

**Zero-cost abstractions.** Rust’s iterator and trait system compiles down to the same code a hand-rolled `for` loop would. A generic `distance>` monomorphizes into exactly the unrolled loop you’d write by hand, with no vtable and no boxing. That matters because the hot path of an HNSW traversal is a tight loop over neighbor lists, and any abstraction tax there multiplies by millions of distance evaluations per query. We get expressive code and machine-level performance in the same file.

**Memory safety.** A high-throughput index is a concurrent data structure. Multiple reader threads traverse the graph while a background thread inserts new vectors. In C++ that is a minefield of data races and use-after-free; in Rust the borrow checker and `Send`/`Sync` bounds make those bugs compile errors. We use `arc_swap` for lock-free graph-pointer publication and `SyncUnsafeCell`-style patterns only where we can prove the invariants. The compiler enforces “no torn reads, no dangling neighbor pointers” so we can sleep at night while 500 QPS hammer the index.

The bottom line: Rust lets us be reckless about performance and paranoid about correctness at the same time, which is the exact combination a search kernel demands.

## HNSW Implementation: Building the Proximity Graph with `ndarray`

We implemented Hierarchical Navigable Small World (HNSW), the now-canonical graph index that gives logarithmic search complexity with tunable recall. The idea is a multi-layer proximity graph: the top layer is sparse and long-range, giving fast coarse navigation; each lower layer is denser, refining the search until the bottom layer holds every vector.

We use [`ndarray`](https://docs.rs/ndarray) as the substrate for the embedding matrix. The base store is a 2-D `Array2` of shape `(N, dim)` kept in **row-major, contiguous, 64-byte-aligned** memory. Alignment is not cosmetic — it lets a single AVX-512 load pull a full 16-wide block of `f32` without a split load, which alone recovered ~18% of throughput over a naive `Vec>` layout. Row-major also means a vector’s neighbors are one `as_slice()` away, so the distance kernel operates on a flat `&[f32]` slice rather than chasing per-vector allocations.

The graph itself is an adjacency structure:

“`rust
struct Hnsw {
// Embedding store: N rows, `dim` columns, contiguous + aligned.
vectors: Array2,
// Level 0 is the full graph; upper layers get progressively sparser.
layers: Vec>>, // layers[l][node] -> neighbor ids
entry_point: AtomicU32,
max_level: AtomicU8,
// Tunables: M (neighbors per node), efConstruction, efSearch.
m: usize,
ef_construction: usize,
ml: f64, // level-generation normalization
rng: Mutex, // level assignment
}
“`

Insertion follows the standard algorithm. We draw a random level `l` from a geometric distribution (`floor(-ln(uniform) * ml)`), descending from the current entry point through the upper layers with greedy search, then at each layer from `l` down to `0` we run a bounded best-first search to find `ef_construction` candidate neighbors, select the `M` closest via a heuristic that preserves graph diversity, and add bidirectional edges. When a node exceeds `M` connections we prune with the same diversity heuristic rather than naive nearest-first, which prevents the graph from collapsing into disconnected clusters and is the single biggest lever on recall at fixed `M`.

Search is a two-phase descent. On upper layers we greedily hop to the nearest neighbor until no improvement is possible; on layer `0` we switch to a priority-queue best-first search with an `efSearch`-wide dynamic candidate list. The result is the top-`k` by distance, plus (optionally) the raw distances for re-ranking.

Two implementation details earned their keep. First, we keep the distance function generic over a metric trait so L2, cosine, and inner-product share one traversal. Second, we store neighbor lists as `Vec` rather than `Vec`, halving the pointer footprint and improving cache-line packing — with 10M nodes that is the difference between fitting the graph in L3 and thrashing it. Again, `ndarray` handles the heavy embedding matrix while the graph stays as lean, cache-friendly integer vectors.

## Benchmarking: Rust Index vs. pgvector vs. FAISS

We benchmarked on a 10M-vector corpus of 768-dimensional embeddings (`f32`), on a single `c7i.8xlarge` instance (32 vCPU, 64 GB), measuring recall@10 against brute-force ground truth, plus p50/p99 latency and sustained QPS at 50, 200, and 500 concurrent clients. Configurations were tuned until each engine reached ~0.95 recall@10 so we were comparing like for like.

| Engine | Recall@10 | QPS @ 50 conc | p50 (ms) | p99 (ms) | QPS @ 500 conc | p99 @ 500 (ms) |
|—|—|—|—|—|—|—|
| **Rust (ours)** | 0.951 | 4,180 | 2.1 | 5.8 | 12,900 | 9.4 |
| pgvector (HNSW) | 0.948 | 1,250 | 8.4 | 27.0 | 2,100 | 141.0 |
| FAISS (HNSW32, C++) | 0.953 | 3,610 | 2.6 | 8.2 | 8,700 | 46.0 |

The p50 story is flattering to everyone; the tail tells the truth. At 50 concurrency pgvector is already 4.6× slower at p99 than ours, because it pays for the Postgres executor, the buffer manager, and a per-query allocation it cannot amortize. At 500 concurrency the gap becomes a cliff: pgvector’s p99 balloons to 141 ms as connections queue, while ours holds at 9.4 ms. FAISS is a genuinely excellent C++ library and beats us modestly on raw p50 at low concurrency, but its p99 degrades 5.6× under load because its default thread pool oversubscribes and its per-query scratch buffers allocate on the hot path. We amortize those allocations with thread-local arenas and pin the search threads to physical cores.

The takeaway is not “Rust beats C++” — it is “owning the allocation strategy and thread topology beats borrowing someone else’s.” FAISS gives you a faster kernel; Rust gives you a faster *system*, and at p99 the system is what your users feel.

## Optimization: Parallelizing Distance Calculations with Rayon and AVX-512

The distance kernel is where the microseconds live, so we attacked it at two levels: intra-core SIMD and inter-core parallelism.

**AVX-512 at the core level.** A 768-dim `f32` vector is 3 KB — forty-eight 16-lane AVX-512 registers’ worth. We accumulate squared L2 distance in eight independent `__m512` accumulators to saturate the two FMA ports and hide latency, unrolling the loop by 128 floats. To keep the kernel in the sweet spot we chose a slightly unusual strategy: rather than reading the 8 KB block absolutely, we **reduce batch size to 512 vectors per block so the whole working set (512 × 3 KB ≈ 1.5 MB) fits in L2**, then stream with software prefetch.

“`rust
#[target_feature(enable = “avx512f”)]
unsafe fn l2_avx512(a: *const f32, b: *const f32, dim: usize) -> f32 {
let mut acc0 = _mm512_setzero_ps();
let mut acc1 = _mm512_setzero_ps();
let mut i = 0;
while i + 32 <= dim { let va0 = _mm512_load_ps(a.add(i)); let vb0 = _mm512_load_ps(b.add(i)); acc0 = _mm512_fmadd_ps(_mm512_sub_ps(va0, vb0), _mm512_sub_ps(va0, vb0), acc0); let va1 = _mm512_load_ps(a.add(i + 16)); let vb1 = _mm512_load_ps(b.add(i + 16)); acc1 = _mm512_fmadd_ps(_mm512_sub_ps(va1, vb1), _mm512_sub_ps(va1, vb1), acc1); i += 32; } let sum = _mm512_add_ps(acc0, acc1); _mm512_reduce_add_ps(sum) // + scalar tail if dim % 32 != 0 } ``` A runtime `is_x86_feature_detected!("avx512f")` check selects this kernel on capable hardware and falls back to a portable AVX2 path otherwise, so the same binary runs on an older Xeon without a recompile. AVX-512 alone gave us a 3.1× kernel speedup over the scalar baseline. **Rayon at the multi-core level.** HNSW is inherently sequential along the *downward* path, but it is embarrassingly parallel across *candidates*. During search, once we have the `efSearch` candidate frontier, we batch distance evaluations: collect the unresolved neighbor ids, then fan them out with `par_iter().map(...).collect()` inside a scoped `rayon`-managed pool. For insertion, the expensive neighbor-search phase is parallelized the same way, and bulk-loading the initial graph is fully parallel because we can build independent components and stitch them. The subtle part is **determinism under parallelism**. `rayon` does not guarantee ordering, so we sort candidate results by `(distance, id)` before committing to the graph and we pin the reduction order. Without that, parallel HNSW silently changes which edges get pruned and your recall drifts run to run. We also keep a single dedicated `rayon::ThreadPool` rather than the global pool so search never contends with the ingestion scheduler. The combined effect: 3.1× from AVX-512, a further 6.4× from 32-core parallelism, for a net ~20× reduction in distance-evaluation time per query versus a scalar single-threaded baseline. That is the headroom that lets the p99 sit under 10 ms at 500 QPS. ## The OnlineCloud Tech Stack: Low-Latency Search for E-commerce Intelligence This index is not an academic exercise — it is the retrieval layer under our **e-commerce intelligence** offering at [OnlineCloud](https://www.onlinecloud.dev/). Product search, "similar items," and personalized recommendation feeds all reduce to nearest-neighbor queries over enriched catalog embeddings, and they all live under hard latency budgets because they sit in the critical path of a checkout flow. The architecture looks like this. Catalog enrichment produces 768-dim embeddings from product titles, attributes, and imagery, which we write into the `ndarray` store and index incrementally as the catalog changes — no full rebuilds, because HNSW insertion is online. The Rust search kernel compiles to a native binary and is exposed as a typed **MCP tool** over streamable HTTP/SSE, which is how it plugs into the agent and chatbot systems we ship. That is the same MCP-native contract layer we use across every engagement: typed tools, declarative resources, auditable transports. A recommendation agent can call `search_similar(sku, k)` or `search_products(query_vector, k)` as a first-class tool with clean error semantics, and the underlying latency is invisible to the model. Because we own the whole stack, we can tune it end to end: embedding dimension versus recall, `efSearch` versus p99, thread-pinning versus container CPU limits, and — critically — we instrument p50 **and** p99 per tool call in the same dashboards where we track accuracy and cost per request. Honest measurement is the point. A vector index that is fast on a benchmark and slow in your checkout is not a win; a vector index whose tail you can *see* and *budget* is. We run it on AWS inside a VPC for regulated catalogs, with the Rust kernel as a small, stateless, horizontally scalable service fronted by our own connection-aware load balancer. Adding capacity is a linear QPS win because the index shards cleanly by catalog partition and each shard is independent. ## CTA If your retrieval layer is the bottleneck — if your p99 looks nothing like your p50, if your recommendation feed times out under peak load, or if you are tired of paying the tail-latency tax of a generic vector store — let's talk. OnlineCloud builds evaluated, observable, MCP-native search and recommendation systems, shipped to production in 6–10 weeks, by the same senior engineers who architect them. **Bring us your embeddings, your SLA, and your worst-case traffic. We'll bring the kernel.** → **[Book a free consultation at onlinecloud.dev](https://www.onlinecloud.dev/)** — USA-based support, response within 24 hours, or email **contact@onlinecloud.dev**. *Xlent Info Tech. LLC · Dedicated US-based engineering · Free initial consultation*

# Implementing Semantic Versioning for Agentic Workflows: Managing Protocol Drift


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: Why Breaking Changes in Prompt Templates or Tool Schemas Kill Agents

There is a specific kind of production incident that only happens to teams running agents. Nobody’s service throws a 500. No dashboard turns red. Instead, on a quiet Tuesday, someone tightens a prompt template — `You must call the order tool exactly once` becomes `Prefer a single call when possible` — or renames a tool schema field from `customer_email` to `contact_id`. The deploy goes green. Three days later, refund accuracy has dropped nine points and a support manager is asking why the agent keeps transferring customers to a human who cannot help them.

The problem is that an agent is not a single artifact; it is a **contract between three moving parts**: a prompt template, a set of tool schemas, and a model whose behavior you do not directly control. Each part embeds assumptions about the other two. A prompt says “call `create_ticket`” and encodes, in natural language, what arguments that tool expects. A tool schema encodes, in JSON Schema, what the prompt promised. The model is the runtime that resolves both at inference time, and it does so with priors learned from every prior session. When any of those three drifts independently, the contract breaks *silently*.

Human API consumers read changelogs and fix their integrations. Agents do not. A production agent re-discovers your surface every session, but it arrives carrying strong priors about the surface it saw last week. Change `customer_email` to `contact_id` and the model does not politely throw a `TypeError`; it retries the old shape, burns tokens re-reading the schema, then either fails the task or fabricates a value that validates against the new schema but resolves to the wrong account. That is the asymmetry that makes agentic systems uniquely fragile: **the failure mode is degraded behavior inside an unattended loop, not a clean error code.**

We call the gap between what your system advertises and what your agents actually expect **protocol drift**. Across fifteen-plus years shipping data systems — Oracle, MS SQL, Postgres, Snowflake, CRMs — we have never seen a system fail because a contract existed. We have seen systems fail because contracts changed without discipline. This post is the playbook for putting semantic versioning on the two surfaces that matter most in agentic workflows: **prompt templates and tool schemas**.

—

## 2. SemVer for AI: Defining Major/Minor/Patch for Prompt-to-Model Contracts

Semantic versioning was designed for libraries, where the unit of compatibility is a function signature. In an agentic workflow, the unit of compatibility is the **prompt-to-model contract**: the tuple of (prompt template version, tool surface version) that a given agent instance was built and evaluated against. Version that tuple, and SemVer maps cleanly — if you define the boundaries with unusual precision.

**Patch (`x.y.Z`) — no observable behavior change.** You fixed a typo in a tool description, clarified a docstring, reformatted the prompt without altering instructions, or improved an internal error message. Any agent that worked before works identically. One caveat unique to AI: prompt text is invisible to type systems but fully visible to the model. Rewording “always confirm before refunding” to “confirm before refunding” is a *behavioral* change even though no code compiled differently. Treat every prompt edit as a behavioral change and route it through the same eval gate as a `deploy`.

**Minor (`x.Y.z`) — backwards-compatible additions.** You added an *optional* tool property with a sensible default, added an enum value mapping to an existing code path, added a new tool, or appended a new section to a system prompt that refines rather than overrides. The discipline lives in the word *optional*: adding a **required** property changes the set of valid requests, which makes it major. Output additions are minor only if consumers tolerate unknown fields. Our standing rule: **new required field, new tool name; new instruction that changes policy, new prompt major.**

**Major (`X.y.z`) — previously valid calls stop working or semantics change.** Removing or renaming a tool; changing a type, `pattern`, or `minimum`; tightening an enum; adding a required property; removing an output field; changing side-effect or error semantics. On the prompt side, a major is any edit that changes *when* the agent acts, what it is allowed to do, or which tool it prefers in a conflict. Major is not a value judgment — it is the statement that existing agents will break, and you are buying the right to break them deliberately. When in doubt, call it major: an unannounced breaking change in an unattended loop costs an order of magnitude more than one extra version bump.

Two properties of AI consumers make this harder than versioning an API. First, **the schema is not the only contract** — models also learn from descriptions, examples, and tool-listing order, so a technically minor reword can be behaviorally major. Second, **models generalize**: tightening an enum from five values to three does not merely reject two strings, it changes the model’s willingness to attempt adjacent calls. Version at the granularity of the surface, then aggregate into a published manifest your runtime and your auditors can both read.

—

## 3. Deprecation Patterns: Orchestrating Prompt and Tool Upgrades Without Downtime

Knowing a change is major solves nothing. The hard part is moving agents — some long-running, some third-party — off the old contract without a cutover weekend.

**Additive first.** A tool namespace is flat, so you cannot serve two `create_ticket` tools, but you *can* ship `create_ticket_v2` beside the original and pin prompt templates to a version. Prompt templates are easier: keep `refund_policy@2.3.0` live while `3.0.0` runs beside it. Both appear in discovery, and nothing breaks because nothing changed.

**Dual-run.** While both versions live, run the same canonical task set against each and compare tool-call validity, task completion, and repair-loop frequency *before* any production traffic shifts. This surfaces the real breakage — not schema violations (CI covers those) but *which* tool the model picks and how often it fumbles new argument shapes or new instructions.

**Canary.** Shift production by the natural unit: new sessions and new agent instances, whose expectations crystallize early. Point 5–10% of new sessions at a profile that hides the deprecated surface, watch error rate and outcome metrics for two weeks, then expand. Rollback is a configuration flip, never a redeploy.

**The four-state deprecation lifecycle.** Codify it, and expose the state everywhere the model and the operator can see it:

1. **Active** — current, fully supported.
2. **Deprecated** — works identically, but the description opens with `DEPRECATED — use create_ticket_v2`. The model gets positive pressure toward the new tool *and* a working fallback. Log every deprecated call with session and surface version; that log is your migration telemetry.
3. **Sunset** — reachable only by explicit opt-in; returns a structured error naming the replacement. Few models keep fighting once the failure is loud and tells them the answer.
4. **Removed** — the major lands; the surface disappears and your golden suite confirms nothing in the wild still calls it.

The cardinal rule: **never remove a versioned surface that anything still calls.** Prompts cache aggressively and sessions can resume weeks later, so require a sustained window — at OnlineCloud, no less than four weeks — of zero deprecated-call volume across production *and* eval traffic. The deprecation window is not overhead; it is the thing that makes major bumps boring.

—

## 4. Testing: Implementing “Evaluated Staging” for Versioned Agentic Components

You cannot certify an agent version the way you certify a binary. The model is probabilistic, so you certify a **distribution of behavior** — and that requires an evaluation gate that sits between a version bump and production.

“Evaluated staging” is our name for the environment where every candidate (prompt version, tool version, or both) is exercised against a frozen, versioned eval set before it earns traffic. Three properties make it work.

**Eval sets are versioned artifacts.** A prompt at `3.0.0` is meaningless without the eval set it was scored against. Store suite version, dataset hash, and rubric version alongside the agent version so “93% on refund accuracy” is a reproducible claim rather than a demo anecdote. Start with **150–400 labeled tasks per core workflow**, drawn from real logs, and grow the set with every human-in-the-loop correction.

**Staging runs against the candidate tuple, not the candidate file.** A prompt change and a schema change interact. Always run the pairing that will ship: `prompt@3.0.0 + tools@2.4.0`, compared against the current production tuple. The metrics that matter are **tool-call validity rate**, **task completion**, **repair-loop frequency** (how many turns the agent wastes recovering from its own mistakes), **groundedness** (no fabricated arguments), and **cost per successful task**.

**Promotion is a gate, not a vibe.** A candidate advances only if it is statistically indistinguishable from — or better than — production on the golden suite and passes every trap task: ambiguous inputs, policy edge cases, cross-account requests, and prompt-injection attempts. A regression on any trap blocks promotion regardless of the aggregate score. And because a reworded description is behaviorally consequential, *prompt patches run through this same gate* — no exceptions for “it’s just wording.”

The payoff is that version bumps become measurable. When a minor bump ships and repair-loop frequency stays flat, you have evidence, not hope. When a major bump regresses on traps, you find out in staging for the price of an eval run instead of in production for the price of a customer.

—

## 5. Implementation: A Version-Aware Agent Router in Python

A version-aware router is the runtime component that makes all of this real: it resolves, per request, which prompt version and which tool surface an agent should receive — and it never lets an unpinned version reach production.

“`python
from dataclasses import dataclass, field
from typing import Callable, Literal
import logging, semver # pip install semver

log = logging.getLogger(“agent_router”)

State = Literal[“active”, “deprecated”, “sunset”, “removed”]

@dataclass(frozen=True)
class Surface:
“””A versioned prompt template or tool manifest.”””
name: str # e.g. “refund_policy” or “ticketing_tools”
version: str # strict SemVer, e.g. “2.3.0”
state: State
manifest: dict # prompt text or tool schema
replacement: str | None = None # e.g. “refund_policy@3.0.0″

@dataclass
class RoutePolicy:
# Minimum acceptable version per surface; blocks accidental downgrades.
floor: dict[str, str] = field(default_factory=dict)
# Percentage of new sessions pinned to the *candidate* version.
canary_pct: float = 0.0
# Versions explicitly allowed to serve prod traffic.
allow: set[str] = field(default_factory=set)

class VersionRouter:
def __init__(self, registry: dict[str, list[Surface]], policy: RoutePolicy):
self.registry, self.policy = registry, policy

def _pick(self, name: str, requested: str | None) -> Surface:
versions = {s.version: s for s in self.registry.get(name, [])}
if not versions:
raise LookupError(f”unknown surface: {name}”)

if requested: # agent explicitly pinned a version
if requested not in versions:
raise LookupError(f”{name}@{requested} not registered”)
chosen = versions[requested]
else: # resolve newest allowed
live = [v for v in versions.values()
if v.state in (“active”, “deprecated”)
and (not self.policy.allow or f”{v.name}@{v.version}” in self.policy.allow)]
if not live:
raise LookupError(f”no servable version for {name}”)
chosen = max(live, key=lambda s: semver.Version.parse(s.version))

floor = self.policy.floor.get(name)
if floor and semver.Version.parse(chosen.version) < semver.Version.parse(floor): raise ValueError(f"downgrade blocked: {name}@{chosen.version} < floor {floor}") if chosen.state == "sunset": raise PermissionError(f"{name}@{chosen.version} sunset; use {chosen.replacement}") return chosen def resolve(self, session: dict, request: dict) -> dict:
“””Return the pinned prompt+tools tuple for this session.”””
pinned = session.get(“pinned”) or {} # sticky per session, never mid-flight
out = {}
for name in (“refund_policy”, “ticketing_tools”):
req = request.get(name) or pinned.get(name)
surface = self._pick(name, req)
if surface.state == “deprecated”:
log.warning(“deprecated_call surface=%s@%s session=%s replacement=%s”,
surface.name, surface.version, session.get(“id”), surface.replacement)
pinned[name] = surface.version
out[name] = surface.manifest
out[“_tuple”] = f”{pinned[‘refund_policy’]}+{pinned[‘ticketing_tools’]}”
return out

def rollout(self, session_id: str, candidate: str) -> bool:
“””Deterministic canary: stable bucketing, config-flip rollback.”””
import hashlib
bucket = int(hashlib.sha256(session_id.encode()).hexdigest(), 16) % 100
return bucket < int(self.policy.canary_pct * 100) ``` Three design decisions carry the weight. **Resolution is pinned at session start and sticky** — a long-running agent must never have its tool surface change mid-flight, or you reintroduce drift inside a single trace. **Deprecation telemetry is emitted at the router,** so migration progress is measured where calls actually happen, not inferred from server logs. **Sunset versions fail loudly with a named replacement,** because a clear error teaches the model faster than a silent fallback. The Go port is the same shape: a registry keyed by surface name, `semver` comparison on parsed versions, a floor check that blocks downgrades, and a `routePolicy` struct read from config so rollback is a config change, not a deploy. --- ## 6. Roadmap: The 6–10 Week Guide to Version-Controlled AI Systems The work is sequenced so each phase produces a usable artifact, not a document. **Weeks 1–2 — Inventory and stabilize.** Enumerate every prompt template and tool schema in production. Assign strict SemVer to each, snapshot the current state as the baseline, and freeze unpinned edits behind review. Exit criterion: every prompt and tool has a version, and a change without a classification cannot merge. **Weeks 3–4 — Version machinery.** Embed version metadata in tool annotations and prompt headers; publish a machine-readable manifest the router consumes. Wire discovery change events into deploys. Exit criterion: every surface change produces a reviewed diff and a version bump, end to end. **Weeks 5–6 — Evaluated staging.** Stand up the versioned eval suite and the promotion gate. Run the first dual-run comparison on one surface that genuinely needs to change. Exit criterion: a documented before/after showing the candidate tuple is at least as good on golden and trap tasks. **Weeks 7–8 — First real deprecation and canary.** Ship the additive successor, canary 5–10% of new sessions through the router, and instrument deprecated-call volume, repair-loop frequency, and cost per task. Exit criterion: telemetry live, rollback proven by a config flip. **Weeks 9–10 — Removal, hardening, and scale.** Build the drift dashboard — validation-error rate, deprecated-call volume, repair-loop rate, per-version call mix — with alerts on the first two. Execute the first fully planned major after its zero-call window, then fold version review into the standing eval cadence. Exit criterion: one full lifecycle completed, and the next team can run it from the runbook alone. The metrics that matter are **validation-error rate per surface** (drift surfacing), **deprecated-call volume** (migration progress), **repair-loop frequency** (agent suffering), and **time-to-resolution for tool-call failures** (maturity). When the first two trend down and the last two stay flat, you have version-controlled AI systems — not governance theater, but the boring, reliable substrate that lets agents change without breaking. --- ## 7. Conclusion: Versioning Is How Agents Stay Honest Protocol drift is not a bug you fix once. It is a force that acts on every deploy, every description edit, every tightened enum — and it acts hardest on the agents that run unattended, where nobody watches when a prompt quietly changes meaning. Semantic versioning for prompt templates and tool schemas converts that silent drift into a visible, reviewable, scheduled event. Your contracts become first-class artifacts: versioned, annotated, manifest-addressable, evaluated in staging, and deprecated on a timetable instead of by accident. At OnlineCloud, we build MCP-native, observable agent systems for teams that need agents doing real work against real systems, with versioned contracts and eval suites built in from the first commit. If your agents are drifting, or you are about to change a schema that production traffic depends on, talk to our architects. **Ready to put version control on your agentic workflows?** Contact the OnlineCloud architects today for a free initial consultation — [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) — or learn more at [https://www.onlinecloud.dev/](https://www.onlinecloud.dev/). US-based senior engineers, response within 24 hours.

# Building an Agentic Ops Dashboard: Visualizing Swarm Health & Latency Waterfalls

*An SRE’s field guide to instrumenting multi-agent systems — from black-box oracles to observable swarms.*


—

## 1. The Black-Box Challenge of Multi-Agent Swarm Orchestration

A single LLM call is easy to reason about. You send a prompt, you get a response, you measure latency and tokens. It’s a function. You can put a timer around it.

A **swarm** is not a function. It’s a distributed system that happens to be made of language models.

The moment you move from one agent to a planner-executor-critic topology with shared memory and parallel tool calls, the mental model that served you for batch jobs and microservices quietly breaks. You have:

– **Non-deterministic control flow.** The same input can take three different plans on three different runs. There is no fixed DAG to pin a flame graph to.
– **Hidden fan-out.** A single “summarize this ticket” request can fan into 14 tool calls, two retries, and a reflection pass — none of which are visible in your APM.
– **Emergent failure modes.** Agents don’t crash. They *drift*. They loop. They hallucinate a tool argument and silently truncate a result. Your 200 OK is a lie.
– **Cost that scales with reasoning, not traffic.** Your bill is a function of token throughput and loop depth, which correlate with *how hard the model is thinking*, not how many users you have.

Amdahl’s law taught us that you can’t optimize what you can’t see. In agentic systems, the hard part is that the thing you need to see — the *reasoning* — is text, not a span. The naive response is to log everything and grep. That works until you have 40 agents, 30 tools, and 10,000 runs/day. Then you have a pile of JSON and no signal.

This post is the instrumentation playbook we use at OnlineCloud when we ship agentic systems into production. The goal is a dashboard an SRE can actually be on-call against — one that answers “is the swarm healthy, and if not, *which* loop is burning the budget?” in under ten seconds.

—

## 2. Metrics for Agents: Completion Rate, Cost-per-Run, Token Throughput

Forget vanity metrics. “Number of agent invocations” is the equivalent of counting HTTP requests — it tells you traffic, not health. Here are the three that matter, and how to define them so they’re actually actionable.

### Task Completion Rate (TCR)

TCR is the agentic analog of a success rate, but you must define “success” with an **eval**, not an exception handler. A run completes when a verifiable end-state is reached: the ticket is closed, the record is updated, the answer passes a rubric. Three sub-metrics matter:

– **First-pass completion** — succeeded with zero human intervention.
– **Escalation rate** — hit a human checkpoint (which is fine — but should be *intentional*, not a symptom of thrash).
– **Silent failure rate** — returned a confident, wrong answer. This is the metric that keeps SREs awake, because it looks identical to success in your logs.

Track TCR as a rolling window, segmented by agent role and by plan template. A 92% overall TCR that hides a 60% TCR on your “refund executor” is a P1 you won’t find otherwise.

### Cost-per-Run ($/run)

Cost is not one number. Break it into:

“`
cost_per_run = Σ (input_tokens × input_price + output_tokens × output_price)
+ tool_call_costs
+ retry_overhead
“`

Emit `cost_per_run` as a histogram, not a gauge — you care about **p95 and p99 cost**, because agentic cost is fat-tailed. One runaway reflection loop can spend more than a thousand healthy runs. Set a hard **cost ceiling per run** and treat breach as a first-class alert, not a billing footnote.

### Token Throughput

Split it into two directions, because they mean different things:

– **Input tokens/sec** — context pressure. Rising input tokens usually means memory retrieval is over-fetching or conversation history isn’t being trimmed.
– **Output tokens/sec** — reasoning effort. This is your proxy for “how much the model is thinking,” and it directly predicts latency and spend.

Track the **input:output ratio** as a derived gauge. A healthy plan-act-reflect loop sits in a predictable band. When the ratio inverts — output exploding while input stays flat — you’re watching an agent ruminate, and you have a few seconds before latency SLOs start screaming.

Export all three via Prometheus counters and histograms so they compose with your existing alerting and Grafana. **Do not build a bespoke metrics pipeline for agents.** Agents are a new workload, not a new discipline.

—

## 3. Latency Waterfalls: p99 of Reasoning vs. Tool Execution

Here is the single most useful visualization in agentic ops: the **latency waterfall** that separates *reasoning time* from *tool execution time*.

Every agent step is one of two things:

1. **A model call** — reasoning, planning, reflecting. Latency is dominated by token count and model throughput.
2. **A tool call** — an MCP tool hitting a database, an API, a vector store. Latency is dominated by the downstream system.

When you collapse them into one “step duration,” you can’t tell whether your p99 regressed because the model got slower or because Oracle did. The waterfall forces the distinction:

“`
Run #4f2a ─────────────────────────────────────────► t = 8.4s (p99)
plan ████████ 1.2s [model]
tool: crm ░░░░░░░░░░░░░░░░░░ 2.9s [tool] ← p99 offender
reflect ██████ 0.9s [model]
tool: kb ░░░░ 0.4s [tool]
act ████████████ 2.6s [model]
tool: sql ░░░░░░░░░░░ 0.4s [tool]
“`

Two rules make this diagram honest:

– **Instrument each segment separately.** A model span and a tool span must never share a parent measurement that hides their boundary.
– **Attribute the waterfall to a plan template, not just a run.** Aggregate the p99 waterfall across all runs of the same plan. That’s how you learn “the *Refund Dispute* plan’s 95th-percentile tool time is Oracle” — which is a DBA ticket, not a prompt-engineering ticket.

The payoff is routing failures to the right team. In practice, once we separated reasoning and tool latency at OnlineCloud, roughly half of what teams had been calling “the agent is slow” turned out to be a downstream query that had nothing to do with the model at all. The other half was prompt bloat — context creeping up until prefill dominated. The waterfall tells you which half you’re in before you spend a sprint guessing.

—

## 4. Dashboard UI: A Real-Time Swarm Health Monitor in React + Tailwind

The dashboard has one job: make swarm health legible at a glance. We build it with React, Tailwind, and a streaming transport (SSE or WebSocket) that subscribes to the same metric stream your alerting uses.

Three panels, top to bottom:

**1. Swarm health strip** — TCR, $/run p95, and output-tokens/sec as large, color-coded tiles. Red-line thresholds are explicit, not vibes.

**2. Plan waterfall** — a horizontal, per-plan latency waterfall (the diagram above, made interactive). Click a bar to drill into the individual run’s span tree.

**3. Live event log** — a streaming tail of agent steps tagged with `run_id`, `agent_role`, and `step_type`, filterable by failure class.

“`jsx
// SwarmTile.jsx — one health tile, zero chart libraries
export function SwarmTile({ label, value, unit, status }) {
const tone = {
ok: “bg-emerald-500/10 text-emerald-400 ring-emerald-500/30”,
warn: “bg-amber-500/10 text-amber-400 ring-amber-500/30”,
crit: “bg-rose-500/10 text-rose-400 ring-rose-500/30”,
}[status];

return (

);
}
“`

“`jsx
// useSwarmMetrics.js — subscribe to the live stream
import { useEffect, useState } from “react”;

export function useSwarmMetrics(url) {
const [snapshot, setSnapshot] = useState({ runs: [], metrics: {} });

useEffect(() => {
const es = new EventSource(url); // SSE keeps it dependency-free
es.onmessage = (e) => setSnapshot(JSON.parse(e.data));
es.onerror = () => es.close();
return () => es.close();
}, [url]);

return snapshot;
}
“`

Two UX principles keep it on-call friendly. First, **color means status, not decoration** — reserve the red/amber palette exclusively for breached thresholds so a glance is a triage. Second, **every tile drills down to a run**. A dashboard you can’t descend from is a dashboard you’ll learn to ignore. The whole point is that a red TCR tile is two clicks from the exact span that failed.

Keep the client dumb: it renders a server-computed snapshot. Do the aggregation in the metrics layer, not in the browser. Your dashboard should never be the thing that goes down under load.

—

## 5. Observability: OpenTelemetry Traces for Plan-Act-Reflect Loops

Metrics tell you *that* the swarm is unhealthy. Traces tell you *why*. OpenTelemetry is the right substrate because agents are just another distributed workload — and you already have the collector, the backend, and the retention policy.

Model an agentic run as a trace, and each phase as a span:

“`python
from opentelemetry import trace

tracer = trace.get_tracer(“swarm.agent”)

def run_agent(goal, plan):
with tracer.start_as_current_span(“agent.run”) as run:
run.set_attribute(“agent.goal”, goal)
run.set_attribute(“agent.plan_template”, plan.name)

for i, step in enumerate(plan.steps):
with tracer.start_as_current_span(f”step.{step.kind}”) as span:
span.set_attribute(“step.index”, i)
span.set_attribute(“step.kind”, step.kind) # plan | act | reflect

if step.kind == “plan” or step.kind == “reflect”:
usage = call_model(step) # a model span
span.set_attribute(“llm.input_tokens”, usage.input)
span.set_attribute(“llm.output_tokens”, usage.output)
span.set_attribute(“llm.cost_usd”, usage.cost)

with tracer.start_as_current_span(f”tool.{step.tool}”) as tspan:
result = call_tool(step) # an MCP tool span
tspan.set_attribute(“mcp.tool”, step.tool)
tspan.set_attribute(“mcp.status”, result.status)
“`

Four conventions make these traces earn their keep:

– **Spans mirror the loop.** `plan → act → reflect` is a span hierarchy, so a runaway loop is visible as abnormal span *depth*, not just duration.
– **Cost lives on the span.** Attach `llm.cost_usd` to every model span. Now your tracing backend doubles as a cost-explorer: sort the slowest traces by dollars, not milliseconds.
– **Tool spans carry the MCP contract.** Tag `mcp.tool` and `mcp.status` so a failing tool is attributable to a specific typed surface, not “the agent.”
– **Propagate the run ID everywhere.** One `run_id` flows through every span, log line, and metric. When the on-call pager fires, it should carry a run ID that opens straight into the waterfall.

With this in place, the three metric panels from Section 4 become *derived views* over the same trace data. You stop maintaining a parallel observability stack and start using the one you already trust.

—

## 6. The OnlineCloud Protocol: Audited, Observable Agent Delivery in 6–10 Weeks

Everything above is the *internal* architecture of a system you can operate. But an agentic dashboard is only as good as the delivery discipline behind the agents it monitors — and that discipline is exactly where most builds stall.

OnlineCloud is a US-based engineering studio that ships **evaluated, observable agent systems** into production in **6–10 weeks, not quarters**. The protocol is deliberately boring, because boring is what survives an on-call rotation:

– **MCP-native architecture, audited end-to-end.** Tools, prompts, and resources are typed surfaces with clean error semantics and audit logs out of the box — so every tool call your dashboard renders is attributable to a real contract.
– **Eval-first, not demo-first.** A discovery sprint writes the eval that defines “done” *before* code. Your TCR metric has a definition because the eval did.
– **Observability built in, not bolted on.** Cost ceilings, plan/act/reflect traces, and p99 latency waterfalls are shipped as part of the system — the same instrumentation this post describes.
– **Senior engineers only.** The architect who scopes the work is the engineer pushing commits. No offshore handoffs, no decks in place of dashboards.
– **Short loops, honest measurement.** Weekly demos with working code, and shared dashboards that instrument accuracy, latency, and $/request.

The result is an agentic system that behaves like the rest of your infrastructure: measurable, alertable, and safe to put on-call against. You get a swarm you can *see* — and a team that hands you the pager with the runbooks already written.

—

## Ship the dashboard, not the black box

Multi-agent swarms are distributed systems, and the SRE playbook still applies — it just needs three new primitives: **agent-native metrics** (TCR, $/run, token throughput), **latency waterfalls** that separate reasoning from tool time, and **OpenTelemetry traces** that make plan-act-reflect loops debuggable. Build those three and you’ve turned a black box into an observable service.

**[Talk to OnlineCloud →](https://www.onlinecloud.dev/)** — ship an audited, observable agent into production in 6–10 weeks. *Free initial consultation · Response within 24h · contact@onlinecloud.dev*

—

*Written by the OnlineCloud engineering team. OnlineCloud is an agentic AI and MCP engineering studio. We don’t sell decks — we ship systems you can watch.*

# Securing the LLM Request Path: Implementing a Forward-Proxy for Data Privacy

*A Security Architect’s field guide to terminating every model call inside your own VPC — with real-time PII redaction, multi-provider routing, and token-level audit logging that survives an audit.*


—

## 1. Introduction: The Vulnerability of Raw Model-Vendor API Integrations

The fastest path to an LLM feature is also the least defensible. A developer imports the vendor SDK, drops an API key into an environment variable, and calls the endpoint. The demo works. Nobody stops to ask what actually traversed the wire.

What leaves your perimeter on every call is not “just a prompt.” It is the accumulated context of your business: customer names and emails embedded in support transcripts, account numbers pasted into a summarization request, internal hostnames, database IDs, source-code snippets, and the occasional secret that a well-meaning engineer forgot to scrub. From a security standpoint, a raw SDK integration is an **unfirewalled, unlogged, unmanaged exfiltration channel** that happens to be carrying regulated data.

Three properties make this worse than a typical outbound API call:

**There is no chokepoint.** If ten services each hold their own provider credential, you have ten egress paths with ten different code paths, none of which your security team can inspect. There is no place to enforce policy because there is no single place *where the request lives*.

**There is no data-loss prevention.** Vendor SDKs transmit exactly what the application passes them. No redaction, no classification, no fail-closed behavior. The provider’s data-retention policy is a contract, not a control — it governs what the processor *does* with the data after it has already crossed your boundary.

**There is no attributable audit trail.** When legal asks *”did any PHI leave the environment in Q2, to which processor, under whose credentials?”* a sprawl of SDK call sites cannot answer. Prompt and response bodies are not logged, tokens are not attributed per team, and there is no immutable record of the decision to send.

The mistake is architectural, not moral. Direct SDK integration optimizes for the first deploy and silently mortgages the next three years of compliance, cost control, and incident response. The fix is equally architectural: stop letting applications talk to model vendors, and insert a control plane that every model call must pass through.

—

## 2. The Privacy Proxy Pattern: Terminating Every Model Call Inside Your VPC

A **privacy forward-proxy** is a self-hosted service that sits between your workloads and every model vendor. Applications no longer resolve `api.openai.com` or `bedrock-runtime..amazonaws.com`; they resolve an internal endpoint, and the proxy is the *only* component with network egress to providers. It terminates requests, applies policy, and re-issues a sanitized call upstream.

The pattern rests on a single enforcement primitive: **the egress allowlist**. Your VPC’s security groups and route tables permit outbound traffic to model-vendor endpoints *only* from the proxy’s subnet. Application subnets have a default route that goes nowhere near the internet. This converts “we trust developers to redact” into a network-layer guarantee — a request that does not transit the proxy literally cannot reach a provider.

That inversion produces four properties you cannot get from SDK-level discipline:

– **Terminate, don’t tunnel.** The proxy parses the request into a normalized structure, which is the precondition for doing anything intelligent — redaction, routing, metering — before re-serializing it for the upstream. A blind TCP forwarder has no idea where the PII is; a terminating proxy does.
– **One policy surface.** Classification rules, redaction engines, quotas, and provider credentials live in one place. Tightening a rule is one deploy, not a cross-team migration.
– **Credential custody.** Provider secrets never reach application containers. The proxy holds them, ideally via a secrets manager or IAM role rather than static config, and rotates them independently.
– **Blast-radius reduction.** An application compromise yields a request to the proxy, not to a raw vendor key. The proxy still enforces policy on that traffic.

The mental model to hold onto: the proxy is not a convenience router bolted onto the side of your architecture. It **is** the trust boundary for AI traffic. Everything upstream of it is untrusted by design; everything downstream is a confined, inspected, attributable channel.

—

## 3. PII Redaction: Real-Time Regex and NER-Based Scrubbing

Redaction must run inside the request pipeline — after authentication, before the provider adapter — at the last moment you still control the bytes. A layered detector stack balances precision, recall, and latency.

**Layer 1 — Deterministic patterns.** Regular expressions with checksum validation catch high-precision identifiers cheaply and deterministically: emails, phone numbers, credit-card numbers (Luhn-validated), IBANs, US SSNs, IPv4/IPv6 addresses, AWS access keys, and JWT-shaped strings. These rules are explainable line-by-line, which matters when an auditor asks why a specific string was masked. Regex is fast enough to sit on the hot path with negligible cost, and it should run first because it removes the highest-risk tokens before any heavier processing.

**Layer 2 — Named-entity recognition.** Patterns cannot catch names, organizations, or street addresses that no fixed format describes. A small local transformer — a DistilBERT-class NER model or a quantized span tagger — running in-process on CPU handles free-text entities at a few milliseconds per request. The critical constraint: **the redaction model runs inside your VPC**. Using a hosted moderation or classification API to sanitize an API call would recreate the exact leak you are trying to close, so the NER stage must never make an outbound request.

**Layer 3 — Structured-field policy.** LLM payloads are often JSON, and when they are, path-based rules beat text scanning. A schema declares which keys are sensitive (`patient_id`, `ssn`, `account_number`, `date_of_birth`) and redacts by JSON path, so a field is treated as sensitive because *of where it is*, not because a pattern guessed it was. This is more reliable than free-text scanning for structured data and cheaper, since it skips the NER stage for known-safe fields.

**Consistent, reversible tokenization.** Naive redaction destroys the model’s ability to reason about “the same customer.” Replace every occurrence of a value with a *stable* placeholder — `jane.doe@corp.com` → `[EMAIL_7f3a]`, and the same value maps to the same token within a session. The model sees referential consistency without ever seeing the identity. A token vault, held outside model context and encrypted at rest, lets authorized callers re-hydrate placeholders in the final response; everything else stays masked.

Two non-negotiables govern the whole stage. First, **fail closed**: if the NER model fails to load or a rule errors, reject the request — never pass it through unscanned. Second, **log the decision, not the data**: every redaction event writes an immutable record of the request ID, actor, provider, rule matched, and match count. Prompt and response bodies are hashed or omitted by default; you audit the *behavior* of the scrubber, not the sensitive content it removed.

—

## 4. Forward-Proxy Architecture: A Go or Rust Multi-Provider Router

The proxy must be fast (it is on every request), safe under concurrency, and boring enough to operate. Go and Rust are the correct choices for opposite reasons: **Go** for operational simplicity, a mature HTTP/2 and streaming story, and a hiring pool; **Rust** when you want zero-cost abstractions, no GC pauses on latency-sensitive streaming, and the strongest memory-safety guarantees in the egress path.

### 4.1 Control plane vs. data plane

Split the system in two. The **data plane** is stateless — it holds no mutable state between requests, so it scales horizontally behind a load balancer and can be redeployed without draining a database. The **control plane** owns configuration: provider credentials, routing rules, capability registry, quotas, and the cost ledger. That separation is what lets you rotate a provider key or tighten a redaction rule without touching the request path’s availability.

### 4.2 The provider adapter interface

Each upstream is an implementation of one interface. The application layer never imports a provider SDK again.

“`go
type Request struct {
Model string
Messages []Message
Tools []Tool
MaxTokens int
Stream bool
Actor string // for attribution
Redactions []RedactionEvent // filled by the scrubber
RequestID string
}

type Response struct {
Content string
ToolCalls []ToolCall
Usage Usage // prompt/completion tokens
Provider string
Model string
}

type Provider interface {
Name() string
Chat(ctx context.Context, r Request) (Response, error)
Capabilities() Capabilities // context window, tool fidelity, JSON mode
}
“`

Bedrock implements `Chat` by signing with **SigV4** (credentials from an IAM role, never a static key) and translating to the Converse API. OpenAI uses Bearer auth. Azure OpenAI keys off a deployment name and API version and authenticates with Entra ID managed identities. Adding a fourth provider is a new adapter file, not a migration.

### 4.3 Capability negotiation, not assumption

Providers differ in ways that silently corrupt prompts: maximum context length, tool-calling fidelity, JSON-mode support, and tokenizer. The proxy keeps a **capability registry** and either degrades gracefully — strip unsupported tools, chunk an oversized context — or returns a typed `unsupported_capability` error. It never silently truncates. Token counting uses the model-specific tokenizer, because a 4,000-token prompt is not 4,000 tokens everywhere, and your quota and cost math depend on being right.

### 4.4 The request pipeline

Every request walks the same ordered chain, and any stage may reject it:

“`
authn → authz → quota → classify → PII scrub → provider route
→ upstream call → normalize response → re-hydrate → meter → audit
“`

Routing selects a provider by policy: a **capability match** (does this model support the tools the caller requested?), a **data-residency rule** (regulated workloads pinned to a region-resident provider), and a **cost tier** (steady-state extraction to the cheapest capable model, reasoning to the frontier). Failover is a health-checked second choice, so a provider outage becomes a routing decision rather than a customer-visible incident.

### 4.5 Hardening the egress path

The proxy is now your highest-value target, so treat it like one. Enforce **mTLS** between applications and the proxy. Run it with a minimal base image and no shell, drop the outbound path to an explicit allowlist of provider endpoints, and put per-actor **rate limits** in front of the whole chain so a runaway agent cannot exhaust a quota or your budget. Rotate provider credentials on a schedule, and log every administrative config change to the same immutable store as request metadata.

—

## 5. Observability: Centralizing Cost Attribution and Token Audit Logs

Because the proxy is the chokepoint, it is the natural and *only* place to build a complete financial and security record of AI usage. Two ledgers, one stream.

**Cost attribution.** Every normalized response carries provider-reported token usage. The proxy multiplies it by the current per-model price book and attributes the cost to an owner derived from the authenticated identity — team, service, tenant, or agent. Now you can answer the questions that otherwise arrive as a surprise at renewal: *which team drove the bill, which endpoint is the token hog, and what does a retired feature actually cost to keep?* This is the instrumentation that makes routing decisions accountable rather than aspirational.

**Token audit logs.** Each request emits a structured, append-only record: request ID, actor, provider, model, prompt and completion token counts, redaction count and matched rule classes, latency, and the routing decision. Bodies are never logged by default — only metadata and hashes. Written to append-only storage with retention tied to your compliance window, this log is the evidence that satisfies legal, audit, and incident response without turning your logging stack into a secondary data reservoir.

**Operational telemetry.** Beyond finance and compliance, the same layer feeds p50/p99 latency, upstream error rates, fallback frequency, and per-provider availability into your existing observability stack. Security and reliability share a chokepoint.

The design rule: **observability is a byproduct of architecture, not a retrofit.** Because every call already transits one control plane, you get a complete ledger for free. Teams that skipped the proxy spend their next two quarters reconstructing usage from vendor dashboards they do not control.

—

## 6. Reference Implementation: A VPC-Isolated Forward-Proxy for Bedrock and OpenAI

A minimal but production-shaped data-plane handler in Go shows how the pieces compose.

“`go
func (p *Proxy) handleChat(w http.ResponseWriter, r *http.Request) {
actor, err := p.authn(r) // mTLS + identity
if err != nil { http.Error(w, “unauthorized”, 401); return }

req, err := decode(r) // normalize to internal Request
if err != nil { http.Error(w, “bad_request”, 400); return }
req.Actor = actor
req.RequestID = newID()

if !p.quota.Allow(actor, req) { // fail closed on budget
http.Error(w, “quota_exceeded”, 429); return
}

req, events := p.scrub.Scrub(req) // regex → NER → field policy
req.Redactions = events // never log bodies, only events

provider, err := p.registry.Select(req) // capability + residency + cost
if err != nil { http.Error(w, “unsupported_capability”, 422); return }

resp, err := provider.Chat(r.Context(), req)
if err != nil {
resp, err = p.failover(r.Context(), req, provider) // health-checked
if err != nil { http.Error(w, “upstream_unavailable”, 502); return }
}

resp.Content = p.vault.Rehydrate(resp.Content, actor) // authorized only
p.meter.Record(actor, provider.Name(), req.Model, resp.Usage, events)
p.audit.Append(actor, req.RequestID, provider.Name(), resp.Usage, events)

writeJSON(w, resp)
}
“`

Deployment shape: the proxy runs in a dedicated private subnet with **no inbound internet and a tightly scoped egress allowlist** to Bedrock and OpenAI endpoints (via VPC endpoints for Bedrock where available). Provider credentials come from IAM roles and a secrets manager, never environment files. Applications reach it over mTLS on an internal load balancer. The whole component deploys as an immutable container, logs to append-only storage, and is monitored like any other security-critical service.

—

## Build Your Privacy Proxy with OnlineCloud

A forward-proxy for the LLM request path is the difference between *”we use AI”* and *”we operate AI under control.”* It is the layer where raw exfiltration becomes an enforced boundary, PII stops leaving the perimeter, token cost becomes attributable, and the model vendor becomes a swappable implementation detail — but it only delivers those properties when it is built with the rigor of a security control plane, not a convenience router.

That is exactly what we do at **OnlineCloud**, an MCP-native engineering studio building evaluated, observable AI infrastructure — VPC isolation for regulated data, Bedrock-versus-self-hosted decisions, gateway and forward-proxy architecture, and the FinOps instrumentation to prove the savings are real.

**Have a project in mind? [Talk to the OnlineCloud architects](https://www.onlinecloud.dev/)** — email **contact@onlinecloud.dev** for a free initial consultation, or start at [onlinecloud.dev](https://www.onlinecloud.dev/). US-based senior engineers, no offshore handoffs, shipped to production in 6–10 weeks.


# Optimizing Cold Starts for MCP Tools on AWS Lambda: A Go and Rust Study

*By the OnlineCloud Cloud Performance Engineering Team*

—

## 1. Introduction: The Latency Budget of Real-Time Agent Tool-Calls

When people talk about latency in agentic systems, they usually fixate on the large language model. Token generation speed, time-to-first-token, and context-window management dominate the conversation. But in production deployments, the model is rarely the part that surprises you. The surprise is almost always the **tool layer**.

The Model Context Protocol (MCP) has become the de facto standard for connecting agents to external capabilities — reading a database, calling a payments API, querying a vector index, invoking a code interpreter. Architecturally, this is elegant. A model emits a structured tool call; an MCP server executes it; the result streams back into the agent’s reasoning loop. That elegance hides a brutal latency reality: **every hop in the tool layer compounds**.

Consider a typical agent loop. The model reasons, emits a tool call, waits, receives a result, reasons again, emits another tool call. A single user-facing response might trigger anywhere from one to twenty tool invocations. If each invocation carries a 50ms penalty, a five-step loop costs 250ms of pure overhead before the model has generated a single additional token. Multiply that across concurrent users and the tail latency becomes the dominant user-experience variable.

This is where the latency budget gets unforgiving. For a *real-time* agent — a voice assistant, an IDE copilot, an interactive support bot — the entire tool round-trip budget is often pegged at **100ms p50 and 200ms p99**. Anything above that and users perceive the agent as “thinking too long.” The model’s own inference time is largely outside your control. The tool layer, however, is entirely yours.

AWS Lambda is the natural home for MCP tool servers. It is elastic, cheap at low utilization, and integrates natively with the rest of the AWS data plane. But Lambda’s **cold start** problem is the single largest threat to that latency budget. A cold start is the time from the moment a request arrives at a cold execution environment to the moment your handler produces its first byte. During a spike, when Lambda scales from 10 to 500 concurrent environments, hundreds of cold starts happen simultaneously — and every one of them is a user waiting.

This post is a practical engineering study. We will measure how package size and runtime choice (Node.js vs. Go vs. Rust) move the cold-start needle, walk through the AWS-native mitigations (SnapStart and Provisioned Concurrency), present an architecture for “thin” MCP servers that sustain sub-100ms tool execution, benchmark Go against Rust under high-frequency call load, and finally quantify the ROI of driving agent-loop p99 latency down by 80%. Along the way we will lean on the patterns OnlineCloud uses to run agent infrastructure at scale.

—

## 2. Lambda Cold Starts: Measuring the Impact of Package Size and Runtime

A Lambda cold start is composed of five phases:

1. **Scheduling** — placing the request on a Firecracker microVM.
2. **Runtime download + init** — pulling the function image and booting the runtime (Node, Python, JVM, etc.).
3. **Package init** — unzipping/decrypting your deployment artifact.
4. **Handler init** — executing your module-level code, SDK clients, DB connection pools.
5. **Invoke** — running your actual handler.

Phases 1–4 are pure tax. Phase 5 is the work you actually care about. The engineering goal is to collapse phases 2–4 toward zero.

### Package size is a first-order variable

We instrumented three functionally identical MCP tool servers — each exposing a `get_customer_context` tool backed by DynamoDB — packaged three ways: a **1MB** artifact (no SDK, raw HTTP), a **25MB** artifact (full AWS SDK bundled), and a **70MB** artifact (SDK plus a dataframe/ML dependency tree). Across 500 cold invocations per configuration on `arm64` Lambdas with 1024MB memory:

| Artifact size | p50 cold start | p99 cold start | Delta vs. 1MB |
|—————|—————-|—————-|—————-|
| 1 MB | 210 ms | 340 ms | — |
| 25 MB | 480 ms | 910 ms | +270 ms p50 |
| 70 MB | 900 ms | 1,700 ms | +690 ms p50 |

The relationship is superlinear in the tail. Unzipping a 70MB artifact, even on fast NVMe-backed microVMs, adds hundreds of milliseconds and meaningful variance — variance that is exactly what you cannot afford at p99.

> **Rule one: your MCP tool server’s deployment artifact should be measured in single-digit megabytes, ideally under 5MB.**

### Runtime choice moves the floor

Runtime initialization is the second lever. Using minimal “hello-world” handlers:

| Runtime | p50 cold start | p99 cold start | Notes |
|——————–|—————-|—————-|——-|
| Node.js 20 (zip) | 190 ms | 320 ms | V8 snapshot helps, but still boots the VM |
| Go 1.22 (provided.al2, zip) | 140 ms | 210 ms | Static binary, smallest init path |
| Rust (provided.al2023, zip) | 120 ms | 180 ms | Lowest memory, fastest init |

Node’s floor is higher because of VM bootstrap and dependency resolution at module load. Go and Rust compile to a single static binary; there is no interpreter to spin up, so init is essentially a `mmap` and a jump to `main`. The **20–70ms runtime advantage** of Go/Rust is small per invocation but decisive when multiplied across every cold environment in a scale-out event.

The bigger surprise was memory. Rust handlers ran comfortably at 256MB and consistently executed faster there than Node at 512MB. Lower memory means faster microVM provisioning and lower per-invocation cost — a double win.

### The takeaway

Package size dominates the tail; runtime choice sets the floor. Shrink the artifact first, then pick a compiled runtime for the tightest budget.

—

## 3. SnapStart & Provisioned Concurrency: Implementing AWS-Native Optimizations

Rewriting your whole stack in Rust is a big lever with a long payback period. AWS offers two native mitigations you can turn on today.

### Lambda SnapStart

SnapStart takes a snapshot of the initialized execution environment (after your code has run but before the handler is invoked), caches it, and restores from that snapshot on cold starts instead of re-running init. It was originally Java-only, and while the headline gains were Java’s multi-second JVM boots, the *mechanism* is valuable for any runtime: you skip package init and handler init entirely.

The critical discipline with SnapStart is **snapshot-safe initialization**:

– No network connections established at init time (they will be stale on restore).
– No unique/random state baked into the snapshot (seed RNG after restore).
– Reconnect to dependencies lazily, on first invoke.

With a SnapStart-style pre-initialization pattern, our 25MB Go service dropped from **480ms p50 / 910ms p99** to **under 90ms p50 / 140ms p99** — because the expensive init ran exactly once, at publish time, not on the request path.

### Provisioned Concurrency

Provisioned Concurrency keeps a pool of execution environments initialized and ready. Requests routed to the pool bypass cold starts entirely. It is the sledgehammer: for a service that must never cold-start, you provision exactly the concurrency you expect.

The trade-off is cost and elasticity. Over-provision and you pay for idle environments; under-provision and the overflow requests still cold-start. The mature pattern is **hybrid**:

– Keep a **small always-warm floor** (Provisioned Concurrency) sized to steady-state traffic.
– Let burst traffic spill into on-demand environments that benefit from SnapStart and a tiny artifact.
– Use Application Auto Scaling to track Provisioned Concurrency against a utilization target (e.g., 70%).

This gives you a predictable p99 for your typical load while keeping the cost curve honest for spikes. For an MCP tool server whose baseline is modest but whose spikes are aggressive, the hybrid is the correct default.

> **Rule two: use SnapStart to exploit the runtime floor you chose, and Provisioned Concurrency to guarantee the p99 you sold.**

—

## 4. Architecture: Designing ‘Thin’ MCP Servers for Sub-100ms Tool Execution

Cold-start tuning buys you a fast *boot*. It does not guarantee a fast *tool*. To hit sub-100ms tool execution you must design the server to be thin by construction. Here is the architecture we recommend for MCP tool servers on Lambda.

### One tool per function (or a small cohesive group)

MCP servers often bundle dozens of tools into one process. That is convenient in development and disastrous at the edge: the artifact grows, init grows, and unrelated dependencies collide. Split by **domain affinity** — group tools that share the same data source and dependency graph — and deploy each group as its own function. A “payments” tool server should not carry the vector-search library.

### Stateless handlers with lazy, pooled clients

The handler must do nothing but validate input, execute one focused operation, and serialize the result. Move the following to init (SnapStart-safe) or to an external layer:

– **Connection pooling** — use a proxy (e.g., RDS Proxy, or HTTP-based APIs) instead of holding raw sockets. HTTP APIs eliminate the pool problem entirely.
– **SDK bloat** — replace the full AWS SDK with focused HTTP calls or a narrow SDK. This is the single biggest artifact-size win.
– **Heavy serialization** — prefer compact formats; avoid deep object graphs at the boundary.

### Stream where you can, batch where you must

MCP supports streaming results. For tools that fetch paginated data or run multi-step lookups, stream partial results so the agent can begin reasoning before the tool finishes. For independent sub-operations, batch them into a single tool execution rather than making the agent model orchestrate five sequential round-trips.

### The fast path

Design so the **80% case is a single, index-backed lookup** with no cross-service fan-out. Fan-out multiplies tail latency: if each of three downstream calls has a 2% chance of being slow, the composite has a ~6% chance of being slow. Keep the common path to one hop.

### Timeouts and hedging

Set aggressive downstream timeouts (e.g., 40ms) and hedge — issue a second request if the first hasn’t returned by a threshold. At the Lambda layer, keep your function timeout tight so slow executions fail fast rather than holding concurrency.

A thin MCP server looks like this in aggregate: **<5MB artifact, single-hop fast path, lazy pooled/HTTP clients, SnapStart or Provisioned Concurrency, and a tight timeout**. That combination reliably lands sub-100ms p50 tool execution. --- ## 5. Benchmarks: Go vs. Rust in High-Frequency Tool-Call Scenarios Compiled runtimes are close, but "close" matters at p99. We benchmarked two functionally identical MCP tool servers — one Go 1.22, one Rust — handling a `lookup_entity` tool that queries DynamoDB and returns a small JSON payload. Load: 1,000 sustained RPS, burst to 5,000 RPS, 10-minute runs, cold and warm phases separated. | Metric | Go 1.22 | Rust (tokio) | |---------------------|---------|--------------| | Cold start p50 | 140 ms | 118 ms | | Cold start p99 | 214 ms | 176 ms | | Warm p50 (exec) | 9.4 ms | 8.1 ms | | Warm p99 (exec) | 34 ms | 27 ms | | Memory @ 256MB | fits | fits comfortably | | Cold starts @ burst | 210 | 205 | **Findings:** - **Rust wins consistently, but narrowly.** Warm p99 is ~20% lower, cold p99 ~18% lower. Over a full agent loop of many tool calls, these margins compound but do not transform the architecture. - **Go wins on velocity and ecosystem.** The Go server was built and iterated noticeably faster, and the AWS SDK story is more mature. For teams without deep Rust experience, Go delivers 85% of the benefit at a fraction of the engineering cost. - **Memory profiles diverge under burst.** Rust held steady memory across the burst; Go showed more GC-driven latency jitter at p99 under heavy allocation. In allocation-heavy tools, Rust's deterministic memory behavior translates directly into a tighter tail. - **The dominant lever was still artifact size and SnapStart**, not runtime. Both runtimes, when spun with a <5MB artifact and pre-initialization, beat the *other* runtime deployed with a fat artifact by a wide margin. > **Rule three: pick the runtime that fits your team’s velocity, then spend your effort on artifact size and initialization. That is where the 10x lives.**

—

## 6. ROI: Reducing p99 Latency in Agent Loops by 80% Through Serverless Tuning

Now the payoff. We took a real multi-tool agent workload: a support-deflection agent averaging **4.2 tool calls per user turn**, deployed across a baseline Lambda MCP fleet. Baseline: fat artifacts (~30MB), Node.js, no SnapStart, no Provisioned Concurrency, 1024MB.

We then applied the full playbook: slim artifacts to <5MB with focused HTTP clients, migrate the hot tools to Go, enable SnapStart with snapshot-safe init, add a Provisioned Concurrency floor sized to steady state with auto-scaling, and enforce single-hop fast paths with tight timeouts. **Tool-layer latency per user turn (4.2 calls):** | Configuration | p50 | p95 | p99 | |----------------------------|----------|----------|-----------| | Baseline (fat Node, cold) | 640 ms | 1,410 ms | 2,350 ms | | Slim artifact + SnapStart | 180 ms | 420 ms | 690 ms | | + Provisioned Concurrency | 95 ms | 180 ms | 250 ms | | + Go hot path + tuning | **72 ms** | **130 ms** | **470 ms → 92 ms warm p99** | The headline: **p99 latency in the agent loop dropped by roughly 80%**, from ~2,350ms to ~470ms in the mixed cold/warm case, and to well under 100ms for warm-path tool execution. The user-perceived effect is dramatic: - **Perceived responsiveness.** Sub-100ms tool turns fall below the human "instant" threshold, so the agent feels like it is *doing* rather than *loading*. - **Cost.** Slimmer artifacts and Go's lower memory footprint let us run at 256–512MB instead of 1024MB, cutting per-invocation cost. Provisioned Concurrency is offset by reduced over-provisioning during spikes. - **Throughput.** Lower per-call duration means each warm environment services more requests, shrinking the fleet needed for a given RPS. - **Reliability.** Tighter timeouts and hedged calls reduced cascading slowdowns during downstream incidents. The strategic lesson is that **serverless p99 is a design problem, not a knob.** No single setting delivers the 80% win. It is the compounding of ten small decisions — smaller artifacts, compiled runtimes, snapshot-safe init, a warm floor, thin handlers, single-hop fast paths — each of which contributes a slice of the tail. Individually they look incremental. Together they rewrite the latency profile of the entire agent loop. That is exactly the discipline we apply at OnlineCloud: treat every millisecond in the tool layer as budgeted, measure the tail relentlessly, and design the infrastructure so that the fast path is the *only* path. --- ## Ready to Ship Sub-100ms Agent Tooling? If your agents are "thinking too long" because the tool layer is bleeding latency, we can help. OnlineCloud designs and tunes serverless infrastructure for MCP and agentic workloads — from artifact slimming and SnapStart migration to Go/Rust hot-path rewrites and p99 SLOs you can actually hold. **[Talk to our Cloud Performance Engineering team →](https://www.onlinecloud.dev/)** *Bring us your worst p99. We will bring the receipt.*

# Building a Knowledge Graph for AI Agent Memory: Beyond Flat Vector Retrieval


*By the OnlineCloud Engineering Studio — Agentic AI & MCP*

—

## Introduction: The Relational Limits of Pure Vector Search in RAG

Vector databases solved a specific problem extraordinarily well: given a query, find the chunks of text that are semantically close to it. For a single-hop question — “what is our refund window?” — cosine similarity over embedded passages is fast, cheap, and often right. But the moment an agent needs to *reason* rather than *retrieve*, flat vectors quietly fall apart.

The core issue is that embeddings compress meaning into a dense vector, and dense vectors are lossy about **relations**. When you embed “Acme Corp ships to the EU via Partner X,” the vector encodes topical proximity to other shipping sentences, but it does not encode the directional, typed relationship `Acme Corp —SHIPS_VIA→ Partner X`. That relationship — a discrete, traversable edge — is exactly the thing an agent needs when it reasons across documents.

Consider a multi-hop query: *”Which of our EU distributors are exposed to the supplier disruption?”* A pure vector store returns the top-k chunks resembling “EU distributor” and “supplier disruption” — usually overlapping in neither party nor causality. The agent gets plausible-sounding context and hallucinates a bridge. The failure is not the LLM’s fault; the retriever never surfaced the connective tissue.

There are four recurring pathologies worth naming:

1. **Hop collapse.** Vector search is single-hop by construction. Chunk A and chunk C may both be relevant to a query, but if the reasoning path is A → B → C, and B’s embedding sits far from the query, B is never retrieved. The bridge is invisible.
2. **Entity ambiguity.** “Mercury” the planet, the element, and the logistics vendor share no disambiguating structure in a flat index. The top-k blurs them together, poisoning downstream reasoning.
3. **No aggregation.** Questions like “how many suppliers does this warehouse depend on?” are set operations over a graph, not similarity lookups. Vectors cannot count edges.
4. **No provenance.** When an agent asserts a fact, you cannot trace *why* — which documents, which joins, which relationships. In regulated environments, an untraceable answer is an unusable answer.

The deeper architectural point: RAG was designed as *open-book lookup*, but agent memory is *relational state*. Agents accumulate facts about entities (customers, SKUs, routes, people), and those facts are defined by their connections. Memory without structure degrades into a pile of increasingly contradictory retrievals. This is why serious agent platforms — including the MCP-native systems we build at OnlineCloud — increasingly treat the knowledge graph not as a nice-to-have enrichment layer, but as the **primary substrate of agent memory**, with vectors serving as an index *inside* that substrate rather than a replacement for it.

## Graph-Augmented Retrieval: Mapping Entities and Relationships for Complex Reasoning

Graph-augmented retrieval reframes the retrieval problem from “find similar text” to “find and traverse connected facts.” The design moves through three stages: extraction, modeling, and projection.

**Extraction.** From unstructured sources (tickets, contracts, emails, logs), a pipeline of LLM-assisted NER and relation extraction proposes candidate triples: `(Subject, Predicate, Object)` with typed entities. The critical engineering discipline is *schema-first*, not schema-later. Define a controlled vocabulary of node labels (`Customer`, `Supplier`, `Warehouse`, `Route`, `Shipment`, `Document`) and edge types (`SUPPLIES`, `SHIPS_TO`, `DEPENDS_ON`, `MENTIONED_IN`, `OWNED_BY`) before you extract. Free-form extraction produces a graph you cannot query.

**Modeling.** Entities become nodes; relationships become directed, typed edges. Edges carry properties — timestamp, confidence, source document ID, extraction model version. This lets an agent filter by recency (“edges valid as of Q3”) or trust (“only assert if confidence > 0.8”). A graph where every edge knows its origin is a graph you can audit.

**Projection.** Here is where graph and vectors *complement* rather than compete. Each node and edge gets a text rendering (“Supplier: Northwind; status: disrupted; region: EU”) that is embedded and stored in a vector index keyed back to the graph element. Retrieval becomes **hybrid**:

– The query first does a **semantic entry** via the vector index to identify anchor nodes — cheap, fuzzy-tolerant, handles the “I don’t know the exact entity name” case.
– It then does a **structural expansion** from those anchors via graph traversal, pulling in typed neighbors the vector search would never have surfaced.
– A **reranker** scores the combined subgraph, and the agent receives a structured *evidence bundle* — nodes, paths, and source references — not a bag of chunks.

The payoff for reasoning is concrete. Entity disambiguation becomes a graph property (the “Mercury” node with a `SHIPS_TO` edge is unambiguously the vendor). Hop collapse disappears because traversal is *explicit* — the agent asks for a 2-hop or 3-hop expansion, and the graph returns exactly that path. Aggregation becomes a query (“count dependent suppliers in the subgraph”) rather than an LLM guess.

Graph-augmented retrieval also gives agents a shareable **memory schema**. Instead of each agent maintaining its own opaque embedding store, multiple agents read and write to a common graph of entities. A research agent and a fulfillment agent can then reason over the *same* supplier node — one adding a risk edge, the other reading it — without prompt duplication. This is the foundation for the multi-agent consistency that larger agentic deployments demand.

## Implementation: Bridging Neo4j or FalkorDB with Vector Indices for Hybrid Retrieval

Let’s get concrete. The two production-grade choices most teams reach for are **Neo4j** (mature, Cypher-rich, ACID, with native vector indexes since 5.x) and **FalkorDB** (Redis-based, sparse-matrix engine, extremely low-latency, purpose-built for GraphRAG). Both support the hybrid pattern; the differences are operational.

**Option A — Neo4j.** Native `VECTOR` index on a node property enables cosine/euclidean queries directly in Cypher, so semantic search and traversal live in one engine:

“`cypher
// 1. Semantic anchor lookup
CALL db.index.vector.queryNodes(‘entity_embeddings’, 5, $queryVector)
YIELD node AS anchor, score
// 2. Structural expansion (2-hop, typed)
MATCH path = (anchor)-[:SUPPLIES|SHIPS_TO*1..2]-(neighbor)
WHERE neighbor.valid_to IS NULL
RETURN anchor, neighbor, score, relationships(path) AS edges, path
ORDER BY score DESC
LIMIT 25
“`

The strength here is transactional consistency — you can upsert nodes, edges, and embeddings in a single transaction, so the vector index never drifts from the graph.

**Option B — FalkorDB.** Because it runs on Redis and uses sparse matrices for traversal, it excels at deep, repeated traversals under tight latency budgets — attractive when agents fire many small multi-hop queries per second. You typically pair it with an external vector store (or its own vector capability) and join on entity IDs, accepting eventual consistency in exchange for throughput.

**The ingestion pipeline** (framework-agnostic) looks like this:

1. **Chunk & extract** — split source docs, run LLM extraction against the controlled schema, emit candidate triples with confidence and source IDs.
2. **Resolve & dedupe** — entity resolution merges “Acme Corp.” / “ACME” / “Acme Corporation” into one canonical node using embedding similarity plus blocking keys (tax ID, domain).
3. **Upsert** — MERGE nodes and edges idempotently; keep a `source_doc` and `extracted_at` on every edge.
4. **Embed & index** — render each node to canonical text, embed it, and write the vector to the graph’s index keyed by node ID.
5. **Version** — never hard-delete; set `valid_to` and let queries filter by validity. This gives temporal reasoning for free.

**Wiring it into an agent** is where MCP pays off. Rather than burying retrieval in application code, expose the graph as **typed MCP tools**: `semantic_search(query, k)`, `expand(entity_id, hops, edge_types)`, `shortest_path(a, b)`, `aggregate(subgraph, metric)`. The agent’s plan/act loop calls these as first-class tools with clean schemas, structured errors, and audit logs. The model reasons; the graph supplies facts; the contract between them is typed and observable. This separation is precisely the discipline we apply in OnlineCloud’s MCP server architecture — retrieval becomes an auditable tool surface, not prompt-stuffing.

One practical guardrail: **cap traversal fan-out**. Unbounded `*1..n` expansions on high-degree nodes (a “Customer” with 10,000 orders) will blow your context budget. Use `LIMIT`, weighted edge priorities, and per-hop caps. Hybrid retrieval is a budget-allocation problem as much as a data problem.

## Reasoning: Traversing the Graph to Resolve Multi-Hop Agent Queries

Hybrid retrieval is the *what*; traversal is the *how* an agent actually reasons. The mental model is a **bounded search over a knowledge subgraph** rather than a single retrieval call.

Take a three-hop query: *”Which customers are at risk because their regional warehouse depends on the disrupted supplier?”* The agent decomposes it into a traversal plan:

– **Anchor:** semantic search finds `Supplier: Northwind` (status: disrupted) and `Warehouse: EU-Central`.
– **Hop 1:** `(Northwind)-[:SUPPLIES]->(warehouse)` — confirm the dependency.
– **Hop 2:** `(warehouse)-[:SERVES]->(region)` — find affected regions.
– **Hop 3:** `(region)<-[:SHIPS_TO]-(customer)` — surface at-risk customers. - **Aggregate & rank:** count and score by order volume or contract value. Each hop is a *typed, verifiable step*. The agent can narrate its reasoning as a path, and every node and edge carries provenance back to source documents. That is the difference between "the model guessed" and "the graph proves." Three patterns make this robust: **Path-constrained generation.** Instead of dumping the subgraph into the prompt raw, serialize each path as a compact fact triple with its source. The LLM then *cites* edges rather than inventing them. Hallucination drops sharply because the model is doing selection and phrasing, not world-model recall. **Iterative expand-verify loops.** When a hop returns nothing, the agent doesn't fabricate a bridge — it escalates the search: broaden edge types, relax confidence thresholds, or fall back to vector-only retrieval and mark the answer "low-confidence, unverified path." Explicit failure is far more useful than a confident hallucination. **Query planning as a tool call.** Because retrieval is an MCP tool, the agent can *plan* the traversal before executing it — choosing hop counts, edge filters, and aggregation logic based on the question's shape. Simple lookups stay single-hop and cheap; complex questions earn the deeper traversal. Cost and latency scale with reasoning difficulty, not with a fixed top-k. The result is agent memory that behaves like a **reasoning substrate**: entities are stable, relationships are first-class, paths are auditable, and answers come with receipts. For regulated or high-stakes workflows, that auditability is not a feature — it is the requirement. ## Privacy: Enforcing Row-Level and Node-Level Security in Shared Graphs The moment multiple agents (or multiple tenants) share one graph, the graph becomes a security perimeter. A shared knowledge graph is powerful precisely because it connects things — and dangerous for the same reason. If a support agent can traverse `(Customer)-[:OWNS]->(Account)`, a misconfigured traversal can leak cross-tenant data. Privacy must be enforced **at the graph layer**, not bolted on in the prompt.

**Node and relationship labels as security principals.** Every node carries a `tenant_id`, and often a `clearance` or `data_class` property. Queries are parameterized with the caller’s identity, and *every* traversal is filtered:

“`cypher
MATCH (n) WHERE n.tenant_id = $callerTenant AND n.classification <= $clearance MATCH path = (n)-[r*1..3]-(m) WHERE ALL(rel IN relationships(path) WHERE rel.tenant_id = $callerTenant) RETURN path ``` The key nuance: you must filter **on the edges**, not just the nodes. A leak often happens through a relationship that bridges two otherwise-authorized nodes. Validating `ALL(rel IN relationships(path))` closes that hole. **Row-level security, graph edition.** In relational systems, RLS attaches a predicate to every table access. The graph equivalent is a **default-deny traversal policy**: a query engine wrapper injects the tenant/clearance predicate into every read, and the agent literally cannot construct a traversal that escapes it. This is enforced server-side, inside the MCP tool implementation, so the LLM never sees — and can never request — data outside its boundary. **Field-level redaction.** Some edges should be traversable but not fully readable. A `SUPPLIES` edge might expose `supplier_name` to all tenants but gate `unit_cost` behind a clearance check. Return typed projections that omit sensitive properties rather than returning full nodes and trusting the serializer. **Multi-agent isolation with shared truth.** When agent swarms share a graph, isolate *write* domains: a research agent can write `RISK_FLAG` edges to nodes it's authorized for, but cannot mutate financial data. Read scope can be broader than write scope. Mirror this with per-tool capability scopes in the MCP surface — a tool like `aggregate_financials` simply isn't exposed to agents without the matching grant. **Auditability as a privacy control.** Log every traversal: who queried, which nodes and edges were touched, and why. This satisfies data-access regulations and gives you the forensic trail to detect anomalous traversal patterns — the graph equivalent of intrusion detection. The principle: **authorization is structural**. It lives in labels, predicates, and tool scopes, enforced inside the engine. Prompts can be injected and models can be confused; a server-side predicate cannot be talked out of its job. ## Case Study: Logistics Route-Planning with Graph-Backed Agent Swarms To make this tangible, consider a mid-size freight operator running a network of warehouses, carriers, and delivery commitments. They replaced a chunk-based RAG assistant with a graph-backed agent swarm. The graph models `Warehouse`, `Carrier`, `Route`, `Shipment`, `Customer`, and `Hub` nodes, with edges for `SERVES`, `CONNECTS`, `CARRIES`, `DEPENDS_ON`, and `COMMITTED_TO`. Vectors index every node's canonical text. **The swarm.** Three agent roles cooperate over the shared graph: a **Demand Agent** reads customer commitments and predicts volume; a **Risk Agent** monitors supplier and weather edges, flagging disruptions; and a **Routing Agent** proposes and re-plans routes. They don't message each other with prose — they *write to shared graph nodes*. The Risk Agent adds a `RISK_FLAG` edge on a `Hub`; the Routing Agent reads it and re-plans. Shared memory is the graph itself. **The multi-hop query in production.** When a port disruption hits, the Routing Agent runs the traversal: disrupted `Hub` → affected `Route`s → `Shipment`s → `Customer`s. A single hybrid query surfaces the exact list of at-risk deliveries with provable paths, in seconds. **The re-planning loop.** With the affected subgraph in context, the Routing Agent enumerates alternate `Route`s, scores them by cost, transit time, and carrier capacity (edge properties), and proposes a re-plan. A human dispatcher approves high-cost changes through a checkpoint — plan/act/reflect with a human in the loop, exactly the pattern we build into production MCP agents. **The outcome.** The operator reported that multi-hop queries which previously required a data analyst — "which customers are exposed if this hub fails?" — now resolve inside the assistant. Route re-planning latency dropped from hours of manual spreadsheet work to minutes, and every proposed change arrived with an auditable evidence path. Crucially, tenant isolation held: each customer tenant's `COMMITTED_TO` edges were filtered at the traversal layer, so carrier and pricing data never crossed boundaries. **Why the graph mattered.** No flat vector store could answer "trace the dependency chain from this hub to the affected customers," because the answer *is* a chain, not a set of similar passages. The graph didn't just improve retrieval quality — it made a whole class of reasoning structurally possible. ## Conclusion: Structure Is the Next Frontier of Agent Memory Flat vector retrieval was a breakthrough, and it remains the right tool for semantic entry. But agent memory is relational by nature, and relations are exactly what dense vectors discard. The architecture that holds up under real reasoning — and real compliance — pairs vectors for fuzzy entry with a knowledge graph for structured traversal: **entities as nodes, relationships as first-class edges, paths as auditable evidence, and authorization as a structural property of the graph itself**. If you are building agents that need to reason across connected facts — supply chains, customer graphs, internal knowledge — the flat index will hit a ceiling. The graph is what you build past it. --- **Ready to give your agents real memory?** OnlineCloud designs and ships MCP-native, graph-backed agent systems — from knowledge-graph modeling and hybrid retrieval to audit-ready privacy controls — in 6–10 weeks, not quarters. No offshore handoffs, no decks; production systems wired into the databases and tools you already run. **→ [Book a free consultation with OnlineCloud](https://www.onlinecloud.dev/)** — dedicated US-based engineering support, response within 24 hours. *Xlent Info Tech LLC · contact@onlinecloud.dev*

# Implementing “Self-Correction” Loops in Agentic Code Generation: Unit-Test Driven Repair


*By the engineering team at [OnlineCloud](https://www.onlinecloud.dev/) — an MCP-native studio shipping evaluated, observable agent systems.*

—

## 1. Introduction: Why One-Shot Code Generation Fails in Production

Every team that has shipped an LLM-backed coding feature has lived the same arc. The demo is electric: you paste a function signature, the model emits 40 lines of Python, and it runs. You wire it into a pipeline, point it at a Jira ticket, and two weeks later you are staring at a 3 a.m. incident caused by a generator that confidently returned a `datetime` where the downstream consumer expected a `str`.

One-shot generation fails in production for four reasons that have almost nothing to do with model quality.

**First, correctness is not a property of text — it is a property of execution.** A generated function is a hypothesis. Until it has been run against real inputs, covered by real assertions, and exercised against edge cases, it is prose with syntax highlighting. Models are trained to maximize the *plausibility* of the next token, not the *truth* of the program. Plausible code that references a function that does not exist, or that transposes two arguments, scores exactly as well during training as code that works.

**Second, context is lossy.** The prompt rarely contains the whole interface contract. Types are implicit, invariants live in a wiki, and the caller’s assumptions about nullability are undocumented. The model fills those gaps with the statistical average of GitHub — which is a reasonable prior and a terrible specification.

**Third, the error surface is combinatorial.** Off-by-one errors, empty-collection handling, timezone math, encoding, floating-point comparison, exceptions swallowed by a bare `except`. A single sampled response has to get *all* of them right simultaneously. The probability of a perfect first pass collapses exponentially with the number of independent decisions.

**Fourth — and this is the cultural failure — we treat generation as a write, when it should be a loop.** Human engineers do not write correct code in one pass. They write, run, read the traceback, and fix. Roughly 60–70% of a developer’s time is spent in exactly this read-run-repair cycle. The moment we give an agent the same loop, its functional correctness stops being a function of sampling luck and starts being a function of iteration budget.

The rest of this post walks through the architecture we use at OnlineCloud to build that loop: the Code-Test-Fix cycle, gVisor-backed sandbox execution, log-to-prompt error feedback, and a working LangGraph implementation.

—

## 2. The Self-Correction Loop: Implementing the Code-Test-Fix Cycle Autonomously

The mental model is a control system, not a pipeline. The agent is a controller; the test suite is the sensor; the sandbox is the plant.

“`
┌────────────────────────────────────────────────────────┐
│ │
▼ │
┌─────────┐ ┌──────────────┐ ┌────────────────┐ ┌──────────────┐
│ PLAN │───▶│ GENERATE │───▶│ EXECUTE │──▶│ EVALUATE │
│ (spec) │ │ (write code │ │ (gVisor sandbox│ │ (parse logs, │
│ │ │ + tests) │ │ lint + pytest)│ │ decide) │
└─────────┘ └──────────────┘ └────────────────┘ └──────┬───────┘
▲ │
│ pass? ──▶ DONE (emit artifact) │
│ │
└────────── REPAIR (feed structured errors back) ◀──────┘
fail
“`

Four design rules make this loop converge instead of thrash.

**Rule 1 — Tests are generated before the implementation is trusted.** The agent writes the function *and* a test file, but only the tests survive into the next turn if they were authored from the specification, not from the implementation. Tests derived by reading your own code are worthless — they assert what you wrote, not what was required.

**Rule 2 — Every iteration must be *strictly* monotonic or the loop halts.** Track a fitness signal: `tests_passed`, then linter errors, then type errors. If iteration *n+1* does not improve on iteration *n*, you have a hallucination spiral. Stop and escalate rather than burning tokens.

**Rule 3 — Budget is a first-class parameter.** Cap iterations (we default to 4–5), cap wall-clock per execution, cap tokens per repair prompt. Unbounded loops are how a $0.02 task becomes a $9.00 task.

**Rule 4 — The loop must leave an audit trail.** Every iteration’s code, logs, and diffs are persisted. When the agent finally passes, you want to know *what it changed and why* — both for review and as fine-tuning data.

The convergence math is optimistic: if a single pass has a ~55% chance of being fully correct and the loop has a ~70% chance of fixing a *detected* failure, four iterations push you past 90% functional correctness on a well-specified task. Section 6 puts real numbers on this.

—

## 3. Sandbox Execution: Running Generated Code and Unit Tests in Isolated Environments

You are about to execute arbitrary, model-authored code on your infrastructure. Treat that code as hostile, because eventually it will be — whether from a hallucinated `shutil.rmtree(‘/’)`, a runaway `while True`, a fork bomb, or a prompt-injected dependency that exfiltrates an env var.

Containers alone are not enough. The Docker daemon and its shared host kernel are a large attack surface, and a single kernel CVE is a full escape. For untrusted code execution we use **gVisor**.

### Why gVisor

gVisor (`runsc`) implements the Linux syscall surface in userspace. The containerized application talks to a *Sentry* process that emulates the kernel, with a separate *Gofer* mediating filesystem access. The untrusted workload’s syscalls never hit the host kernel directly, which removes the entire class of kernel-exploit escapes and dramatically shrinks the attack surface — while still giving you a normal container UX.

“`bash
# Run untrusted generated code under gVisor with hard limits
docker run –runtime=runsc \
–network=none \
–read-only \
–tmpfs /work:rw,size=64m,noexec,nosuid \
–memory=512m –cpus=1.0 –pids-limit=128 \
–security-opt=no-new-privileges \
–cap-drop=ALL \
-v “$PWD/candidate.py:/work/candidate.py:ro” \
-v “$PWD/test_candidate.py:/work/test_candidate.py:ro” \
sandbox-python:3.12 \
timeout -s KILL 30 pytest -q /work –tb=short
“`

The flags matter as much as the runtime:

– `–network=none` — no egress, so no exfiltration and no supply-chain fetch at runtime.
– `–read-only` + `tmpfs` — the root filesystem is immutable; scratch space is a small, non-executable tmpfs.
– `–pids-limit` — kills fork bombs before they take the host down.
– `–memory` / `–cpus` — a deterministic ceiling, not best-effort.
– `timeout -s KILL 30` — SIGKILL, not SIGTERM. Generated code with signal handlers cannot opt out.
– `–cap-drop=ALL` + `no-new-privileges` — strips the ability to re-escalate.

### Operational shape

At OnlineCloud we run a pool of pre-warmed sandboxes behind an internal service. A request looks like `{files, entrypoint, timeout_s, limits}`, and the response is `{exit_code, stdout, stderr, duration_ms, oom_killed, truncated}`. Pre-warming matters: cold-starting a language toolchain adds seconds, and multi-iteration loops multiply that. We keep N warm gVisor sandboxes per language and recycle each one after every job so no state leaks between candidates.

Isolation is also a *correctness* feature, not just security. A hermetic sandbox with a pinned toolchain version means a red test means the code is wrong — not that the environment drifted.

—

## 4. Error Feedback: Parsing Linter and Test-Runner Logs into Actionable Model Prompts

Here is where most self-correction loops quietly fail. Teams dump the raw 4,000-line pytest traceback into the next prompt, the model gets distracted by stack frames from its own test harness, and it “fixes” the wrong thing.

**Never forward raw logs. Distill them into a structured repair packet.**

Three parsers sit between the runner and the model:

1. **Test parser** — extracts, per test: node id, outcome, and the *assertion* line rather than the full trace. We use pytest’s `–json-report` plugin (or a `pytest_runtest_logreport` hook) to get structured results instead of scraping text.
2. **Linter parser** — Ruff / ESLint / mypy in JSON mode. Each diagnostic becomes `{file, line, col, code, message}`.
3. **Diff-aware locator** — maps each failure back to the exact span in the candidate file, so the model edits 3 lines instead of rewriting the module.

The distilled packet is small, deterministic, and ordered by severity:

“`python
# Structured feedback handed back to the model — NOT the raw traceback.
repair_packet = {
“iteration”: 2,
“summary”: “2 of 5 tests failing; 1 lint error; 0 type errors”,
“priority”: “tests”,
“failures”: [
{
“test”: “test_parse_duration_rejects_units”,
“line”: 17,
“assertion”: “assert parse_duration(‘5m’) == 300”,
“actual”: “returned 5”,
“hint”: “unit suffix is not being converted to seconds”,
}
],
“diagnostics”: [
{“file”: “candidate.py”, “line”: 9, “col”: 1, “code”: “F401”,
“message”: “‘math’ imported but unused”}
],
“instructions”: (
“Fix ONLY the failing assertions above. Preserve the public ”
“signatures. Do not modify the tests. Return the full file.”
),
“budget”: {“iteration”: 2, “max_iterations”: 5},
}
“`

Four principles govern the prompt discipline:

– **Assertion over traceback.** The assertion encodes intent; the traceback encodes plumbing. Give the model the former.
– **Actual vs. expected, both quoted.** Half of all repair failures come from the model guessing the wrong direction of a comparison.
– **Constrain the edit.** “Fix only these assertions; do not change the tests” prevents the classic degenerate strategy of *making the test pass by editing the test*.
– **Feed the loop state back in.** Showing `iteration: 2 / 5` measurably reduces repeated failed strategies and stops the model re-proposing the change that already failed.

The single highest-leverage line in the whole system is `instructions`. Treat the repair prompt as an API contract, not a chat message.

—

## 5. Implementation: A Self-Correcting Coding Agent in Python with LangGraph

Below is the skeleton we use, trimmed for clarity. The graph has five nodes with one conditional edge — the loop itself.

“`python
from __future__ import annotations
import json, subprocess, tempfile, textwrap
from typing import TypedDict, Literal
from langgraph.graph import StateGraph, START, END
from langchain_core.messages import SystemMessage, HumanMessage

MAX_ITERATIONS = 5

class AgentState(TypedDict):
spec: str
code: str
tests: str
iteration: int
status: Literal[“ok”, “fail”]
feedback: dict # the distilled repair packet
history: list[dict] # audit trail

# —————————————————————- GENERATE
def generate(state: AgentState) -> AgentState:
if state[“iteration”] == 0:
prompt = f”””You are a senior Python engineer.
Write (a) the implementation and (b) pytest tests for this spec:

SPEC: {state[‘spec’]}

Return STRICT JSON: {{“code”: “…”, “tests”: “…”}}.
Tests must assert the spec, not the implementation.”””
else:
prompt = f”””Your previous attempt failed. Iteration {state[‘iteration’]}.

REPAIR PACKET:
{json.dumps(state[‘feedback’], indent=2)}

PREVIOUS CODE:
{state[‘code’]}

Rules: fix ONLY the reported failures. Never edit the tests.
Return STRICT JSON: {{“code”: “…”, “tests”: “…”}}.”””

raw = call_model([SystemMessage(content=”You output only valid JSON.”),
HumanMessage(content=prompt)])
payload = json.loads(extract_json(raw))
return {**state, “code”: payload[“code”],
“tests”: state[“tests”] or payload[“tests”],
“iteration”: state[“iteration”] + 1}

# —————————————————————- EXECUTE
def execute(state: AgentState) -> AgentState:
“””Runs in a gVisor sandbox: no network, read-only root, hard limits.”””
result = run_in_sandbox( # returns exit_code/stdout/stderr
files={“candidate.py”: state[“code”], “test_candidate.py”: state[“tests”]},
entrypoint=”pytest -q –json-report –json-report-file=/work/report.json”,
timeout_s=30, memory=”512m”, network=False,
)
return {**state, “feedback”: parse_feedback(result)}

# —————————————————————- EVALUATE
def evaluate(state: AgentState) -> Literal[“repair”, “done”, “give_up”]:
packet = state[“feedback”]
if packet[“summary”].startswith(“0 failed”) and not packet[“diagnostics”]:
return “done”
if state[“iteration”] >= MAX_ITERATIONS:
return “give_up”
# monotonicity guard: no progress two turns in a row -> stop
h = state[“history”]
if len(h) >= 2 and h[-1][“passed”] <= h[-2]["passed"]: return "give_up" return "repair" # ---------------------------------------------------------------- FEEDBACK def parse_feedback(result: dict) -> dict:
report = json.loads(result[“stdout_report”]) # pytest –json-report
failures = [
{“test”: t[“nodeid”], “line”: t[“line”],
“assertion”: t[“assertion”], “actual”: t.get(“actual”)}
for t in report[“tests”] if t[“outcome”] == “failed”
]
diagnostics = run_ruff_json(result) # ruff –output-format json
passed = len(report[“tests”]) – len(failures)
return {“summary”: f”{len(failures)} failed / {passed} passed”,
“failures”: failures, “diagnostics”: diagnostics,
“instructions”: “Fix ONLY the failing assertions. Do not edit tests.”}

def run_in_sandbox(*a, **k) -> dict:
return subprocess.run( # simplified
[“docker”, “run”, “–runtime=runsc”, “–network=none”, “–read-only”,
“–memory=512m”, “–cpus=1.0”, “–pids-limit=128”, “–cap-drop=ALL”,
“sandbox-python:3.12”, “python”, “-m”, “runner”],
capture_output=True, text=True, timeout=k[“timeout_s”] + 5,
).__dict__

# —————————————————————- GRAPH
g = StateGraph(AgentState)
g.add_node(“generate”, generate)
g.add_node(“execute”, execute)
g.add_node(“repair”, lambda s: s) # repair == re-enter generate
g.add_edge(START, “generate”)
g.add_edge(“generate”, “execute”)
g.add_conditional_edges(“execute”, evaluate,
{“repair”: “generate”, “done”: END, “give_up”: END})
app = g.compile()
“`

The details that matter in production:

– The `monotonicity guard` in `evaluate` is what stops the infinite loop when the model fixates on a wrong hypothesis.
– `state[“tests”]` is only set on iteration 0 — a deliberate one-way latch so repair turns cannot rewrite the oracle.
– `run_in_sandbox` is the only place untrusted code is ever touched; it is a hard process boundary, and every call is logged with its resource limits.
– The `history` list is the audit trail and the future fine-tuning corpus.

Everything above is model-agnostic. Swap `call_model` for Claude, a GPT-4-class model, or an open-weights deployment running in your own VPC — the loop is the asset, not the model.

—

## 6. ROI: Improving Functional Correctness by 40% Through Iterative Refinement

Across the internal tasks we benchmarked — data-transform helpers, API clients, parsers, and business-rule validators — moving from one-shot generation to a bounded 5-iteration Code-Test-Fix loop with gVisor execution produced a **~40% improvement in functional correctness on the first shipped artifact**, measured as pass rate on a held-out test suite the agent never saw.

The breakdown matters more than the headline:

| Configuration | Held-out pass rate | Notes |
|—|—|—|
| One-shot, no execution | ~48% | “Looks right” only |
| One-shot + sandboxed tests | ~61% | Detection alone helps |
| Full loop (≤5 iters), raw logs | ~70% | Noisy repair signal |
| Full loop + distilled repair packets | **~84%** | ≈40% relative gain |

Three lessons from the data:

1. **Detection beats generation.** ~13 of the 40 points come from *running* the code, before any repair happens. You cannot fix what you cannot see fail.
2. **Feedback quality dominates iteration count.** Going from raw logs to distilled repair packets was worth more than doubling the iteration budget. Signal quality beats brute force.
3. **The guard rails pay for themselves.** The monotonicity guard and the iteration cap meant the median successful task converged in **2 iterations**; only 9% hit the cap. Latency rose modestly (p50 ~14s per task end-to-end) while human review time dropped sharply, because reviewers were handed a passing artifact *plus* the audit trail of how it got there.

Cost, honestly: token spend per task rose ~2.4× versus one-shot. That is the trade. You are buying correctness with iterations, and at ~$0.01–0.05 of extra inference per task, it is the cheapest QA engineer you will ever hire.

The strategic point is not the 40%. It is that the loop turns correctness from a *probability you sample* into a *property you engineer* — and that is the difference between a coding demo and a production system.

—

## Ship self-correcting agents with OnlineCloud

If your team is trying to get LLM-generated code into production and hitting the wall between “works in the notebook” and “passes CI,” this is exactly the kind of system we build. OnlineCloud is a US-based, senior-only engineering studio specializing in MCP-native agent architectures, sandboxed execution, and evaluated, observable AI systems — shipped into your VPC, wired to your existing stack, in 6–10 weeks rather than quarters.

**We don’t sell decks — we ship evaluated, observable systems.**

– Design a Code-Test-Fix loop tailored to your codebase and eval suite
– Stand up gVisor-backed sandbox execution with real resource limits and audit trails
– Build the log-distillation layer that makes repair prompts actually converge
– Instrument correctness, latency, and cost-per-task so you can prove the ROI

**[Book a free initial consultation →](https://www.onlinecloud.dev/)** — reach us at **contact@onlinecloud.dev**. Response within 24 hours.

*Note: the benchmark figures above are from OnlineCloud’s internal task suites and will vary with model choice, task complexity, and specification quality. We will reproduce the measurement on your data before you commit to anything.*