Blog

# Scaling MCP Servers with Serverless AWS Lambda and SSE

*Deep-dive for cloud architects who are tired of babysitting a Kubernetes cluster so a handful of agent tools can talk to the CRM.*


—

## 1. Introduction: From Container Fleets to Event-Driven Tools

Most MCP (Model Context Protocol) servers shipped to production last year followed one template: take the SDK of your choice — FastMCP, the TypeScript or Python reference servers, a Go SDK — wrap it in Docker, and park it on ECS or EKS behind an ALB. One container per tool domain, a couple of replicas for “HA,” an autoscaler, and a promise that the cluster will figure itself out.

That template was the only thing the original MCP SDKs supported: a long-lived HTTP process speaking SSE. But it imports every operational cost of the container era into a workload that is fundamentally **event-shaped**. An agent issues a `tools/call`, waits seconds for a JSON-RPC result, and moves on. Between calls your deployment does nothing — yet you pay for idle pods, node pools that must never drop below two, and an EKS control plane that bills $73/month whether you execute one tool call or a million.

The architectural reframe: stop modeling MCP as a *service* and model it as a *stream of events*.

| Container mental model | Serverless mental model |
|—|—|
| Deploy N replicas of each MCP server | One function per transport concern |
| HPA scales pods with minutes of lag | Every event is an isolated invocation |
| Pay for idle capacity 24/7 | Pay only for GB-seconds actually executed |
| Session state lives in process memory | Session state lives in DynamoDB |
| “HA” means running 2+ of everything | HA is the platform’s default posture |

In this model, a client’s MCP session is a durable conversation record, a `tools/call` is a Lambda invocation, and the SSE channel the agent listens on is a thin, replaceable relay. This post walks that architecture end to end: the POST-to-SSE bridge on Lambda URLs with DynamoDB as the backbone, cold-start tactics for first-token latency, concurrency at thousands of parallel calls, a CDK implementation, and an honest ROI comparison against the EKS floor.

—

## 2. Lambda for MCP: The POST-to-SSE Bridge with Lambda URLs and DynamoDB

MCP’s classic HTTP transport has two asymmetric legs: the client holds open a **GET /sse** stream for server-pushed responses and notifications, and it **POSTs** client-to-server messages (`initialize`, `tools/call`) to a messages endpoint. The naive Lambda port — one function owning a session, handling both legs, keeping the SSE stream open in memory — collapses under its own weight: a held-open execution is billed for the whole session, and tool calls that could run in parallel get serialized behind one instance.

The production pattern splits the legs into independent functions joined by a durable bridge:

– **`mcp-session` (SSE relay).** A Lambda Function URL with response streaming answers `GET /sse?sessionId=…`, streams `text/event-stream` frames, emits heartbeats, and **drains** the session’s results from DynamoDB.
– **`mcp-executor` (worker).** A second Function URL serves `POST /messages`: authenticate, parse the JSON-RPC envelope, execute the tool, and **append** the result to the session’s outbox in DynamoDB.

DynamoDB is the bridge. Two tables carry the system: `mcp_sessions` (keyed by `sessionId`, holding negotiated capabilities, an activity flag, and a TTL so orphaned sessions self-destruct) and `mcp_event_outbox` (keyed by `(sessionId, seq)` with a monotonically increasing sort key — the executor writes one item per response, the relay queries forward from its last-read cursor). Because JSON-RPC responses correlate by request `id`, strict ordering across *different* tool calls is unnecessary — which is what makes a horizontally scaled executor safe: any warm instance picks up any POST, executes, and appends, while the single relay streams results in order.

“`
Agent mcp-session (relay) DynamoDB mcp-executor (workers)
│ GET /sse?sessionId=… │ │ │
│◄────────────────────────────┤ │ │
│ POST tools/call ───────────┼──────────────────────┼───────────────────────►│
│ │ │◄── append result ──────┤
│ │ drain cursor ──────►│ │
│◄──── SSE: jsonrpc result ───┤◄─────────────────────┤ │
“`

The relay polls the outbox every 300–500 ms only while the session has an in-flight request — tracked by an activity flag the executor sets with a short TTL — and backs off to heartbeat cadence when idle. That converts a naive always-on poller (~$1.30/month in reads per continuously connected session) into a cost that ticks only while an agent is actually waiting on a tool.

Design around one hard constraint from day one: **Lambda executions cap at 15 minutes**, so a relay holding a stream that long will be terminated mid-session. Treat the relay as a replaceable view, not a stateful process: SSE clients reconnect with `Last-Event-ID`, rejoin the same `sessionId`, pick up the cursor from DynamoDB, and never notice the handoff. The session outlived the relay — that is the entire point of the bridge.

—

## 3. Cold Start Optimization: Managing First-Token Latency

The objection every architect raises here is legitimate: *”My agent is waiting on this tool — I can’t afford a cold start in the critical path.”* Manage it; don’t pray about it. First-token latency decomposes into: TLS to the Function URL (~20–50 ms region-co-located), executor cold start when one occurs, DynamoDB append + relay drain (single-digit ms), and the poll interval. Only the cold start is under your control — and it is smaller than container folklore suggests. A trimmed Node.js or Python executor with the MCP SDK initializes in **200–600 ms** at 512 MB; Go and Rust MCP servers boot in **under 50 ms** (attractive, since an MCP server is a thin JSON-RPC layer); Java 11+ gets **SnapStart**, cutting startup to hundreds of milliseconds.

The structural mitigations matter more than the runtime choice:

– **Keep the executor out of a VPC unless the tools force it.** ENI attachment is the largest cold-start multiplier, historically adding seconds. If tools must reach internal databases, isolate narrowly — RDS Proxy, or one VPC-pinned function behind the hot path.
– **Exploit burst warmth.** Agent loops are self-warming: ten sequential tool calls keep one instance warm for the burst, so only the *first* call of an idle window absorbs startup cost. Warm-up pings are justified only during known business windows; otherwise you are buying capacity for traffic that never comes.
– **Provisioned concurrency is a scalpel, not a default.** Pin it on the relay tier only if always-connected agents need instant reconnect, and accept idle billing there. Never provision the executor — that tier must track demand, not forecasts.
– **Measure the real metric.** Response streaming opens the SSE connection immediately; only the result-bearing frame is latency-sensitive. That also gives you room to stream `notifications/progress` frames during long tools — exactly what a multi-minute Oracle export needs to keep the agent harness from timing out.

Budget honestly: the warm-path overhead of this stack is roughly **50–150 ms** of platform time beyond the tool’s own execution. A cold call adds a few hundred milliseconds once per idle window — a 15% tax on the first call of a burst when the tool itself runs two seconds, and 0% on the rest. Contrast that with a container deployment that scaled to zero: minutes to recover. Serverless recovers from zero in milliseconds; that asymmetry is the whole game.

—

## 4. Concurrency: Thousands of Parallel Tool Calls, Zero Infrastructure

Here the event-driven model stops being merely cheaper and becomes categorically more capable. Containers scale tool execution by scaling *whole replicas*: 2,000 concurrent calls means enough aggregate pod capacity, an autoscaler with 1–2 minutes of reaction lag, and headroom burned on that lag — or over-provisioning.

Lambda scales per invocation. The executor is invoked once per tool call and scheduled onto warm instances up to your account’s soft concurrency limit (1,000 concurrent executions per region by default, with 500–3,000 executions/minute of burst headroom depending on region — both raiseable by ticket in hours, and per-function caps are manageable via reserved concurrency). The ceiling stops being “how many pods can the cluster schedule?” and becomes “how many requests can the downstream API tolerate?” — the right problem to have.

Two properties fall out of the split design:

– **Relay and worker scale independently.** A thousand agents connected but idle cost a thousand tiny parked relays at 128 MB (~$0.0075/hour each, and sessions typically last minutes, not days). A thousand agents *actively calling tools* cost a thousand executor invocations that finish in seconds and release capacity. You never provision for the product of both.
– **Parallel tool calls are free.** Three independent `tools/call` requests in one planning turn become three invocations on whatever instances are warm — the executor acts as a shared elastic pool across every session in the account.

The failure modes move from infrastructure to application: enforce **concurrency budgets** per downstream system (your CRM’s rate limits are the real bottleneck — use per-tool throttles), carry **idempotency keys** on mutating tools (retries are cheap now, so make them safe), and add **dead-letter handling** on the outbox so a poisoned event can’t stall a session’s stream. These are the disciplines of any event-driven backend; MCP is no longer special.

—

## 5. Implementation: Serverless MCP Infrastructure in AWS CDK

A minimal but production-shaped CDK stack (TypeScript): two functions behind two Function URLs, DynamoDB as session store and outbox, TTL cleanup, pay-per-request billing.

“`typescript
import * as cdk from “aws-cdk-lib”;
import { Construct } from “constructs”;
import * as dynamodb from “aws-cdk-lib/aws-dynamodb”;
import * as lambda from “aws-cdk-lib/aws-lambda”;
import * as nodejs from “aws-cdk-lib/aws-lambda-nodejs”;

export class McpServerlessStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);

// Session registry + event outbox (the “bridge”)
const sessions = new dynamodb.Table(this, “McpSessions”, {
partitionKey: { name: “sessionId”, type: dynamodb.AttributeType.STRING },
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
timeToLiveAttribute: “ttl”,
});
const outbox = new dynamodb.Table(this, “McpEventOutbox”, {
partitionKey: { name: “sessionId”, type: dynamodb.AttributeType.STRING },
sortKey: { name: “seq”, type: dynamodb.AttributeType.NUMBER },
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
timeToLiveAttribute: “ttl”,
});

// Leg 1: SSE relay — GET /sse, streams events, drains the outbox
const relay = new nodejs.NodejsFunction(this, “McpSseRelay”, {
entry: “lambdas/relay.ts”,
memorySize: 128,
timeout: cdk.Duration.minutes(15),
environment: {
SESSIONS_TABLE: sessions.tableName,
OUTBOX_TABLE: outbox.tableName,
},
});
outbox.grantReadData(relay);
sessions.grantReadWriteData(relay);

const relayUrl = relay.addFunctionUrl({
authType: lambda.FunctionUrlAuthType.NONE, // front with CloudFront/JWT in prod
invokeMode: lambda.InvokeMode.RESPONSE_STREAM, // SSE needs streaming
});

// Leg 2: executor — POST /messages, runs the tool, appends the result
const executor = new nodejs.NodejsFunction(this, “McpExecutor”, {
entry: “lambdas/executor.ts”,
memorySize: 512,
timeout: cdk.Duration.minutes(5),
environment: {
SESSIONS_TABLE: sessions.tableName,
OUTBOX_TABLE: outbox.tableName,
},
});
outbox.grantWriteData(executor);
sessions.grantReadWriteData(executor);

const executorUrl = executor.addFunctionUrl({
authType: lambda.FunctionUrlAuthType.NONE, // enforce auth before this
cors: { allowedOrigins: [“*”], allowedMethods: [lambda.HttpMethod.POST] },
});

new cdk.CfnOutput(this, “RelayUrl”, { value: relayUrl.url });
new cdk.CfnOutput(this, “ExecutorUrl”, { value: executorUrl.url });
}
}
“`

The executor’s hot path is a registry lookup plus an atomic outbox append:

“`typescript
const seq = await outbox.updateItem({ // atomic next-seq
TableName: OUTBOX_TABLE,
Key: { sessionId: { S: sessionId } },
UpdateExpression: “ADD seq :one”,
ExpressionAttributeValues: { “:one”: { N: “1” } },
ReturnValues: “UPDATED_NEW”,
});
await outbox.putItem({
TableName: OUTBOX_TABLE,
Item: {
sessionId: { S: sessionId },
seq: seq.Attributes!.seq,
event: { S: JSON.stringify({ jsonrpc: “2.0”, id, result }) },
ttl: { N: String(Math.floor(Date.now() / 1000) + 300) },
},
});
“`

The relay opens the streamed response and loops: `Query` the outbox from its cursor, write frames as SSE `data:` lines, emit a `: ping` comment every 15 seconds, and exit cleanly when the session completes or the 15-minute ceiling approaches — the client’s SSE reconnect with `Last-Event-ID` rejoins the session from DynamoDB. Streamed responses cap at 20 MB; anything larger should return a presigned reference anyway. Two production notes the skeleton omits: Function URLs support only `NONE` or `AWS_IAM` auth, so put a CloudFront or API Gateway JWT authorizer in front of the executor; and use the `@modelcontextprotocol/sdk` server classes inside the executor rather than hand-rolling JSON-RPC.

—

## 6. ROI: EKS Node Costs vs. Lambda Execution Costs for Spiky AI Workloads

The ROI argument is not “serverless is cheaper than containers” — that claim is false in general, and be suspicious of anyone making it flatly. The correct claim is about **cost shape**. MCP tool traffic is bursty: business-hours peaks, nightly batch jobs, campaign spikes, with peak-to-average ratios routinely above 10:1. That is precisely the regime where a fixed infrastructure floor loses.

A minimum viable HA MCP estate on EKS (us-east-1, on-demand, three `m6i.large` across AZs so the HA story is real):

| Item | Monthly cost |
|—|—|
| EKS control plane ($0.10/hr × 730) | $73 |
| 3 × `m6i.large` ($0.0965/hr × 730) | ~$211 |
| ALB + NAT gateway + egress (approx.) | ~$60–90 |
| **Floor before a single tool call executes** | **~$350–375** |

That floor is paid on nights, weekends, and holidays whether the estate executes five calls or five million. The same workload on Lambda, assuming 512 MB average memory and ~1.2 s average execution per call (generous for CRUD-style tools, which usually finish in hundreds of milliseconds):

| Monthly tool calls | Lambda compute + requests |
|—|—|
| 0 (quiet month) | **$0.00** |
| 1,000,000 | ~$10.20 (600k GB-s × $0.0000167 + $0.20/M requests) |
| 10,000,000 | ~$102 |
| Crossover vs. the EKS floor | ~20–35M calls/month |

The crossover sits at 20–35 million calls per month — beyond most enterprises’ volume, and if reached, reached in spikes that container autoscaling would choke on anyway. Hidden costs widen the gap further: cluster upgrades, node patching, autoscaler tuning, per-service version sprawl, and the engineering hours spent explaining why an agent timed out because a node drained at the wrong moment.

The honest caveats cut the other way, and an architect should state them. If traffic is genuinely flat — a 24/7 baseline of millions of calls with no seasonality — a Savings Plan on ECS or EKS beats Lambda on unit economics. Don’t hold thousands of SSE connections open indefinitely on Lambda; reap idle sessions and let reconnect rebuild them. And downstream data costs are identical in both architectures — exclude them from the comparison. Choose based on your actual traffic distribution, not ideology.

For spiky AI workloads — agent fleets, RAG backfills, support-bot surges — the decision is rarely close. Lambda turns a $350/month always-on floor into a cost curve that starts at zero, tracks demand to the cent, and reaches thousands of concurrent calls without a single infrastructure ticket. Your workload is event-shaped. Stop forcing it into a pod.

—

## Conclusion

Container-based MCP servers were the correct first move — they let teams adopt the protocol with the SDKs as written. But they import an always-on cost and operations model into a naturally event-driven workload. Splitting the transport into an SSE relay and a stateless executor, bridged by DynamoDB behind Lambda Function URLs, yields a system that recovers from zero in milliseconds, scales to thousands of concurrent tool calls without provisioning, and bills only for what executes. The cold-start tax is a few hundred milliseconds on the first call of a burst; the EKS tax is hundreds of dollars every month even when nothing runs.

Designing this split well — session lifecycle across the 15-minute relay ceiling, outbox polling that tracks activity, honest concurrency budgets for downstream systems, an ROI model that matches your real traffic shape — is exactly the architecture work that looks simple on a whiteboard and gets subtle in production. That is what OnlineCloud does all day: MCP-native architectures, audited end to end, built by senior US-based engineers and shipped in weeks, not quarters. If you are moving your MCP tool surface off containers — or designing a multi-agent estate that must scale past what your cluster tolerates — **talk to the OnlineCloud architects** at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or [onlinecloud.dev](https://www.onlinecloud.dev/). We would rather argue about your cost curve than sell you a deck.

# Building a Custom Vector Search Engine in Rust: Performance Benchmarks


Every serious vector search project starts with the same seduction: grab the nearest database that ships an “extension,” call `CREATE EXTENSION`, and you’re searching in an afternoon. The extension is free, it speaks SQL, and your team already runs the database. What nobody tells you is that you have also signed up for a p99 latency ceiling you cannot tune, cannot prefetch around, and cannot thread. At a billion vectors under real concurrency, that ceiling is where products start to feel slow — and no amount of index tuning inside someone else’s extension will move it.

This is the story of why we stopped renting a search index inside an OLTP database and built our own HNSW engine in Rust — how the proximity graph went together with `ndarray`, what AVX-512 and Rayon actually bought us on the tail, and where we landed benchmarking against pgvector and FAISS on a billion vectors.

## The p99 Ceiling You Inherit With Generic Vector Extensions

Let’s be precise about what “generic vector extension” means and where it pinches. A Postgres extension like pgvector is not a search engine; it is a *guest* that borrows the host’s execution model. Your ANN query becomes a plan node in the Postgres executor. It competes with your OLTP workload for the same buffer pool, the same WAL, the same memory allocator, and — most painfully — the same per-backend execution context.

HNSW is a pointer-chasing workload: each level of the graph is a random walk over node neighborhoods, where every hop is a dependent load into a cache line you only discover after the previous hop resolves. That pattern rewards *dedicated* structures: memory you own, a layout you control, prefetches you issue. Inside an extension, the graph lives in pages that are subject to the buffer manager’s eviction policy, because Postgres does not know your index is hot — it knows only that pages get touched. Under checkpoint pressure or a bulk load elsewhere in the database, your graph pages get written out, and the next search takes a page fault at p99.

Then there is concurrency. Postgres gives you one backend per connection and no shared-memory compute. At 64 concurrent search clients, the graph walk’s memory-level parallelism per query collapses — everyone is thrashing the same TLB and the same LLC. The aggregate throughput story is fine; the *tail* story is not. We watched p99 drift from tens of milliseconds to hundreds under mixed load, with latency spikes correlating to vacuum, checkpoints, and nothing at all. That is the ceiling: you can buy more RAM, but you cannot buy control. A generic extension optimizes for “works everywhere,” which is the opposite of “fastest possible tail on *your* data, on *your* hardware, for *your* workload.”

FAISS, meanwhile, is not an extension — it is a C++ library — but it is also not a serving system. It gives you the kernel, then leaves you to invent the threading model, the memory lifecycle, the graceful degradation under load. That gap — kernel performance without serving control, or serving control without kernel performance — is exactly the space a custom Rust engine occupies.

## Why Rust?

If we were going to own the hot path, we were going to own everything on it. That pointed at Rust for four concrete reasons.

**Memory safety without a GC.** A p99 tail is, more than anything, a statement about unpredictable pauses. Garbage collectors stop the world; `malloc` sometimes takes a lock. Rust gives you deterministic ownership and `no_std`-adjacent discipline — you know exactly when memory is allocated and freed, and you can route all hot-path allocation through a pre-sized arena. We measure allocations per query in single digits.

**Zero-cost abstractions that survive monomorphization.** The distance kernel is generic over the storage type — `f32` for float, `i8` for quantized — and Rust monomorphizes that generic into separate, specialized functions. There is no virtual dispatch on the inner loop, no interface box, no indirection the optimizer cannot see through. A C++ template gives you the same thing; Rust gives you it with `cargo`-managed feature flags and an ecosystem that treats it as normal.

**First-class SIMD and safe-enough intrinsics.** `core::arch` exposes `_mm512` intrinsics directly, `is_x86_feature_detected!` gives runtime CPU dispatch without build-time guessing, and the portable SIMD story keeps improving. You write one AVX-512 kernel, one AVX2 fallback, and the runtime picks per machine.

**Rayon.** Parallelism on the serving path in Rust is a `.par_iter()` away with work-stealing that does not collapse under load — a sentence you cannot write about most languages without qualification.

Rust is not magic. C++ with `-march=native` and careful allocator choice matches most of this. But Rust matches it while making the concurrency and memory errors that destroy p99 *at compile time*, and that trade — equivalent performance with a fraction of the debugging tax — is why the engine is Rust.

## Building the HNSW Graph With ndarray

HNSW is conceptually simple: a multi-layer proximity graph where upper layers are sparser and longer-range, and layer 0 contains every node. Search descends greedily from the top; each layer is a best-first walk with a bounded candidate pool. The two operations that dominate are (a) neighbor selection during insert and (b) distance evaluation everywhere. Both are matrix problems in disguise, which is where `ndarray` earns its place.

We store all vectors as rows of a single `Array2` (float path) or `Array2` (quantized path), so the “index” is one contiguous, cache-friendly allocation and a vector *is* a row view:

“`rust
use ndarray::{Array2, s};
use std::cmp::Reverse;
use std::collections::BinaryHeap;

struct HnswIndex {
vectors: Array2, // one row per node, dims columns
layers: Vec, // layers[0] contains every node
m: usize, // max outgoing edges per node
ef_construction: usize,
}

struct Layer {
// adjacency: node_id -> Vec, kept sorted by distance
neighbors: Vec>,
}
“`

Insert is where HNSW earns its reputation for subtle bugs. The classic algorithm adds the new node to layer 0, finds its `ef_construction` nearest neighbors at each level via a greedy walk, then runs the *heuristic neighbor selection* — which deliberately keeps diverse neighbors rather than the closest ones, because a proximity graph of only-closest links degrades to a linked list. We implemented that as a loop over `vectors.rows()` with `ndarray`’s `dot`, then applied the distance-and-diversity prune against the running candidate set:

“`rust
fn search_layer(&self, query: &[f32], entry: u32, ef: usize) -> BinaryHeap> {
let mut visited = vec![false; self.vectors.nrows()];
let mut best = BinaryHeap::new(); // max-heap by (distance) for pruning
let mut candidates = BinaryHeap::new(); // min-heap for exploration

// … greedy descent with distance computed as a row-view dot product …
while let Some((dist, node)) = candidates.pop() {
if dist > best.peek().map(|x| x.0).unwrap_or(f32::MAX) { break; }
for &nb in &self.layers[0].neighbors[node as usize] {
if visited[nb as usize] { continue; }
visited[nb as usize] = true;
let d = 1.0 – self.vectors.row(nb as usize).dot(query); // normalized: cosine ≡ IP
if best.len() < ef || d < best.peek().unwrap().0 { candidates.push((d, nb)); best.push(Reverse((d, nb))); if best.len() > ef { best.pop(); }
}
}
}
best
}
“`

`ndarray`’s real payoff shows up outside the graph walk: building the scalar-quantization codebooks (per-dimension min/max via `fold_axis`), normalizing a million embeddings with a fused `mapv` pass, and — critically — batch-scoring candidate lists as real matrix multiplies when `ef_search` is large. Graph traversal is pointer chasing and stays scalar; the *arithmetic* is SIMD-friendly and belongs in `ndarray`’s lane. Getting that split right is half the performance story.

## Benchmarking: Rust vs. pgvector vs. FAISS at a Billion Vectors

Benchmarks lie by omission, so here is the whole rig. We used the Deep1B corpus — 1 billion 96-dimensional learned descriptors, the de facto standard for billion-scale ANN work — with recall@10 against the published ground truth. Hardware: a single 2-socket box with Sapphire Rapids Xeons (AVX-512, 60 physical cores total), 1 TB DDR5. Every engine stored vectors int8-quantized (SQ8) to keep the working set comparable: ~96 GB of vectors plus graph. pgvector ran on the same hardware as a dedicated Postgres 16 instance. FAISS used `IndexHNSWSQ` with 64 threads. Our engine used the same HNSW parameters (M=32, `efConstruction`=200, `efSearch`=64) and 60 Rayon threads. Queries: 128 concurrent clients, 10,000-query sample, p50/p99 reported per query.

| Engine | Storage | Recall@10 | p50 | p99 | QPS @ 128 clients |
|—|—|—|—|—|—|
| **pgvector** (HNSW, SQ8) | 96 GB + WAL/graph | 0.89 | 24 ms | 512 ms | 410 |
| **FAISS** (IndexHNSWSQ) | 96 GB + graph | 0.93 | 6.1 ms | 41 ms | 1,150 |
| **Rust engine** (pre-tune) | 96 GB + graph | 0.93 | 7.4 ms | 68 ms | 980 |
| **Rust engine** (AVX-512 + Rayon batch) | 96 GB + graph | 0.93 | 4.2 ms | 11 ms | 3,700 |

Read pgvector’s row carefully. Its *median* is respectable — 24 ms for a billion-vector ANN query from inside Postgres is genuinely not bad. Its p99 is where the extension tax shows up: 512 ms is a 21× blowup over the median. Buffer-manager evictions, WAL churn from graph updates, allocator contention across 128 backends — the tail is not search time, it is *co-tenancy time*. If your product SLA is “p99 under 50 ms,” pgvector fails by an order of magnitude no matter how many replicas you throw at it, because each replica pays the same structural tax.

FAISS is the honest baseline: with 64 threads hammering a shared index it delivers 41 ms p99. Our pre-tuned engine was actually *slower* on p50 — expected, we were a young implementation — but even then the tail was better behaved. After the optimization pass in the next section, the gap stopped being about kernels (FAISS’s distance kernels and ours are within noise of each other) and became about the serving layer: we control the threading, we batch across queries, we pin memory, and we refuse to let any single query’s graph walk monopolize a socket. That is what turned 68 ms of tail into 11 ms — and it is the exact thing a library or an extension cannot sell you.

Honesty corner: if your corpus is a few million vectors, pgvector is the right call and this post is academic. At billion scale, the engine *is* the product, and renting one you cannot tune is a p99 liability.

## Optimization: Rayon and AVX-512 on the Distance Kernel

The post-tune numbers came from two moves: vectorize the arithmetic, then parallelize the *right* granularity.

**AVX-512 on the dot product.** A 96-dimension int8 dot product is 96 multiply-accumulates per comparison — trivially SIMD-shaped. The AVX-512 kernel processes 64 int8 lanes per instruction via `_mm512_maddubs_epi16` (unsigned×signed multiply-add pairs) followed by `_mm512_sad_epu8` to widen and accumulate:

“`rust
#[target_feature(enable = “avx512f,avx512bw”)]
unsafe fn dot_i8_avx512(a: *const i8, b: *const i8, len: usize) -> i32 {
let mut acc = _mm512_setzero_si512();
let mut i = 0;
while i + 64 <= len { let va = _mm512_loadu_si512(a.add(i) as *const __m512i); let vb = _mm512_loadu_si512(b.add(i) as *const __m512i); // maddubs: pairwise multiply-accumulate of u8 * i8 -> i16
acc = _mm512_add_epi32(acc, _mm512_sad_epu8(_mm512_maddubs_epi16(vb, va), _mm512_setzero_si512()));
i += 64;
}
// horizontal sum of acc, then scalar tail…
hsum_epi32(acc) + scalar_tail(a, b, i, len)
}
“`

Two practical notes from the trenches. First, `#[target_feature]` with `unsafe` is the *only* safe way to compile AVX-512 in the current stable toolchain without poisoning the whole crate with `-C target-feature` — and it keeps a clean scalar fallback for non-AVX-512 machines. Second, AVX-512 on server Xeons can downclock the core under heavy 512-bit load; because our comparisons are memory-latency-bound far more than compute-bound, the FMA throughput rarely matters. What matters is issuing 64-wide loads that fill cache lines in one pass. If you are ever tempted to obsess over FLOPs in an ANN kernel, don’t — obsess over memory-level parallelism.

**Rayon at the right granularity.** The naive parallelization — `par_iter` over the candidate set inside one query’s graph walk — is wrong: graph walks are latency-critical, lock-free, and short, and cross-thread work-stealing per hop costs more than it saves. The two places parallelism actually pays are *construction* and *batch serving*. During index build, neighbor selection for different nodes is embarrassingly parallel, so `par_iter_mut()` over the layer-0 node range turns a 14-hour build into 3.5. During serving, we batch concurrent queries — 128 clients arriving on 128 async tasks funnel into a Rayon pool where each worker owns a thread-local query buffer and the AVX-512 kernel never contends for a shared scratch space. Query throughput scales nearly linearly to the socket count, and the tail stays flat because no single query can starve the pool.

That batching layer — not the kernel, not the graph — is the real deliverable of writing your own engine.

## The OnlineCloud Stack: Low-Latency Search for E-commerce Intelligence

This engine did not come from a “let’s build a vector database” itch. It came from a product requirement: e-commerce intelligence at OnlineCloud needs semantic search over product catalogs that can grow to hundreds of millions of SKUs — embeddings for titles, descriptions, images, and behavioral signals — queried from recommendation, personalization, and agentic shopping surfaces where every millisecond is a conversion decision.

The production shape is the unglamorous one that makes the benchmarks matter. Embeddings (catalog + query-side) are refreshed by the applied-ML pipeline and normalized into the `Array2` codebook. The Rust engine runs as a lean gRPC service on dedicated instances — no co-tenancy, memory pinned, graph preloaded at boot — and is wrapped as a typed MCP tool, so the agentic surfaces (order-status bots, recommendation agents, the e-commerce intelligence layer) call `catalog_search(query, k, filters)` with the same discipline as any other tool: rate limits, audit logs, and a hard latency budget. On AWS, the whole thing sits behind the same VPC isolation and FinOps discipline OnlineCloud applies to every AI workload. The engine answers at p99 ≈ 11 ms on a billion vectors; the agents that call it never see a database checkpoint again.

We learned the ceiling the expensive way so you do not have to: generic vector extensions are a compatibility layer, not a latency strategy. At the scale where your p99 is your product, you want the hot path in a language that lets you own it — and a team that has already built it.

—

**Ready to ship search that holds its p99 under real load?** OnlineCloud is a US-based engineering studio that builds MCP-native, evaluated, production-observable systems — including custom vector search for e-commerce intelligence — and ships them in 6–10 weeks, not quarters. We would rather argue about your `efSearch` than sell you a deck. [Contact us](https://www.onlinecloud.dev/) or write to contact@onlinecloud.dev — free initial consultation, response within 24 hours.

# Implementing Audit Logging for Agentic Decisions: A Schema-First Approach


*By the compliance engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Evidence, Not Transcripts: Why Chat Logs Fail Audit

Regulators do not ask what your agent *intended*; they ask what it *did* — which systems it touched, which records it read or modified, under whose authority, at what moment, and with what approval. When the SOC 2 assessor or the external auditor arrives, the most common defense I see is a database full of chat transcripts, and it collapses in the first fifteen minutes.

An unstructured chat log is a conversational artifact, not an evidentiary record. It captures what the model *said*, paraphrased for a human reader — not what it actually executed. The refund the agent processed appears in the transcript only if the model happened to mention it, and models summarize, elide, and occasionally confabulate. The tool call that mutated a ledger row, the exact arguments passed, the row-level security context under which the query ran, the approval that authorized it: none of that was ever designed to be in the transcript.

Three structural failures make chat logs unusable for audit. **Incompleteness:** an agentic run is mostly invisible machinery — planning, tool invocations, retries, aborted branches — so a transcript cannot attest to activity that was never recorded. **Non-verifiability:** transcript text cannot be cryptographically anchored, so you cannot prove it was not edited, reordered, or regenerated. **Non-reconcilability:** transcripts carry no stable identifier linking an utterance to the database transaction it caused, so you cannot trace an anomalous write back to the decision behind it.

The fix is a change of frame: stop treating agent runs as conversations to be stored and treat them as *transactions to be journaled*. Every execution should emit a structured, versioned, append-only stream of decision events — a schema that defines, up front, what an auditable decision looks like. This post is the field guide: the schema, the log anatomy, immutable storage, the control mapping, and the replay tooling that turns raw events into evidence.

## 2. The Decision Schema: A Parent-Child Tree of Execution

Schema-first means the audit contract is defined *before* the agent runs, not reconstructed after an incident. If every tool, sub-agent, and orchestration loop must emit events conforming to one versioned schema, completeness stops being a matter of developer discipline and becomes a property of the system: an event that does not conform cannot be written.

The core insight: an agentic execution is not a flat sequence of log lines — it is a **tree**. A run begins with a goal; the orchestrator plans and spawns branches; each branch invokes tools that return results, trigger retries, request approval, or spawn sub-agents; some branches abort. Linear logs destroy that structure. A parent-child event model preserves it, so an auditor can walk from the initiating user request down to the tool call that changed a row — aborted branches included, because proving what was considered and rejected is often as important as what succeeded.

Every event carries a common envelope: a globally unique `event_id`; a `run_id` grouping the whole decision; `parent_event_id` linking the event to the step that caused it; an `event_type` from a closed vocabulary; `occurred_at` in UTC with sub-second precision; the acting principal and its full delegation chain; and a monotonic sequence number for ordering within the run.

“`json
{
“$schema”: “https://schemas.onlinecloud.dev/agentic-audit/event.schema.json”,
“title”: “AgenticAuditEvent”,
“type”: “object”,
“required”: [
“schema_version”, “event_id”, “run_id”, “parent_event_id”,
“event_type”, “occurred_at”, “sequence”, “principal”
],
“properties”: {
“schema_version”: { “const”: “1.0” },
“event_id”: { “type”: “string”, “format”: “uuid” },
“run_id”: { “type”: “string”, “format”: “uuid” },
“parent_event_id”: { “type”: [“string”, “null”], “format”: “uuid” },
“event_type”: {
“enum”: [
“run_started”, “plan_created”, “sub_agent_spawned”,
“tool_invoked”, “tool_completed”, “tool_failed”,
“approval_requested”, “approval_granted”, “approval_denied”,
“branch_aborted”, “run_completed”, “run_failed”
]
},
“occurred_at”: { “type”: “string”, “format”: “date-time” },
“sequence”: { “type”: “integer”, “minimum”: 0 },
“principal”: {
“type”: “object”,
“required”: [“initiating_user”, “delegation_chain”],
“properties”: {
“initiating_user”: { “type”: “string” },
“delegation_chain”: { “type”: “array”, “items”: { “type”: “string” } }
}
},
“tool”: {
“type”: “object”,
“properties”: {
“name”: { “type”: “string” },
“arguments”: { “type”: “object” },
“argument_hash”: { “type”: “string” },
“result_status”: { “enum”: [“success”, “error”, “timeout”, “rejected”] },
“result_ref”: { “type”: “string”, “description”: “Pointer to stored result in the evidence store” }
}
},
“model”: {
“type”: “object”,
“properties”: {
“model_id”: { “type”: “string” },
“prompt_version”: { “type”: “string” },
“reasoning_ref”: { “type”: “string” },
“tokens_in”: { “type”: “integer” },
“tokens_out”: { “type”: “integer” }
}
},
“access_context”: {
“type”: “object”,
“properties”: {
“tenant_id”: { “type”: “string” },
“database_role”: { “type”: “string” },
“effective_scope”: { “type”: “string”, “description”: “RLS predicate actually enforced” }
}
}
}
}
“`

Three design rules matter in practice. First, the schema is **closed for event types but open for attributes**: add fields by bumping `schema_version`, never by overloading existing ones, so replay tools migrate deterministically. Second, every node carries its `parent_event_id` — an event without a parent means a malformed run, and the stream should alarm. Third, the schema is owned by the compliance function, not the model team — a feature that cannot emit conforming events cannot ship.

## 3. Audit Log Anatomy: Tool Inputs, RLS Context, and Reasoning

An envelope without payload is a timestamp. The evidentiary value lives in three payload classes, each with its own capture discipline.

**Tool inputs and outputs.** The most audited fact in an agentic system is: *what did the tool actually receive, and what did it return?* Capture the validated arguments *after* policy enforcement — schema-validated, scoped, sanitized — not the raw model proposal, which may contain hallucinated or injected content. Record an `argument_hash` over the canonical serialization so payloads can be integrity-checked without re-reading them, and write full results to a separate evidence store behind a `result_ref` pointer. Redact secrets, tokens, and full PII at the boundary before writing; a compliant log must not become a data breach. Record the outcome truthfully — success, error, timeout, or policy rejection — because a rejected call is an audit event too, often the most interesting one in an incident review.

**RLS context.** Row-level security is what makes agentic database access defensible, but the auditor needs proof the boundary was *actually enforced* at the moment of the call. Capture the live access context: the tenant or account ID, the database principal, the propagated delegation chain (initiating user → orchestrator → sub-agent), and the effective predicate the RLS layer appended — the `SYS_CONTEXT` values that drove an Oracle VPD policy, or the session context a Postgres or SQL Server row-security policy read. When you can show the auditor that a customer query ran under `tenant_id = 4421` with the initiating user’s delegation chain, you have converted “the agent can only see its own data” from an assertion into a demonstrated control.

**Model reasoning steps.** The uncomfortable question is whether to log chain-of-thought. My guidance: capture *decision-relevant* reasoning, not the raw hidden trace. Log the plan the model committed to, the retrieved-context identifiers that informed each step (which documents and tool results entered the window), the model and prompt versions, and the outcomes at approval gates. If you persist full reasoning tokens — and there are legitimate forensic reasons to — treat them as the most sensitive material in the estate: separately encrypted, access-controlled to the compliance role, excluded from end-user transcripts, and bound to their own retention schedule, because reasoning traces routinely contain PII that was never meant to be stored. The schema models this with a `reasoning_ref`: a pointer, not an inline dump.

## 4. Immutable Storage: Object Lock or Write-Only RDS

An audit trail you can edit is a narrative. Immutability has one job: make the record *survive the compromise it is designed to detect*. Two production patterns dominate, and both rest on the same rule — **the write path must be reachable only by credentials the agents cannot hold, and no code path in the agent runtime may delete or modify events.**

**Pattern A: S3 with Object Lock in Compliance mode.** Ship events as objects keyed by `run_id/sequence`, versioned, into a bucket with Object Lock in Compliance mode and a retention period exceeding your records schedule — align it to legal counsel and any SOX-relevant retention. In Compliance mode, not even the root account can shorten retention or delete an object before the period lapses; that is the property you are buying. Give the agent runtime’s IAM role `s3:PutObject` only, with a bucket policy denying `s3:DeleteObject`, `s3:DeleteObjectVersion`, and `s3:PutBucketPolicy` to every role the model can assume, and enforce SSE-KMS with a key the agent role cannot use. Buffer events through SQS or Kinesis so the producer never talks to the bucket directly — the queue is the only door — and alarm on any write that bypasses the conforming path. Lifecycle-tier to Glacier after the active window; the retention clock still runs on the locked object.

**Pattern B: write-only RDS.** For teams that want the trail beside the business data, use a dedicated audit schema where the application role holds `INSERT` only — `UPDATE`, `DELETE`, and `TRUNCATE` privileges simply do not exist on the role, `DROP` belongs to a separate compliance role, and a trigger rejects any mutation of the event payload. Anchor the chain cryptographically: each row stores a hash of the previous row’s canonical serialization, producing a tamper-evident hash chain that makes silent back-dating detectable on replay. A periodic integrity job recomputes the chain and pages compliance on any break.

Choose between the patterns on your threat model and cost curve — object storage is cheaper at volume and carries native WORM semantics; write-only RDS keeps events queryable in place. The non-negotiable is identical: the audit store is independent infrastructure, its credentials are provisioned outside the agent’s identity graph, and its integrity is monitored as a control, not assumed as a feature.

## 5. Compliance: Reconciling Decision Trails with SOC 2 and SOX

The schema and storage above are not abstractions; they map one-to-one onto the control frameworks your auditors will invoke.

**SOC 2 CC-series.** The AICPA Trust Services Criteria are explicit that logical access must prevent unauthorized access to systems and data (CC6), that operations must be monitored for anomalies (CC7), and that change management must be evidenced (CC8) — and an agentic deployment strains all three. CC6.1 through CC6.3 require access to be granted by role and logged: the decision schema’s delegation chain is your evidence of which principal held which capability at decision time. CC6.6 requires the audit trail itself to be protected from alteration and unauthorized destruction — that is Section 4, verbatim. CC7.2 and CC7.3 require anomaly detection and response: your malformed-event alarms and hash-chain integrity jobs are the monitoring controls. When the assessor asks how you know the agent only touched approved data under approved identities, the answer is a replayable, immutable decision tree — not a screenshot of a chat window.

**SOX.** Where agents touch financial processes — revenue recognition, refunds, journal entries, close activities, order-to-cash — the trail becomes evidence over internal control over financial reporting. Under Sections 302 and 404, management must certify, and auditors attest to, the design and operating effectiveness of controls around financial data, including the *automated* controls your agents now embody. An agent that posts a journal entry is a control activity; the question is whether its execution is auditable. The parent-child tree supplies the ITGC evidence the engagement team will demand: the initiating user’s authority (segregation of duties — the agent cannot approve its own irreversible action, because `approval_granted` events come from a human checkpoint identity), the change record (model and prompt versions pinned per event), and the data integrity chain (hash-linked events reconciling to the ledger). Retain SOX-relevant decision trails as long as the underlying financial records — commonly seven years — and design the Object Lock window accordingly, because retroactive retention is not a thing in object storage.

The reconciling rule is simple to state and hard to fake: **every control assertion about the agent must cite a decision-tree artifact.** “Access was scoped” cites RLS-context events; “changes were authorized” cites approval-checkpoint events; “nothing was altered” cites the hash chain. A control that cannot cite an event does not exist.

## 6. Forensic Replay: Rebuilding the Trajectory for Incident Review

The final layer pays for all the others: tooling that turns the event stream back into a decision. Replay takes a `run_id`, assembles the tree by resolving `parent_event_id` links, and renders the trajectory as an interactive timeline — goal, plan, each tool invocation with its validated arguments and outcome, approval gates, aborted branches, and reasoning pointers. Because every event carries the delegation chain and UTC timestamps, the tool answers the questions incident reviewers actually ask: *What did user X’s session do between 14:02 and 14:09? Which tool calls touched records outside tenant 4421? Which write operations ran without an approval checkpoint? Did this refund decision precede or follow Tuesday’s prompt-version change?*

Build replay with four properties. **Full fidelity:** include failed and rejected events, not just successes — incidents hide in branches that were denied and retried with mutated arguments. **Role-based redaction:** the replay layer re-applies the ingestion redaction policy, so a Tier-1 investigator sees arguments with secrets masked while the compliance role can expand them under a documented access reason. **Integrity verification first:** validate the hash chain before rendering; if the store shows tampering, the tool says so before it shows anything else. **Exportable evidence:** every view exports to a signed, timestamped artifact (CSV, JSON, or PDF) that reaches an auditor or opposing counsel without another engineer in the loop. The benchmark of a good replay tool is boring: an external reviewer can reconstruct what happened, in order, with citations, in an afternoon — with no access to the production agent.

## The Audit Trail Is the Product

The agentic era has not changed the first law of auditable systems: if you cannot prove what happened, from the outside, after the fact, it did not happen in the eyes of your auditors. Unstructured chat logs fail that test on day one. A schema-first decision tree — structured events, captured tool inputs, enforced RLS context, immutable storage, framework mapping, replay tooling — passes it, because it treats the audit trail as a first-class system property, not a logging afterthought. Organizations that deploy agents without it will learn the lesson in an incident review; those that ship it will hand the assessor a replayable decision tree and watch the room relax. Choose which meeting you want to be in.

—

**Designing an audit trail for agentic decisions? Talk to the OnlineCloud architects.** We are a US-based Agentic AI & MCP engineering studio — senior engineers only, no offshore handoff — and we ship evaluated, observable, audited systems into production in 6–10 weeks. From decision schemas and MCP tool surfaces to Object Lock pipelines and write-only audit stores, we build compliance into the architecture, not the slide deck. Contact us at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/) for a free initial consultation — we respond within 24 hours.

# Designing a Cost-Aware Router for Model Cascades: Bedrock vs. Self-Hosted

*A FinOps Architect’s guide to paying for reasoning only where reasoning is required.*


—

## 1. Introduction: The Economics of Frontier Reasoning vs. Steady-State Inference

Every LLM workload is really two workloads wearing the same API shape. There is the **reasoning tail** — the 10–20% of requests that genuinely need a frontier model to think: multi-step planning, adversarial code review, unstructured “figure out what I’m asking” prompts. And there is the **steady-state bulk** — the 80–90% of production traffic that is structurally repetitive: extraction from a known schema, classification, summarization of a fixed document type, routing a support ticket, rewriting a paragraph.

The gap is not semantic; it is **pricing architecture**. Frontier providers meter reasoning separately from output: every chain-of-thought token the model “thinks” is billed, putting reasoning-class models at $15–$75 per million output tokens plus a surcharge. Meanwhile, a self-hosted 8B–70B model on amortized GPU capacity delivers $0.10–$0.50 per million tokens — a **10–30× spread** — with sub-second time-to-first-token and data that never leaves your VPC.

The FinOps failure mode is treating these as one pool. Route every request to the frontier model because “it’s the best model” and you convert a 10–20% reasoning need into a 100% reasoning bill: the line item balloons, p95 latency climbs with queueing, and — counterintuitively — quality doesn’t improve. Frontier models are not measurably better at schema-constrained extraction than a tuned 8B model; they are just more expensive at it.

A model **cascade** fixes this. Put a router in front of the model tier, classify each prompt by complexity, and send the bulk to self-hosted inference on EKS while the reasoning tail goes to Bedrock. Done well, a cascade cuts GenAI inference spend by 60–75% at a bounded, measurable accuracy cost — typically under one point on your eval set. This post walks through the router pattern, the decision heuristics that make it trustworthy, the Go architecture we run in production, and a worked ROI model at 50k requests/day.

—

## 2. The Router Pattern: Classifying Prompts by Complexity and Routing by Cost/Latency

A cascade router is an intercepting layer between your application and your model endpoints. Every request arrives at one ingress and is dispatched to one of two tiers:

| Tier | Backend | Cost profile | Latency profile | Best for |
|—|—|—|—|—|
| **Frontier** | Bedrock (Opus/Sonnet class) | $3–$15+ per M tokens | 2–10s+ (reasoning) | Novel, ambiguous, multi-step reasoning |
| **Self-hosted** | EKS + vLLM/TGI (8B–70B) | ~$0.10–$0.50 per M tokens, amortized | 150–800ms | Extraction, classification, known-shape transforms |

The router performs three jobs, in order:

1. **Policy override.** Explicit client intent wins: an `X-Route: frontier` header, a `requires_reasoning: true` field, or a retry-after-failure escalation must always reach the frontier tier. The router is a cost optimizer, not a quality gate that can be silently skipped.
2. **Classification.** A cheap scoring step decides which tier the request belongs to (Section 3). It must cost under ~1% of the request’s downstream price — easy, since it’s a few milliseconds of CPU/GPU on a tiny model, not another LLM call.
3. **Dispatch with failover.** The chosen backend handles the request; on timeout, 5xx, or low-confidence output, the router escalates to the other tier exactly once and records the escalation as a metric.

Two properties make the pattern production-grade rather than a demo:

**Cascading is asymmetric.** Sending a request to the frontier model is always allowed and always safe; serving it from the self-hosted tier is only safe when you can bound the classifier’s error. Hence the design principle: *when in doubt, escalate.* The router’s default on uncertainty is the frontier tier, and savings come from making the *confident* cheap calls fast and correct.

**Cost is a routing dimension, not just a bill.** Tier choice sets both price and latency, so routing is a two-objective optimization with a constraint: per-request accuracy stays above the product’s floor. You are not minimizing cost; you are minimizing cost *subject to an accuracy budget* — and everything downstream (thresholds, heuristics, dashboards) exists to keep that constraint observable and enforceable.

—

## 3. Decision Heuristics: Logprob Variance and Classification

The router’s entire value rests on one question: *can I cheaply predict that the self-hosted model will answer acceptably before I let it answer?* Two families of heuristics answer it, and production routers use both in sequence.

### 3.1 Logprob variance (a cheap “does the model know?” probe)

When a model is confident, its sampled outputs converge; when it is guessing, they diverge. You can exploit this without touching the frontier tier: sample **K = 4 generations at low temperature** from the *self-hosted* model (nearly free at $0.10–$0.30/M) and measure agreement across the samples.

The cleanest signal is **semantic entropy**: cluster the K outputs by normalized semantic similarity (or exact-match on forced-extracted answers) and compute the variance of the model’s token logprobs around the majority answer. Low variance + high agreement → the model has the answer → serve it. High variance → the model is improvising → escalate.

“`text
logprob_variance = E[ (log p(token_i) – mean_logprob)^2 ]
agreement = (K – minority_count) / K

route = self-hosted if logprob_variance < τ_var AND agreement > 0.75
route = frontier otherwise
“`

The probe is **self-calibrating per request** — no labeled data, because it measures the model’s own epistemic state. In practice it catches the classic failure modes: adversarial phrasings, out-of-vocabulary entity names, ambiguous instructions with two plausible readings. Its cost is 3 extra cheap generations (30–90ms of added latency), amortized against a 10–30× price differential.

### 3.2 Learned classification

Probing multiplies cheap inference by K; for the 60–70% of traffic that is *obviously* steady-state — “extract the invoice number,” “classify this ticket,” “summarize this changelog” — you want a microsecond decision instead. That is the **small classifier** run before any generation:

– **Features:** prompt length, reasoning verbs (“prove,” “compare,” “design,” “why”), number of explicit constraints, entity density, structured-output requests, instruction complexity, domain markers.
– **Model:** a fine-tuned DeBERTa-v3-small (or gradient-boosted model over the feature vector) trained on 10–20k labeled exemplars from your traffic mix — single-digit milliseconds on CPU, emitting a score in [0, 1] plus calibrated confidence.
– **Decision:** below a low threshold → confidently cheap → self-hosted, skip the probe. Above a high threshold → reasoning needed → frontier. In the middle (the *uncertainty region*) → run the logprob probe, or default to frontier if latency-bound.

The two-stage design keeps the router **graceful under distribution shift**: when traffic drifts (a new product ships, a new template lands), the classifier’s scores shift and the probe stage absorbs the uncertainty instead of silently misrouting. You detect the drift on the dashboard (Section 6) and retrain.

### 3.3 Calibration and the accuracy budget

None of this works without a **held-out eval set** — a golden set of 1–3k prompts with reference answers, run nightly against both tiers. You tune τ_var and the classifier thresholds on a single axis: *the measured accuracy delta between the cascade and the all-frontier baseline*, held under a committed budget (e.g., Δ ≤ 0.5–1.0 points). Thresholds are not tuned to maximize savings; they are tuned to spend the accuracy budget, and savings is whatever falls out. That inversion of priorities is what keeps a cascade from becoming a silent quality regression users notice three weeks later.

—

## 4. Architecture: A Go Router in Front of Bedrock and EKS

Go is the right substrate here: it is the lingua franca of ingress proxies (the router *is* a proxy), it streams HTTP without the GC pauses that break SSE, and it deploys as a single static binary on EKS beside the inference tier.

“`
┌─────────────────────────────────────────┐
│ Go Cascade Router │
App traffic ───────► │ policy override → classifier → probe │
│ dispatch · retry · escalate · metrics │
└───────────────┬───────────┬─────────────┘
│ │
┌───────────────▼──┐ ┌────▼──────────────┐
│ EKS (self-hosted) │ │ Bedrock (frontier)│
│ vLLM · 8B-70B │ │ Opus/Sonnet class │
└───────────────────┘ └───────────────────┘
“`

The router is a plain `net/http` server whose decision core is deliberately small and pure — unit-testable without network:

“`go
type Route string

const (
RouteSelfHosted Route = “eks”
RouteFrontier Route = “bedrock”
)

type Router struct {
classifier *Classifier // feature-based scorer
prober *ConsistencyProber // logprob variance probe
cfg Config // thresholds, timeouts, retries
}

// Decide returns the tier for a request. It never returns an error
// for an ambiguous request — ambiguity escalates to frontier.
func (r *Router) Decide(ctx context.Context, req *Request) Route {
// 1. Policy override — explicit intent always wins.
if req.Header.Get(“X-Route”) == “frontier” {
return RouteFrontier
}
// 2. Cheap classifier first.
score, conf := r.classifier.Score(req.Messages)
if conf >= r.cfg.Confident && score <= r.cfg.SelfHostedThreshold { return RouteSelfHosted } if conf >= r.cfg.Confident && score >= r.cfg.FrontierThreshold {
return RouteFrontier
}
// 3. Uncertainty band — probe the cheap model’s own confidence.
if variance := r.prober.LogprobVariance(ctx, req, 4); variance < r.cfg.VarianceThreshold { return RouteSelfHosted } return RouteFrontier // when in doubt, escalate } ``` The dispatch layer wraps that decision in the concerns that keep the system alive: - **Streaming.** Both tiers are proxied with `http.Flusher` passthrough. The classifier runs *before* streaming begins, so the added latency is one ~5ms round-trip, not a serialized LLM call. - **Failover.** Every upstream call gets a deadline (frontier: 120s, self-hosted: 30s). On timeout, connection error, or a 429/5xx burst, the router retries once on the *other* tier and stamps `x-cascade-escalated: true` so callers and dashboards both see it. - **Bedrock specifics.** Calls use Bedrock's streaming invoke API with IAM SigV4 signing and `contentBlockDelta` decoding; the EKS tier is a vLLM `OpenAI-compatible /v1/chat/completions` endpoint on an internal service. One shared request/response schema keeps the router thin. - **Deployability.** The router ships as its own Deployment behind the service mesh, scales on CPU (the classifier is the only compute), and exposes `/-/healthz` plus a `/-/routes` endpoint dumping live threshold state for on-call engineers. The highest-leverage operational detail is **making the decision observable on the request itself**: every response carries `x-cascade-route`, `x-cascade-score`, and `x-cascade-escalated`, so application teams can build product behavior on top of routing (forcing frontier for a user segment, surfacing "answered by fast model" badges) without touching router code. --- ## 5. ROI Calculation: Monthly Savings at 50k Requests/Day A worked model with stated assumptions, so you can re-run it with your own rates: **Assumptions** - Traffic: 50,000 requests/day, average 2,000 input tokens and 500 output tokens each. - Frontier (Bedrock, Sonnet-class blended): **$3/M input, $15/M output**. - Self-hosted: 8B model on vLLM, fleet of 6× A10G at **$0.60/GPU-hr effective** (savings plans/spot), all-in for the bulk tier. - Cascade accuracy budget: Δ ≤ 0.01 on the golden eval set. **Baseline — everything to the frontier model:** - Input: 100M tokens/day × $3/M = $300/day · Output: 25M tokens/day × $15/M = $375/day - **API spend ≈ $675/day ≈ $20,250/month** — with no GPU cost at all, because there is no self-hosted tier. **Cascade — 85% confidently self-hosted, 15% frontier (probes and escalations absorbed):** - Frontier tier: 7,500 req/day → 15M input ($45) + 3.75M output ($56) ≈ $101/day ≈ **$3,030/month** - Self-hosted tier: 42,500 req/day on the 6× A10G fleet → 6 × $0.60 × 24h ≈ **$86/day ≈ $2,590/month** (classifier, probes, router CPU included) - **Cascade total ≈ $5,620/month ≈ 72% below baseline → ~$14,630/month saved.** | Scenario | Monthly cost | vs. all-frontier | |---|---|---| | All traffic → frontier | ~$20,250 | — | | Naive 50/50 static split | ~$15,300 (half frontier spend + larger GPU fleet) | −24% | | **Classifier-driven cascade (85/15)** | **~$5,620** | **−72%** | Two findings worth internalizing. First, **a naive static split is a false economy**: 50% of traffic to each tier still burns ~$10k/month on frontier tokens *and* forces GPU over-provisioning for requests the classifier would have kept at home — the savings live in the *routing decision*, not the split ratio. Second, **the GPU fleet is the floor, not the ceiling**: below ~15–20k self-hosted requests/day, fixed fleet cost dominates and a pure-API cascade (small model on Bedrock's cheap tier, no EKS) can win on paper. Model the fixed floor against per-token savings, then decide whether EKS earns its place. Annualized, that is **~$175k/year** at 50k req/day — the router's dev cost (a few engineer-weeks plus the classifier training set) pays back in its first month, and the nightly eval keeps the accuracy budget honest so savings arrive without waiting for user complaints. --- ## 6. Monitoring: Cascade Hit-Rates and Accuracy Trade-offs A cascade you can't observe is a cost leak with a latency SLA. The stack has one north-star metric and three supporting views, all exported as Prometheus metrics from the Go router and rendered in Grafana, with cost data reconciled from CloudWatch/Bedrock usage and EKS node bills. **North star — accuracy delta (the constraint).** Every night a batch job replays the 2k-prompt golden set against (a) the frontier tier and (b) the live cascade, publishing `cascade_accuracy_delta`. The alert is blunt: if the trailing-7d Δ exceeds the 0.01 budget, page the owning team. This is the *only* metric that can veto a threshold change — savings gains are real only while Δ stays inside budget. **View 1 — Cascade hit-rate and routing telemetry.** Per-route counters and histograms labeled by route, model, and escalation flag: - `cascade_hit_rate = self_hosted_requests / total_requests` — the efficiency gauge; watch it per segment (product, prompt template) to catch segments silently degrading. - `cascade_escalation_rate` — how often router failover fired. A sustained rise usually means the self-hosted tier is degrading (slow TTFT, timeouts), *not* that the classifier is wrong. - `router_decision_us{stage="classifier|probe"}` and the *uncertainty-band share* — traffic landing between thresholds; a widening band over weeks means drift and a retraining trigger. **View 2 — Cost and latency per request, by tier.** `cost_per_request{route}` (token meters × contract rates, plus fleet $/hr spread across requests) alongside p50/p95 latency per tier. The money line: `monthly_spend = sum(rate(cost_per_request) * seconds_in_month)` compared against the all-frontier counterfactual — the view the FinOps review actually reads: *did the cascade save what the model said it would, this week?* **View 3 — Classifier health and drift.** The distribution of classifier scores per segment, the logprob-variance distribution of probe-stage requests, and per-threshold confusion on the nightly eval (`self-hosted-but-wrong` is the true cost of misrouting). Thresholds are only meaningful relative to the eval set they were calibrated on, so the dashboard pins current τ_var and cutoffs next to the eval Δ — one screen answering "is this router still doing its job?" **Operational loops worth wiring from day one:** - **Shadow mode for threshold changes.** Replay 48h of logged traffic through the proposed config before moving a threshold; diff the predicted Δ with zero production risk. - **Escalation as a quality canary.** A sustained rise in escalations *to* frontier is a leading indicator of self-hosted quality drift — investigate before users file bugs. - **Accuracy sampling in production.** Beyond the golden set, sample 0.5–1% of self-hosted responses for LLM-as-judge scoring. Golden sets go stale; production samples don't. --- ## Build the Cost-Aware Tier on OnlineCloud The pattern is proven: classify, probe, escalate, and police the accuracy budget — letting the frontier model do the thinking only where thinking is required. The remaining question is operational: you need managed Bedrock access, an EKS tier that scales your GPU fleet to the *cascade's* actual demand, and cost telemetry spanning both — without stitching five consoles together. That's what OnlineCloud delivers: unified access to frontier models and your own Kubernetes inference clusters, with per-cluster and per-route cost visibility out of the box. **Ship your first cascade this week — start free at [onlinecloud.dev](https://www.onlinecloud.dev/)**, and let the dashboards show you where the 72% went.

# Building a Low-Code MCP Dashboard with React and Tailwind

*How to turn a stream of MCP JSON-RPC notifications into a real-time, human-checkpointed control room — without leaving the React ecosystem.*


## 1. Why Your Agent Needs a Dashboard It Never Asked For

The first time you watch a production agent run unattended, you’ll notice something uncomfortable: it’s a black box with a pulse. It plans, calls tools, mutates rows in your ERP, and reports back in terse JSON that no human reads until something breaks. That’s the fundamental trust problem with agentic systems — not model quality, but *observability of intent*. A plan/act/reflect loop that touches a production database needs a human who can answer one question at any moment: *what is the agent doing right now, and should it be allowed to finish?*

This is the human-in-the-loop gap. Model Context Protocol (MCP) gives you typed tools, declarative resources, and auditable transports, but it doesn’t hand you a UI. The transport is a contract, not a cockpit. If you’re running MCP servers in front of Oracle, MS SQL, Snowflake, or a CRM, an autonomous loop that can issue writes deserves a control surface between the LLM and your business logic — a surface that visualizes each tool call as it happens, makes the agent’s decision tree legible, and inserts a hard checkpoint before high-impact mutations.

Think about what actually goes wrong in production. An agent tasked with reconciling invoices chains five tools: a read query, a fuzzy match, a write to the accounting system, a notification to finance, and a retry path when a row is locked. Each step is individually reasonable; the *combination* is where risk hides. Without a live view, your only signals are a completed run or a timeout — you find out an hour later that the agent emailed 40 customers with a miscomputed total because a JOIN dropped a filter. A dashboard doesn’t prevent the bad idea; it compresses the time between the bad idea and a human seeing it, from hours to seconds. That compression is the entire value proposition of human-in-the-loop tooling.

The fastest way to build that surface, without dragging in a heavy BI stack, is a single-page dashboard: React for state and component modeling, Tailwind for the visual layer, and a thin adapter that bridges MCP’s streamable transports into React state. Low-code here doesn’t mean “no code.” It means the *dashboard* logic stays small enough that a team can ship it in a sprint, because the hard protocol work is already done by MCP. You’re not building an agent platform; you’re building a window onto one.

## 2. Dashboard Architecture: From MCP Transport to React State

MCP servers expose tools over two mainstream transports: the original SSE-based transport and the newer streamable HTTP transport. From a dashboard’s perspective, both collapse into the same shape: a client-to-server JSON-RPC channel for requests, and a server-to-client event stream for notifications. The notifications you care about are the ones that make agent behavior visible: `notifications/progress` for tool execution progress, tool call lifecycle messages, and server requests that need human input (which MCP models as a `sampling/createMessage` request).

The architectural rule that keeps this dashboard “low-code” is **never let a UI component talk to the wire**. Every transport event funnels through one place — a reducer that normalizes JSON-RPC into a flat, indexable domain model. Everything downstream (progress bars, decision trees, modals, the inspector) is a pure function of that state. That single choke point is also where you attach audit logging, which matters when the dashboard is your compliance story for “who approved this tool call.”

“`tsx
// bridge.ts — normalize raw MCP notifications into app events.
// One adapter per transport; the reducer never sees JSON-RPC.
export type McpEvent =
| { kind: “tool:started”; callId: string; tool: string;
parentCallId: string | null; args: unknown }
| { kind: “progress”; callId: string; pct: number | null; message?: string }
| { kind: “tool:finished”; callId: string; result: unknown }
| { kind: “checkpoint:requested”; callId: string; tool: string };

export function fromJsonRpc(msg: any): McpEvent | null {
if (msg.method === “notifications/progress”) {
const p = msg.params;
return {
kind: “progress”,
callId: p.progressToken, // our correlation id
pct: p.total ? Math.round((p.progress / p.total) * 100) : null,
message: p.message,
};
}
// … map tool:started / finished / sampling requests the same way
return null;
}
“`

With an `EventSource` (SSE) or a fetch-based reader (streamable HTTP) feeding `dispatch(fromJsonRpc(raw))`, React becomes a pure projection of the agent’s runtime state — the same trick that makes Redux DevTools useful, applied to your agent. A `parentCallId` field is the seed of your decision tree: because MCP tool calls nest (a plan tool calls a query tool), retaining the parent link on every event lets you render the agent’s reasoning structure for free.

## 3. Real-Time UI: Progress Bars and Decision Trees in Tailwind

Once events are normalized, the UI is almost mechanical. Tool-call progress is a per-call percentage plus a status enum — `queued`, `running`, `awaiting-approval`, `done`, `error`. A progress bar in Tailwind is a borderless rounded track with a gradient fill whose width is driven by a style binding, animated purely by `transition-[width]`. The status enum drives color: emerald for healthy execution, amber when the tool is parked at a human checkpoint, rose for failures.

“`tsx
// ToolProgress.tsx — one running tool call
export function ToolProgress({ tool }: { tool: ToolState }) {
return (

          {JSON.stringify(pending.args, null, 2)}