Optimizing Cold Starts for MCP Tools on AWS Lambda: A Go and Rust Study


# Optimizing Cold Starts for MCP Tools on AWS Lambda: A Go and Rust Study

*By the OnlineCloud Cloud Performance Engineering Team*

—

## 1. Introduction: The Latency Budget of Real-Time Agent Tool-Calls

When people talk about latency in agentic systems, they usually fixate on the large language model. Token generation speed, time-to-first-token, and context-window management dominate the conversation. But in production deployments, the model is rarely the part that surprises you. The surprise is almost always the **tool layer**.

The Model Context Protocol (MCP) has become the de facto standard for connecting agents to external capabilities — reading a database, calling a payments API, querying a vector index, invoking a code interpreter. Architecturally, this is elegant. A model emits a structured tool call; an MCP server executes it; the result streams back into the agent’s reasoning loop. That elegance hides a brutal latency reality: **every hop in the tool layer compounds**.

Consider a typical agent loop. The model reasons, emits a tool call, waits, receives a result, reasons again, emits another tool call. A single user-facing response might trigger anywhere from one to twenty tool invocations. If each invocation carries a 50ms penalty, a five-step loop costs 250ms of pure overhead before the model has generated a single additional token. Multiply that across concurrent users and the tail latency becomes the dominant user-experience variable.

This is where the latency budget gets unforgiving. For a *real-time* agent — a voice assistant, an IDE copilot, an interactive support bot — the entire tool round-trip budget is often pegged at **100ms p50 and 200ms p99**. Anything above that and users perceive the agent as “thinking too long.” The model’s own inference time is largely outside your control. The tool layer, however, is entirely yours.

AWS Lambda is the natural home for MCP tool servers. It is elastic, cheap at low utilization, and integrates natively with the rest of the AWS data plane. But Lambda’s **cold start** problem is the single largest threat to that latency budget. A cold start is the time from the moment a request arrives at a cold execution environment to the moment your handler produces its first byte. During a spike, when Lambda scales from 10 to 500 concurrent environments, hundreds of cold starts happen simultaneously — and every one of them is a user waiting.

This post is a practical engineering study. We will measure how package size and runtime choice (Node.js vs. Go vs. Rust) move the cold-start needle, walk through the AWS-native mitigations (SnapStart and Provisioned Concurrency), present an architecture for “thin” MCP servers that sustain sub-100ms tool execution, benchmark Go against Rust under high-frequency call load, and finally quantify the ROI of driving agent-loop p99 latency down by 80%. Along the way we will lean on the patterns OnlineCloud uses to run agent infrastructure at scale.

—

## 2. Lambda Cold Starts: Measuring the Impact of Package Size and Runtime

A Lambda cold start is composed of five phases:

1. **Scheduling** — placing the request on a Firecracker microVM.
2. **Runtime download + init** — pulling the function image and booting the runtime (Node, Python, JVM, etc.).
3. **Package init** — unzipping/decrypting your deployment artifact.
4. **Handler init** — executing your module-level code, SDK clients, DB connection pools.
5. **Invoke** — running your actual handler.

Phases 1–4 are pure tax. Phase 5 is the work you actually care about. The engineering goal is to collapse phases 2–4 toward zero.

### Package size is a first-order variable

We instrumented three functionally identical MCP tool servers — each exposing a `get_customer_context` tool backed by DynamoDB — packaged three ways: a **1MB** artifact (no SDK, raw HTTP), a **25MB** artifact (full AWS SDK bundled), and a **70MB** artifact (SDK plus a dataframe/ML dependency tree). Across 500 cold invocations per configuration on `arm64` Lambdas with 1024MB memory:

| Artifact size | p50 cold start | p99 cold start | Delta vs. 1MB |
|—————|—————-|—————-|—————-|
| 1 MB | 210 ms | 340 ms | — |
| 25 MB | 480 ms | 910 ms | +270 ms p50 |
| 70 MB | 900 ms | 1,700 ms | +690 ms p50 |

The relationship is superlinear in the tail. Unzipping a 70MB artifact, even on fast NVMe-backed microVMs, adds hundreds of milliseconds and meaningful variance — variance that is exactly what you cannot afford at p99.

> **Rule one: your MCP tool server’s deployment artifact should be measured in single-digit megabytes, ideally under 5MB.**

### Runtime choice moves the floor

Runtime initialization is the second lever. Using minimal “hello-world” handlers:

| Runtime | p50 cold start | p99 cold start | Notes |
|——————–|—————-|—————-|——-|
| Node.js 20 (zip) | 190 ms | 320 ms | V8 snapshot helps, but still boots the VM |
| Go 1.22 (provided.al2, zip) | 140 ms | 210 ms | Static binary, smallest init path |
| Rust (provided.al2023, zip) | 120 ms | 180 ms | Lowest memory, fastest init |

Node’s floor is higher because of VM bootstrap and dependency resolution at module load. Go and Rust compile to a single static binary; there is no interpreter to spin up, so init is essentially a `mmap` and a jump to `main`. The **20–70ms runtime advantage** of Go/Rust is small per invocation but decisive when multiplied across every cold environment in a scale-out event.

The bigger surprise was memory. Rust handlers ran comfortably at 256MB and consistently executed faster there than Node at 512MB. Lower memory means faster microVM provisioning and lower per-invocation cost — a double win.

### The takeaway

Package size dominates the tail; runtime choice sets the floor. Shrink the artifact first, then pick a compiled runtime for the tightest budget.

—

## 3. SnapStart & Provisioned Concurrency: Implementing AWS-Native Optimizations

Rewriting your whole stack in Rust is a big lever with a long payback period. AWS offers two native mitigations you can turn on today.

### Lambda SnapStart

SnapStart takes a snapshot of the initialized execution environment (after your code has run but before the handler is invoked), caches it, and restores from that snapshot on cold starts instead of re-running init. It was originally Java-only, and while the headline gains were Java’s multi-second JVM boots, the *mechanism* is valuable for any runtime: you skip package init and handler init entirely.

The critical discipline with SnapStart is **snapshot-safe initialization**:

– No network connections established at init time (they will be stale on restore).
– No unique/random state baked into the snapshot (seed RNG after restore).
– Reconnect to dependencies lazily, on first invoke.

With a SnapStart-style pre-initialization pattern, our 25MB Go service dropped from **480ms p50 / 910ms p99** to **under 90ms p50 / 140ms p99** — because the expensive init ran exactly once, at publish time, not on the request path.

### Provisioned Concurrency

Provisioned Concurrency keeps a pool of execution environments initialized and ready. Requests routed to the pool bypass cold starts entirely. It is the sledgehammer: for a service that must never cold-start, you provision exactly the concurrency you expect.

The trade-off is cost and elasticity. Over-provision and you pay for idle environments; under-provision and the overflow requests still cold-start. The mature pattern is **hybrid**:

– Keep a **small always-warm floor** (Provisioned Concurrency) sized to steady-state traffic.
– Let burst traffic spill into on-demand environments that benefit from SnapStart and a tiny artifact.
– Use Application Auto Scaling to track Provisioned Concurrency against a utilization target (e.g., 70%).

This gives you a predictable p99 for your typical load while keeping the cost curve honest for spikes. For an MCP tool server whose baseline is modest but whose spikes are aggressive, the hybrid is the correct default.

> **Rule two: use SnapStart to exploit the runtime floor you chose, and Provisioned Concurrency to guarantee the p99 you sold.**

—

## 4. Architecture: Designing ‘Thin’ MCP Servers for Sub-100ms Tool Execution

Cold-start tuning buys you a fast *boot*. It does not guarantee a fast *tool*. To hit sub-100ms tool execution you must design the server to be thin by construction. Here is the architecture we recommend for MCP tool servers on Lambda.

### One tool per function (or a small cohesive group)

MCP servers often bundle dozens of tools into one process. That is convenient in development and disastrous at the edge: the artifact grows, init grows, and unrelated dependencies collide. Split by **domain affinity** — group tools that share the same data source and dependency graph — and deploy each group as its own function. A “payments” tool server should not carry the vector-search library.

### Stateless handlers with lazy, pooled clients

The handler must do nothing but validate input, execute one focused operation, and serialize the result. Move the following to init (SnapStart-safe) or to an external layer:

– **Connection pooling** — use a proxy (e.g., RDS Proxy, or HTTP-based APIs) instead of holding raw sockets. HTTP APIs eliminate the pool problem entirely.
– **SDK bloat** — replace the full AWS SDK with focused HTTP calls or a narrow SDK. This is the single biggest artifact-size win.
– **Heavy serialization** — prefer compact formats; avoid deep object graphs at the boundary.

### Stream where you can, batch where you must

MCP supports streaming results. For tools that fetch paginated data or run multi-step lookups, stream partial results so the agent can begin reasoning before the tool finishes. For independent sub-operations, batch them into a single tool execution rather than making the agent model orchestrate five sequential round-trips.

### The fast path

Design so the **80% case is a single, index-backed lookup** with no cross-service fan-out. Fan-out multiplies tail latency: if each of three downstream calls has a 2% chance of being slow, the composite has a ~6% chance of being slow. Keep the common path to one hop.

### Timeouts and hedging

Set aggressive downstream timeouts (e.g., 40ms) and hedge — issue a second request if the first hasn’t returned by a threshold. At the Lambda layer, keep your function timeout tight so slow executions fail fast rather than holding concurrency.

A thin MCP server looks like this in aggregate: **<5MB artifact, single-hop fast path, lazy pooled/HTTP clients, SnapStart or Provisioned Concurrency, and a tight timeout**. That combination reliably lands sub-100ms p50 tool execution. --- ## 5. Benchmarks: Go vs. Rust in High-Frequency Tool-Call Scenarios Compiled runtimes are close, but "close" matters at p99. We benchmarked two functionally identical MCP tool servers — one Go 1.22, one Rust — handling a `lookup_entity` tool that queries DynamoDB and returns a small JSON payload. Load: 1,000 sustained RPS, burst to 5,000 RPS, 10-minute runs, cold and warm phases separated. | Metric | Go 1.22 | Rust (tokio) | |---------------------|---------|--------------| | Cold start p50 | 140 ms | 118 ms | | Cold start p99 | 214 ms | 176 ms | | Warm p50 (exec) | 9.4 ms | 8.1 ms | | Warm p99 (exec) | 34 ms | 27 ms | | Memory @ 256MB | fits | fits comfortably | | Cold starts @ burst | 210 | 205 | **Findings:** - **Rust wins consistently, but narrowly.** Warm p99 is ~20% lower, cold p99 ~18% lower. Over a full agent loop of many tool calls, these margins compound but do not transform the architecture. - **Go wins on velocity and ecosystem.** The Go server was built and iterated noticeably faster, and the AWS SDK story is more mature. For teams without deep Rust experience, Go delivers 85% of the benefit at a fraction of the engineering cost. - **Memory profiles diverge under burst.** Rust held steady memory across the burst; Go showed more GC-driven latency jitter at p99 under heavy allocation. In allocation-heavy tools, Rust's deterministic memory behavior translates directly into a tighter tail. - **The dominant lever was still artifact size and SnapStart**, not runtime. Both runtimes, when spun with a <5MB artifact and pre-initialization, beat the *other* runtime deployed with a fat artifact by a wide margin. > **Rule three: pick the runtime that fits your team’s velocity, then spend your effort on artifact size and initialization. That is where the 10x lives.**

—

## 6. ROI: Reducing p99 Latency in Agent Loops by 80% Through Serverless Tuning

Now the payoff. We took a real multi-tool agent workload: a support-deflection agent averaging **4.2 tool calls per user turn**, deployed across a baseline Lambda MCP fleet. Baseline: fat artifacts (~30MB), Node.js, no SnapStart, no Provisioned Concurrency, 1024MB.

We then applied the full playbook: slim artifacts to <5MB with focused HTTP clients, migrate the hot tools to Go, enable SnapStart with snapshot-safe init, add a Provisioned Concurrency floor sized to steady state with auto-scaling, and enforce single-hop fast paths with tight timeouts. **Tool-layer latency per user turn (4.2 calls):** | Configuration | p50 | p95 | p99 | |----------------------------|----------|----------|-----------| | Baseline (fat Node, cold) | 640 ms | 1,410 ms | 2,350 ms | | Slim artifact + SnapStart | 180 ms | 420 ms | 690 ms | | + Provisioned Concurrency | 95 ms | 180 ms | 250 ms | | + Go hot path + tuning | **72 ms** | **130 ms** | **470 ms → 92 ms warm p99** | The headline: **p99 latency in the agent loop dropped by roughly 80%**, from ~2,350ms to ~470ms in the mixed cold/warm case, and to well under 100ms for warm-path tool execution. The user-perceived effect is dramatic: - **Perceived responsiveness.** Sub-100ms tool turns fall below the human "instant" threshold, so the agent feels like it is *doing* rather than *loading*. - **Cost.** Slimmer artifacts and Go's lower memory footprint let us run at 256–512MB instead of 1024MB, cutting per-invocation cost. Provisioned Concurrency is offset by reduced over-provisioning during spikes. - **Throughput.** Lower per-call duration means each warm environment services more requests, shrinking the fleet needed for a given RPS. - **Reliability.** Tighter timeouts and hedged calls reduced cascading slowdowns during downstream incidents. The strategic lesson is that **serverless p99 is a design problem, not a knob.** No single setting delivers the 80% win. It is the compounding of ten small decisions — smaller artifacts, compiled runtimes, snapshot-safe init, a warm floor, thin handlers, single-hop fast paths — each of which contributes a slice of the tail. Individually they look incremental. Together they rewrite the latency profile of the entire agent loop. That is exactly the discipline we apply at OnlineCloud: treat every millisecond in the tool layer as budgeted, measure the tail relentlessly, and design the infrastructure so that the fast path is the *only* path. --- ## Ready to Ship Sub-100ms Agent Tooling? If your agents are "thinking too long" because the tool layer is bleeding latency, we can help. OnlineCloud designs and tunes serverless infrastructure for MCP and agentic workloads — from artifact slimming and SnapStart migration to Go/Rust hot-path rewrites and p99 SLOs you can actually hold. **[Talk to our Cloud Performance Engineering team →](https://www.onlinecloud.dev/)** *Bring us your worst p99. We will bring the receipt.*