# Implementing Asynchronous Tool Execution: Handling Multi-Minute Jobs in MCP
*Deep-dive for engineers who need MCP tool calls to survive what synchronous RPC cannot: ETL jobs, batch inference, and data exports that run for minutes while agents, proxies, and humans wait.*
—
## 1. Introduction: The Failure of Synchronous JSON-RPC for Long-Running Agent Tasks
Model Context Protocol is JSON-RPC 2.0 under the hood: a client sends a `tools/call` request with an `id`, and the server must eventually answer *that* `id` with a `result` or an `error`. The correlation is elegant and stateless-friendly — and completely wrong as a contract for work that takes minutes.
Consider what actually happens when a tool call runs long. Most MCP deployments terminate at an HTTP boundary — an nginx `proxy_read_timeout` defaulting to 60 seconds, a cloud load balancer dropping idle connections, an API gateway enforcing its own ceiling. If the job survives the network, the agent harness is next: orchestrators impose per-turn and per-tool timeouts so a stuck call cannot consume a session forever. Over stdio the problem is worse — the server process is pinned to the client that spawned it, with no way to shed load. Three independent layers — proxies, hosts, and the LLM application loop — all assume a tool call resolves in seconds.
The workloads enterprises actually want agents to run violate that assumption exactly: exporting two million CRM rows, regenerating embeddings for a document corpus, backfilling a normalized schema across Oracle and MS SQL. Done synchronously, they turn a crisp RPC protocol into a parking lot: the connection idles, the agent stalls mid-plan, and every layer above interprets the silence as failure.
The fix is architectural: stop treating *tool execution* and *job completion* as the same event. **A tool call should return a decision quickly; the work continues outside the request/response lifecycle.** The client gets an opaque job handle, the server tracks a durable state machine, and progress and results travel on channels designed for asynchronous delivery. This is also the direction the protocol itself is moving — the 2025-11-25 MCP revision introduced **Tasks**, an experimental but SDK-supported primitive for “call now, fetch later” execution. Tasks standardized what production teams had already built by hand: job IDs, status polling, progress notifications, and a state store that outlives any single connection. That pattern is what this post is about.
—
## 2. MCP Async Patterns: Job IDs and Progress Notifications
Before any fancy queuing, two protocol mechanisms do most of the work: *progress tokens* and *job handles*.
### Progress tokens and `notifications/progress`
MCP has supported progress reporting since its earliest revisions. A client that wants progress attaches a token to request metadata:
“`json
{
“jsonrpc”: “2.0”,
“id”: 1,
“method”: “tools/call”,
“params”: {
“name”: “run_export”,
“arguments”: { “source”: “crm”, “destination”: “warehouse” },
“_meta”: { “progressToken”: “tok-export-41” }
}
}
“`
The server then emits zero or more notifications referencing that token:
“`json
{
“jsonrpc”: “2.0”,
“method”: “notifications/progress”,
“params”: {
“progressToken”: “tok-export-41”,
“progress”: 51234,
“total”: 2000000,
“message”: “row batch 26 of ~100”
}
}
“`
Three spec rules matter operationally: progress values **must** increase monotonically, notifications **must stop** at completion, and tokens **must be unique across active requests** — monotonic counters collide across sessions, so use UUIDs or ULIDs. Under Tasks, a progress token stays valid for the entire task lifetime, so this plumbing carries forward.
### Job handles: making the tool return fast
A progress token alone does not make a tool asynchronous — the server still holds the JSON-RPC response hostage until the work ends. The job-handle pattern breaks that coupling: the tool returns immediately with a description of the job, not the result:
“`json
{
“jsonrpc”: “2.0”,
“id”: 1,
“result”: {
“content”: [
{
“type”: “text”,
“text”: “Job 01J5QX2F3X0VX0 accepted. Poll get_job({\”jobId\”:\”01J5QX2F3X0VX0\”}) or wait for notifications/progress. Retained 24h.”
}
],
“structuredContent”: {
“jobId”: “01J5QX2F3X0VX0”,
“status”: “queued”,
“pollIntervalMs”: 5000,
“ttlMs”: 86400000
}
}
}
“`
The JSON-RPC exchange now completes in milliseconds, and the job lives in a namespace the client can interrogate: a `get_job` tool, a `job://` resource URI, or — under Tasks — the standardized `tasks/get`, `tasks/result`, `tasks/cancel`, and `tasks/list` methods over a `working` / `input_required` / `completed` / `failed` / `cancelled` state machine. Job IDs gate access to state and results: generate them unguessable, bind them to the creating session’s auth context, and enforce that binding on every follow-up call — a `tasks/get` from another tenant should fail as if the job never existed.
—
## 3. State Polling vs. Callbacks: Designing a Robust Event-Driven Tool Execution Loop
Given a job handle, how does the agent learn the job finished? Two mechanisms exist, with complementary failure modes.
**Polling** is requestor-driven and authoritative. The client calls `get_job` or `tasks/get` on an interval, and the server returns ground truth from durable storage. Polling survives everything: dropped connections, missed notifications, server restarts, reconnects. Its costs are latency (you discover completion up to one interval late), agent turns and tokens, and server load — hence the server-suggested `pollInterval` and backoff as jobs age.
**Callbacks** — progress and status notifications pushed over the live stream — are receiver-driven and best-effort. They give near-zero latency and keep the UI alive, but are fire-and-forget: a notification emitted while the client is disconnected is gone, and one racing a reconnect can be delivered twice or not at all. Notifications can **never** be your source of truth, because delivery is not guaranteed.
The robust design is therefore not a choice but a combination: **poll for truth, listen for speed.** The client’s loop looks like this:
“`
job = call(“run_export”, {progressToken}) # returns immediately
while job.status not in {completed, failed, cancelled}:
wait_for_notification(timeout = job.pollIntervalMs)
if timeout or reconnect: # notification lost or stream dead
job = call(“tasks/get” / “get_job”, {jobId})
else:
apply_status_update(notification.payload) # reconcile, never trust alone
result = call(“tasks/result” / “get_job”, {jobId}) # exactly the sync result, delayed
“`
Two properties make this loop robust: **notifications are an acceleration layer over a pollable state** (every message is idempotent to apply, every missed message recoverable), and **the loop always terminates by polling** — the terminal fetch is an authoritative read, so no duplicate or reordered notification can corrupt the outcome. Server side, keep the same discipline: state transitions are append-only (a `completed` task never returns to `working`, even if a late worker event says so), and terminal results stay retrievable long after notifications stop.
—
## 4. Architecture: Using Redis Pub/Sub or SQS for Background Job Status Propagation
The job-handle pattern forces a topology change. If the MCP server executes tools inline, async buys you nothing — the process is still blocked. In production you split the system: **gateway nodes** own MCP sessions, streams, and protocol state, while a **worker fleet** executes jobs. They scale independently — the gateway is I/O-bound and chatty, the worker CPU- and database-bound. Between them you need a *queue* for dispatch, a *bus* for status propagation back to the session owner, and a *state store* as the authoritative record of every job.
Redis and SQS are the two workhorses for queue-and-bus, and choosing between them is a durability-vs-latency trade, not a religious one.
**Redis Pub/Sub** gives single-digit-millisecond fan-out. Workers publish progress events to a channel; gateway nodes subscribe and forward them as `notifications/progress` to whichever session owns the job. It is operationally simple — you almost certainly already run Redis — and its fire-and-forget semantics suit notifications, which are best-effort by protocol design. The trap is treating it as a system of record: messages with no subscribers are dropped, a restarting gateway misses every event published while down, and nothing is persisted or replayable. So with Redis the rule is: **use Pub/Sub only for the delta channel, and keep authoritative job state in Redis keys with TTLs** (a `SET` for status, `HSET` for counters, `EXPIRE` for retention). A reconnecting client polls the keys, not the pub/sub history.
**SQS** (or any durable queue) inverts the guarantees. Messages persist, survive restarts, replay on failure, and move across AZs; the visibility-timeout mechanism doubles as a natural worker lease — a worker that dies mid-job releases the message for retry, and DLQs catch poison jobs. The costs are latency (hundreds of milliseconds minimum per hop), no native fan-out (pair SNS topics with queue subscriptions), and at-least-once delivery, which *forces* idempotent workers. When the job is an enterprise data export that auditors will ask about — or when a lost completion event means re-running a two-million-row extract — durability wins.
| Decision | Redis Pub/Sub | SQS (+ optional SNS) |
|—|—|—|
| End-to-end latency | ms | 100ms–seconds |
| Message persistence | None | Durable, replayable |
| Delivery semantics | At-most-once (fire & forget) | At-least-once |
| Restart recovery | Reconcile from state store | Replay from queue |
| Poison handling | Manual (state store flags) | Native DLQ |
| Ops footprint | You operate it | Managed |
| Best for | Live progress fan-out, low-latency UX | Job dispatch, compliance, cross-AZ reliability |
The reference topology combines both: a queue for dispatch, a durable state store as the source of truth, and a Redis Pub/Sub channel for low-latency progress fan-out to connected gateways. The MCP client never touches the queue — it talks to the gateway, which translates internal events into protocol notifications and answers polls from state. That indirection lets one surface serve stdio, SSE, and Streamable HTTP clients without leaking infrastructure.
—
## 5. Error Handling: Managing Timeouts and Partial Failures in Long-Lived Sessions
An async job multiplies your failure surface, because every handoff is a new place to lose work. Enumerate them: the request never reaches the queue; a worker crashes mid-job; the state store evicts the job before the client fetches it; a notification is dropped; the session disconnects and reconnects under a new transport; the *agent* times out even though the job is healthy; and — the nastiest case — the job *partially* fails, completing 48 of 50 shards.
**Idempotency is the foundation.** If a client retries a timed-out submission, you must not spawn a second export of the same two million rows. Accept an idempotency key (or derive one from a stable hash of tool name and arguments) and return the *same* job ID for the same logical submission. Tasks leaves idempotency to implementers — real networks do not — so build it yourself: check the key before enqueueing, and make the check and enqueue atomic.
**Leases prevent zombie work.** Every executing job carries a lease the worker must renew by heartbeat. Redis: `SET job:ID lease
**Decouple the agent’s patience from the job’s lifetime.** The client’s tool-call timeout firing is *information*, not a cancellation order. The job keeps running, and the result must be retrievable after the client reconnects — which is exactly why `tasks/list` exists: a restarted agent rediscovers in-flight work instead of orphaning it. Retain terminal results for the TTL the server committed to (hours to days, not seconds), and make terminal states immutable: once `completed` or `failed`, no late event may move it.
**Model partial failure honestly.** A batch job that fails on 2 of 50 shards should not fail the whole job. Return a structured result that separates protocol errors from execution outcomes:
“`json
{
“content”: [{ “type”: “text”, “text”: “Export completed with 48/50 shards.” }],
“structuredContent”: {
“status”: “completed”,
“partial”: true,
“completed”: 48,
“failed”: [{ “shard”: “orders_2019”, “error”: “ORA-04030: out of process memory” }],
“retryCommand”: “run_export”
}
}
“`
The client can retry exactly the failed shards with the same idempotency key — cheap, safe, visible recovery. Under Tasks, execution failure is `status: failed` with diagnostics, while malformed requests stay JSON-RPC errors; keeping those layers separate keeps error handling composable. And put metrics on the loop — enqueue latency, lease expiry rate, time-to-terminal — because the earliest symptom of trouble in a job system is a metric, not an error.
—
## 6. Implementation: An Async MCP Tool with Progress Updates in Node.js
Here is the pattern in one Node.js sketch: an MCP gateway process (Express + the official SDK over SSE) and a worker sharing a Redis store. The tool returns in milliseconds; the worker does the minutes-long work; progress flows back as `notifications/progress`.
“`js
// gateway.mjs — MCP server exposing an async tool
import { McpServer } from “@modelcontextprotocol/sdk/server/mcp.js”;
import { SSEServerTransport } from “@modelcontextprotocol/sdk/server/sse.js”;
import { createClient } from “redis”;
import { z } from “zod”;
import { createId } from “@paralleldrive/cuid2”;
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();
const server = new McpServer(
{ name: “data-mcp”, version: “1.0.0” },
{ capabilities: { tools: {} } }
);
// In-memory: activeToken -> { transport, sessionId }
// Used to forward progress to the right live session.
const tokenIndex = new Map();
server.registerTool(
“run_export”,
{
source: z.enum([“crm”, “erp”, “datawarehouse”]),
destination: z.string(),
},
async ({ source, destination }, extra) => {
const jobId = createId(); // unguessable, sortable
const progressToken = extra._meta?.progressToken ?? `job:${jobId}`;
// Durable state: source of truth for polling/reconnect.
await redis.hSet(`job:${jobId}`, {
status: “queued”, progress: “0”, destination,
token: progressToken, createdAt: Date.now(),
});
await redis.expire(`job:${jobId}`, 86_400); // 24h retention
// Keep a mapping so the worker can find this session later.
tokenIndex.set(progressToken, jobId);
// Dispatch — worker picks this up within ms.
await redis.lPush(“jobs:pending”, JSON.stringify({
jobId, source, destination, progressToken,
}));
return {
content: [{
type: “text”,
text: `Queued export job ${jobId}. Poll get_job, or expect progress notifications.`,
}],
structuredContent: { jobId, status: “queued”, pollIntervalMs: 5000, ttlMs: 86_400_000 },
};
}
);
// Worker progress events arrive here; forward as MCP notifications.
// Gateway subscribes once and routes by token -> live session.
async function forwardProgress(channel, raw) {
const evt = JSON.parse(raw);
const job = await redis.hGetAll(`job:${evt.jobId}`);
const session = tokenIndex.get(job.token); // resolve to live stream
if (!session) return; // client disconnected — state store covers recovery
session.transport.session.sendNotification({
method: “notifications/progress”,
params: { progressToken: job.token, progress: evt.progress, total: evt.total, message: evt.message },
});
}
await redis.subscribe(“job:events”, forwardProgress);
// get_job: the poll side of “poll for truth, listen for speed”
server.registerTool(“get_job”, { jobId: z.string() }, async ({ jobId }) => {
const job = await redis.hGetAll(`job:${jobId}`);
if (!job?.status) throw new Error(`Unknown job: ${jobId}`);
return { content: [{ type: “text”, text: JSON.stringify(job) }] };
});
// HTTP wiring (SSE transport) — abbreviated, see companion post on SSE.
const sessions = new Map();
import express from “express”;
const app = express();
app.get(“/sse”, async (req, res) => {
const t = new SSEServerTransport(“/messages”, res);
sessions.set(t.sessionId, t);
await server.connect(t);
});
app.post(“/messages”, express.json(), async (req, res) => {
const t = sessions.get(String(req.query.sessionId));
if (t) await t.handlePostMessage(req, res);
});
app.listen(3001);
“`
“`js
// worker.mjs — drains jobs, updates state, publishes progress
import { createClient } from “redis”;
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();
while (true) {
const [, raw] = await redis.brPop(“jobs:pending”, 0); // blocking pop
const { jobId, source, destination, progressToken } = JSON.parse(raw);
const lease = await redis.set(`lease:${jobId}`, workerId, { NX: true, EX: 60 });
if (!lease) continue; // another worker claimed it — idempotent skip
try {
await redis.hSet(`job:${jobId}`, { status: “running”, worker: workerId });
const total = await countRows(source);
for await (const [i, batch] of streamBatches(source, destination)) {
await renewLease(jobId); // heartbeat
await redis.publish(“job:events”, JSON.stringify({
jobId, progress: i, total, message: `batch ${i}/${total}`,
}));
await insertBatch(batch); // the actual minutes-long work
}
await redis.hSet(`job:${jobId}`, { status: “completed”, progress: total });
await redis.publish(“job:events”, JSON.stringify({ jobId, done: true }));
} catch (err) {
await redis.hSet(`job:${jobId}`, { status: “failed”, error: String(err) });
// or push to a retry/DLQ list with the idempotency key intact
} finally {
await redis.del(`lease:${jobId}`);
}
}
“`
On the client side, attach `_meta.progressToken` and treat `notifications/progress` as a hint layer only — always confirm the terminal state via `get_job` before acting. If your SDK has adopted Tasks, the shape maps directly: send `task: { ttl }`, poll `tasks/get`, and block on `tasks/result` for the final payload; the durability, lease, and idempotency machinery above is unchanged.
—
## Conclusion
Synchronous JSON-RPC is the right tool for the wrong job the moment a tool call exceeds an HTTP proxy’s patience. The fix is architectural: keep the RPC fast by returning a job handle, make the job a durable state machine in a store the client can poll, and use notifications only as an acceleration layer over that truth. Queue the work for a fleet that outlives any connection, lease it so crashes cannot orphan it, and make retries idempotent so a client cannot double-run an export. That combination — fast ack, durable state, event-driven progress, pollable truth — is what turns MCP from a demo protocol into something that drives real enterprise workloads for minutes at a time.
This is the kind of architecture that looks simple on a whiteboard and gets subtle in production: session ownership across a gateway fleet, state stores, idempotency keys, and knowing which channel is authoritative. It is what the senior engineers at OnlineCloud build every day — MCP-native systems with Redis and AWS at the core, audited end-to-end, shipped in weeks, not quarters. If you are putting long-running tools behind MCP, **talk to the OnlineCloud architects** — we would rather argue about your lease timeout than sell you a deck. [Contact us](https://www.onlinecloud.dev/) or write to contact@onlinecloud.dev — free initial consultation, response within 24 hours.
Leave a Reply