Author: sandeepsingh

# Optimizing Prompt Caching in Anthropic Bedrock for Complex Schemas


*By the OnlineCloud Engineering Team — LLM Infrastructure Engineers*

—

## 1. Introduction: The Quiet Tax on Large Tool Surfaces

Every agentic system hits the same inflection point: the moment the tool belt outgrows the model’s default economics. Early demos carry two or three tools and prefixes stay small. Then production happens. Your MCP server exposes forty tools, each carrying a name, a prose description, and a JSON Schema — and the tools wrapping your Oracle or CRM objects run to hundreds of tokens alone. Add a system prompt with governance rules and output contracts, and every request now opens with a 30,000-to-80,000-token preamble.

Here is the part that surprises platform teams: **that preamble is re-billed and re-processed on every request.** Each invocation of the Messages API is a fresh prefill; the provider charges full input price for every token it processes, system prompt and schemas included. At $3 per million input tokens, a 50,000-token prefix is $0.15 of dead weight before the model emits a single useful word. At millions of requests a month, the tool surface becomes a six-figure line item — and because time to first token (TTFT) grows with prompt-processing cost, the biggest tool set is also the slowest one.

The good news: this cost is almost entirely avoidable. Anthropic’s prompt caching on Bedrock is built for exactly this pattern — a large, stable, repeated prefix — and a well-structured MCP deployment is the ideal workload. This post covers how the cache works, how to structure MCP tool schemas so you hit it, what the 90% input discount looks like in production arithmetic, how to benchmark the TTFT win, and how to govern prompt changes so the cache stays warm on purpose rather than by accident.

## 2. How Anthropic Prompt Caching Actually Works

Prompt caching is a **prefix cache with exact-match semantics at the token level**. You annotate positions in the request with cache breakpoints — a `cachePoint` marker in the Bedrock Converse API (`cache_control` in the raw Anthropic API) — and the platform caches everything preceding each breakpoint. The next request that opens with an identical token prefix is served from cache instead of re-running the model’s prefill stage.

Three mechanics drive every engineering decision:

**Prefix, not semantic, matching.** The cache key is the token stream, not the meaning of the request. One changed byte upstream — a reworded system-prompt sentence, a tool inserted mid-list, a JSON object whose keys serialized in a different order — produces a different stream and a complete miss. Matching is strictly positional: once the stream diverges, everything after it is uncached.

**Minimum size and TTL tiers.** A breakpoint must sit at least 1,024 tokens into the prompt for the 5-minute TTL tier (the 1-hour tier requires a longer prefix). Entries are sliding-window: each read refreshes the TTL, so a hot prefix stays resident indefinitely while a cold one expires minutes after its last request. A prefix that sees traffic only once an hour will never stay warm on the 5-minute tier — that is what the 1-hour tier is for.

**Pricing asymmetry.** Cache reads cost a fraction of the base input price; writes cost a premium:

| Tier | Cache write (per input token) | Cache read (per input token) |
|—|—|—|
| 5-minute TTL | 1.25× base input price | 0.1× base input price |
| 1-hour TTL | 2.0× base input price | 0.1× base input price |

Reads are where the 90% headline comes from; writes are the price of admission, paid once per TTL window per distinct prefix. The strategy is therefore to maximize the read-to-write ratio by keeping the prefix byte-identical across as many requests as possible.

A Bedrock Converse request with breakpoints after the system prompt and after the tool list:

“`python
response = bedrock.converse(
modelId=”anthropic.claude-sonnet-…:0″,
system=[
{“text”: SYSTEM_PROMPT}, # static, versioned, cached
{“cachePoint”: {“type”: “default”}},
],
toolConfig={
“tools”: [
*serialize_tools(TOOL_REGISTRY), # byte-stable, ordered
{“cachePoint”: {“type”: “default”}},
]
},
messages=[{“role”: “user”, “content”: [{“text”: user_query}]}],
)
“`

Bedrock supports multiple breakpoints per request on compatible model versions, so you can cache the fully static region and keep volatile content — per-tenant instructions, retrieved context — behind your own breakpoint. The rule of thumb: **everything static goes before a breakpoint; everything that varies per request goes after one.**

## 3. Structuring MCP Tool Definitions and Schemas for Cache Hits

Getting cache hits is not a configuration task — it is a serialization discipline. In an MCP-native architecture, the MCP server is the source of truth for tool definitions, and an adapter layer converts them into the tool payload of every model request. That conversion must be deterministic, or you will silently invalidate the cache on every request.

**Canonical, byte-identical serialization.** The cache matches on tokens, so two semantically identical requests that serialize differently are two different prefixes. Pick one canonical serialization — stable tool ordering plus fixed JSON key order — and enforce it in every process that builds requests. Beware runtimes that hash-map keys into arbitrary order, and never build tool dicts ad hoc in request handlers. Serialize once from the registry and reuse the bytes.

**Freeze ordering; grow append-only.** Token streams are positional: inserting a tool mid-list invalidates every cached segment after it, fleet-wide. New tools go at the end of the registry; reordering is a cache-invalidation event, not a cosmetic change. Removing a tool is a breaking deployment — version-bump, warm the new prefix, then shift traffic.

**Keep dynamic content out of the cached region.** Tool descriptions and system text must not be templated with per-request, per-tenant, or time-varying values — no “as of {date}”, no environment names, no user clauses inside the cached segment. The most common cache-killer we audit is a system prompt interpolating a timestamp at the top, quietly changing the prefix on every request.

**Curate the giant schemas.** Enterprise tools wrapping real database objects carry most of your prefix tokens — which is precisely why caching matters for them. Trim redundant `description` prose, drop field documentation duplicated between description and `inputSchema`, and avoid scattering the same large schema across near-duplicate tools. Every cached token is billed at write premium and saved at read discount; verbosity has a price either way.

**Right-size the tool surface per agent.** If your MCP server exposes 60 tools but an agent class uses eight, every request pays for the other 52. Partition tool sets by domain — order-status agents get the read-only CRM and order tools, reconciliation agents get finance tools — so each agent class has a smaller, cheaper, more stable prefix, with less drift surface and cheaper re-warm after changes.

**Lint the wire format in CI.** Add a golden-file test that builds the exact request payload and diffs it against the committed artifact. Any change fails the build and forces an explicit decision — which is what you want, because in a cached system a prompt edit is a production change with cost and latency consequences.

## 4. Cost Analysis: Measuring the 90% Discount in Production

The discount is real, but it should be evaluated end-to-end on your own workload. Take a representative profile: a tool-heavy agent with a 12,000-token system prompt and 33,000 tokens of MCP schemas — 45,000 input tokens per request — serving 100,000 requests/day against a Claude-class model at $3/MTok input, $15/MTok output, ~800 output tokens per request.

**Without caching**, every request pays full price for the prefix: 100,000 × 45,000 × $3/MTok = **$13,500/day** in input cost, on top of ~$1,200/day of output tokens.

**With a stable cached prefix**, each 5-minute window pays one 1.25× write and the rest read at 0.1×. At roughly uniform traffic, that is ~288 writes and ~99,700 reads per day:

– Writes: 288 × 45,000 × $3 × 1.25 ≈ **$49/day**
– Reads: 99,700 × 45,000 × $3 × 0.10 ≈ **$1,345/day**
– Total input cost: **~$1,394/day — a 90% reduction**; output cost unchanged.

That ratio holds only while the prefix stays byte-identical and traffic keeps the TTL alive. Fail either and every request pays write premiums — which is why instrumentation is mandatory.

Bedrock exposes the counters you need — `cacheReadInputTokens` and `cacheWriteInputTokens` in the invocation metrics — which you can stream to CloudWatch alongside raw token counts. Track **hit ratio = reads / (reads + writes)** per prompt version, model, and region. Healthy systems sit above 95%; anything below is almost always serialization drift, tool reordering, region-split traffic, or TTL expiry during low-traffic hours. If inter-request gaps routinely exceed five minutes, the 1-hour tier (2× write, same 0.1× read) is worth the premium — run the arithmetic with your actual cadence before choosing.

## 5. Benchmarking TTFT with Cached Context

Cost is half the story. TTFT is dominated by prefill — the model processing the entire prefix before emitting anything — so a 45,000-token preamble is not just expensive, it is slow. Caching removes most of that work from the critical path, and on long prefixes the latency win matches the cost win.

Methodology matters. Measure TTFT from request submission to the first streamed token, and confirm cache hits per request via the read-token counters — otherwise you are comparing a hit population against a miss population and calling it a model improvement. Run an A/B against production-shaped traffic: the same system prompt and tool registry, many different conversations, so you measure real prefix reuse rather than identical repeated queries.

Representative numbers for a 45,000-token prefix on Bedrock (p50/p95 across a production mix):

| Metric | Cache miss (cold prefix) | Cache hit (warm prefix) |
|—|—|—|
| TTFT p50 | 4.9 s | 0.8 s |
| TTFT p95 | 8.2 s | 1.6 s |
| Input cost per request | $0.135 | ~$0.014 |

An 80–85% TTFT reduction is typical at this prefix size, and the absolute gap grows with prefix depth — the larger your schemas, the bigger the win. Two consequences follow. First, **interactive quality changes**: a tool-calling agent that responds in under a second instead of after a five-second pause feels fundamentally different, and gateway timeout budgets stop being a design constraint.

Second, **savings compound across multi-step loops.** An agent resolving a ticket may run a ten-step tool chain, and nearly every step re-sends the full prefix. Cache each step and the 90% input discount and ~85% prefill reduction apply ten times per resolved task — the difference between a loop that costs dollars per completion and one that costs cents. That is why we treat prompt caching as a first-class architectural constraint when designing MCP tool surfaces, not a billing optimization applied afterward.

## 6. Governance: Automating Prompt Versioning for Cache Stability

A cached deployment turns every prompt edit into an infrastructure event. Change one system-prompt sentence and you invalidate the cache for the entire fleet: the next request from every agent pays a write premium and eats a cold prefill. Edit prompts by hand in request handlers and you will do this silently and constantly. The fix is to treat the system prompt plus tool registry as a versioned, deployable artifact with the same discipline as application code.

**The artifact.** The canonical prefix — system text, ordered tool list, serialized schemas — lives in version control (or an S3-backed registry) as one immutable bundle. A build step produces the byte-serialized prefix and its content hash. Runtimes load a pinned version; nothing constructs prompts at request time.

**The CI gates.** Changes to the bundle run through checks encoding the caching rules: a golden-file test asserting byte-identical serialization; an ordering lint enforcing append-only tool growth; a scan rejecting time-varying or per-request interpolation inside the cached region; a token check confirming the prefix still exceeds the 1,024-token minimum. A prompt that fails these gates cannot reach production.

**Versioned rollout with cache warm-up.** Because a release is a cache-invalidation event, deploy it like a risky change: release the new prefix version, run synthetic warm-up traffic to populate the cache before users arrive, then ramp 5% → 100%. This prevents the classic post-release symptom — a normal traffic day billing like a cold start because every request is a cache write. Tag request observability with the prompt version so hit rate and cost are attributable per release.

**Rollback is nearly free.** Entries persist for the TTL after their last read, so reverting to the previous version often restores a still-warm cache within minutes — the platform rolls back with your artifact. That asymmetry (cheap rollback, expensive forward drift) argues for small, reviewed prompt changes over large rewrites.

**The SLO.** Track cache hit ratio per prompt version and alert on drops. A sudden hit-rate cliff is the earliest signal of serialization drift, an accidental reorder, or a rogue templated value — visible in the token counters hours before it shows up in the cost report.

None of this is exotic. It is the release engineering you already run for services, applied to the one artifact most teams still edit by hand — and it is the difference between prompt caching as a lucky discount and prompt caching as a reliable production lever.

## Conclusion: Make Your Schema the Asset

Large tool surfaces and long system prompts are not a mistake — they are the shape of real enterprise agentic systems talking to real schemas. The mistake is paying full price for them on every request. Bedrock prompt caching turns the tax on complex schemas into a 90%-discounted, dramatically faster repeating prefix — but only if you engineer for it: canonical serialization, frozen tool ordering, volatile content quarantined behind breakpoints, and prompt changes governed like deployments.

At OnlineCloud we build MCP-native agentic systems for a living, designing the tool surface, serialization layer, and caching strategy as one system — measured by cost per resolved task and p95 TTFT rather than demo quality. If your agent fleet is paying full freight for a tool belt it re-sends on every call, talk to us. We are a US-based engineering studio of senior engineers — no offshore handoffs — shipping evaluated, observable systems to production in 6–10 weeks.

**Contact the OnlineCloud team today** — [onlinecloud.dev](https://www.onlinecloud.dev/) · [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) · Free initial consultation, response within 24 hours.

# Multi-Agent Conflict Resolution: Implementing Borda Count and Voting Patterns

*By the Principal AI Scientist team at OnlineCloud — Agentic AI & MCP Engineering Studio*


—

## 1. Introduction: When Agents Disagree

Every production agentic system eventually collides with one fact: **the same prompt, tools, and context do not guarantee the same answer.** Sample one LLM ten times at nonzero temperature and you get ten ranked candidates. Put ten differently-instructed agents — a retrieval agent, a code agent, a reasoning agent, each with its own model and tool access — on one question, and you get ten *confident, mutually exclusive* answers. This is not a defect to engineer away; it is the fundamental non-determinism of stochastic language models reasoning over partial information.

Teams react in one of two ways. The first: pick a “best” agent and ignore the rest — cheap, but it inherits that agent’s blind spots. The second: run a naive majority vote over final answers — better, but it silently assumes every opinion is equally trustworthy. A specialist who has worked with this exact schema for three years and a generalist who has seen it for three seconds get equal weight. That is not consensus; it is noise averaging.

The mature position treats disagreement as a **first-class signal** and resolves it with a principled decision procedure. This post walks through what that looks like in practice: Plurality, Borda Count, and Condorcet methods adapted to swarms; weighting that respects confidence and expertise; the central-Judge-versus-peer-to-peer architecture choice; and bounding time-to-consensus so resolution never becomes the bottleneck it was meant to prevent. Working implementations are included throughout — a consensus layer you cannot ship is just a slide.

—

## 2. Voting Algorithms: Plurality, Borda Count, and Condorcet in Swarms

### 2.1 Why ranking beats single-choice ballots

The most important design decision is the *ballot format*. Plurality asks each agent for one preferred answer — easy to collect but pathologically lossy. It discards everything the agent knows about its *second-best* candidate, and when agents disagree because each holds different evidence, second choices frequently encode the compromise first choices cannot see. The fix: ask each agent for a **total or partial ordering** over the candidates. Elicitation is cheap — an LLM can rank ten candidates in one structured-output call — and it unlocks every ranked method in social choice theory.

### 2.2 Plurality: the baseline

Plurality is the counting rule you already know: each ballot contributes one point to its first-ranked candidate; the most points wins.

“`python
def plurality(ballots: list[list[str]]) -> str:
scores: dict[str, int] = {}
for ballot in ballots:
top = ballot[0]
scores[top] = scores.get(top, 0) + 1
return max(scores, key=scores.get)
“`

It is O(N) to tally and trivially explainable — which is why it survives in production. But it fails the moment the swarm splits three ways while every agent’s second choice is the same. Plurality can elect a candidate a majority actively ranks last.

### 2.3 Borda Count: reward the whole ranking

Borda Count scores every position, not just the first. With *m* candidates, an agent’s top choice receives *m−1* points, its second *m−2*, and so on; the highest total wins. It is the workhorse of swarm consensus because it is **easy to elicit, deterministic to tally, and robust to small perturbations** in individual orderings.

“`python
def borda(ballots: list[list[str]]) -> list[tuple[str, float]]:
m = len(ballots[0]) # assume complete rankings
scores: dict[str, float] = {}
for ballot in ballots:
for rank, cand in enumerate(ballot):
scores[cand] = scores.get(cand, 0.0) + (m – 1 – rank)
return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
“`

Implementation notes from the field:

– **Handle incomplete ballots.** Some agents refuse to rank candidates outside their expertise. Padding unranked candidates with zeros silently punishes honesty; prefer *normalized Borda*, scaling each ballot so its total is constant. That neutralizes partial ballots and over-confident agents alike.
– **Decide the tie-break policy in advance.** Borda ties are common in small swarms: break by plurality count, then mean self-reported confidence, then deterministically by hash. A policy invented during an incident is a policy written by whoever is loudest.

Borda’s known weakness is strategic voting — an agent can bury a rival by ranking it last. Inside a swarm this is less malice than *prompt drift*: an agent told to “be decisive” systematically distorts its ranking. Mitigate with explicit instructions to rank by genuine preference and by monitoring ballot distributions for anomalies.

### 2.4 Condorcet: the gold standard (and its cycle problem)

The Condorcet criterion asks a sharper question: *is there a candidate who beats every other candidate in pairwise head-to-head comparison?* Such a Condorcet winner is the most defensible outcome — the candidate a majority prefers against each alternative.

“`python
from itertools import combinations

def pairwise(ballots: list[list[str]]) -> dict[tuple[str, str], int]:
“””Returns { (a, b): number of ballots ranking a above b }.”””
wins: dict[tuple[str, str], int] = {}
for ballot in ballots:
pos = {c: i for i, c in enumerate(ballot)}
for a, b in combinations(ballot, 2):
key = (a, b) if pos[a] < pos[b] else (b, a) wins[key] = wins.get(key, 0) + 1 return wins def condorcet_winner(ballots: list[list[str]]) -> str | None:
wins = pairwise(ballots)
cands = set().union(*(set(b) for b in ballots))
for c in cands:
if all(wins.get((c, o), 0) > wins.get((o, c), 0)
for o in cands if o != c):
return c
return None # no Condorcet winner — cycle
“`

The catch: majority preferences can be *intransitive* — agent A prefers X over Y, Y over Z, yet Z over X. This is Condorcet’s paradox, and in swarms it is not an exotic corner case; agents weighting different evidence dimensions is precisely the structure that produces cycles. When no Condorcet winner exists, fall back to a *completion method* — Ranked Pairs, Schulze’s beatpath, or simply Borda. My production policy is two-stage:

1. If a Condorcet winner exists, return it (and log it — that is strong evidence).
2. Otherwise, resolve with Borda Count and *surface the cycle in the audit trail*.

A cycle is not a failure; it is a diagnostic gift. It tells you the evidence is genuinely polarized and no answer carries majority legitimacy — the exact moment to escalate to a weighted Judge (§4) or a human checkpoint.

—

## 3. Weighted Decisions: Confidence and Expertise

Unweighted voting assumes agent opinions are exchangeable. They are not. Two factors matter most, and both should be explicit.

### 3.1 Calibrated confidence

Have each agent report a confidence score alongside its ranking — but *never* trust the raw number. LLM confidence is notoriously miscalibrated: overconfident out-of-distribution, underconfident on routine inputs. Calibrate reported confidence against historical accuracy the way you would any classifier.

“`python
def borda_weighted(ballots: list[list[str]],
weights: dict[str, float]) -> dict[str, float]:
“””Borda with per-agent weights, e.g. calibrated confidence.”””
m = len(ballots[0])
scores: dict[str, float] = {}
for agent, ballot in ballots:
w = weights.get(agent, 1.0)
for rank, cand in enumerate(ballot):
scores[cand] = scores.get(cand, 0.0) + w * (m – 1 – rank)
return scores
“`

Collect per-agent accuracy over time — against ground truth when available, cross-agent agreement and human review otherwise — and fit a monotonic map from reported confidence to empirical precision. Deploy the map, not the raw score. If an agent’s confidence tracks its accuracy at all, calibration shifts outcomes more than any cleverness in the tally rule.

### 3.2 Expertise as a prior weight

The second weight is *situational expertise*: how relevant is this agent to *this question*? A SQL agent queried on legal jurisdiction should not vote with the authority of a legal agent. Compute a task-expertise score at query time — from tool-usage history, retrieval overlap with the question’s domain, or a lightweight query classifier — and multiply it in. The effective weight of agent *i* on question *q* is:

“`
w_i(q) = calibration_i(confidence_i) × expertise_i(q)
“`

One caution: expert weights can ossify into a self-fulfilling prophecy — the legal agent always wins legal questions, so it gets more legal questions, so its weight grows. Add a mixing term or periodic re-evaluation so minority experts keep a voice, and monitor the *effective number of voters* to catch when the swarm collapses into a single-agent echo. Also cap the weight ratio between strongest and weakest voter (5:1 works), and require the winner to clear a *confidence floor* τ — calibrated probability, not raw margin. If nothing clears τ, the correct output is not the best available answer; it is *abstention plus escalation*.

—

## 4. Architecture: Central Judge vs. Peer-to-Peer Consensus

The tally rule is orthogonal to the *architecture* that collects ballots. Two patterns dominate, mapping onto classic distributed-systems trade-offs.

### 4.1 The central Judge

A single orchestrator (the “Judge”) fans out the question, collects ranked ballots, applies the weighted tally, and returns a verdict. This fits MCP-native orchestration naturally: the Judge is one more agent with tool access, and ballot collection is N tool calls. Judge-based systems are simple to reason about, trivially auditable (one component owns the tally), and easy to make deterministic — the Judge can be a temperature-0 call over pre-computed scores, so the *voting layer* has zero non-determinism even though the voters do not.

The risks are concentration and bias: a single point of failure, a single point of capture, and a tempting prompt-injection target if it alone sees the full picture. Mitigations: keep the Judge *stateless and dumb* — it should tally, not opine, because the moment it injects its own reasoning you have an unweighted extra voter — run it deterministically, and log the full ballot transcript.

### 4.2 Peer-to-peer consensus

At the other extreme, agents exchange and revise rankings directly — a gossip or all-pairs protocol where the aggregate belief state is the ballot and agents iterate toward a fixpoint. This fits federated swarms across organizations, or any setting where a central Judge is itself a latency and trust bottleneck.

P2P consensus inherits decades of distributed-systems machinery: you are effectively running a Byzantine agreement protocol whose “messages” are rankings and whose “faults” are miscalibrated or adversarial agents. Guaranteed agreement under faults means something PBFT-like — O(N²) messages, 3f+1 replication. For most commercial workloads that is overkill: honest agents disagreeing is not the Byzantine generals problem, and treating it as such multiplies infrastructure cost for zero accuracy gain.

My guidance: **start with a Judge, instrument the failure modes, and decentralize only when a concrete requirement — no single point of trust, cross-org boundaries, latency — forces it.** The hybrid is strongest: P2P *deliberation* (agents exchange summaries and revise rankings — this is where new evidence surfaces) followed by Judge-based *resolution* (a deterministic weighted tally on the final rankings). Separate the conversation from the vote and you get the epistemic benefit of consensus with the auditability of a central record.

—

## 5. Performance: Bounding Time-to-Consensus

Consensus layers fail in production when they are slow, because latency is the metric everyone notices. Bound it explicitly with a three-term cost model:

“`
T_consensus = T_elicit + T_tally + T_rounds
“`

– **T_elicit** — one LLM round-trip per agent, fully parallelizable. This dominates: serial elicitation of ten agents is ten latencies; parallel is one.
– **T_tally** — microseconds for any method in §2; all-pairs Condorcet is O(N·m²), trivial at N=20, m=10. Tallying is never your bottleneck; stop optimizing it.
– **T_rounds** — the hidden cost. Multi-round deliberation multiplies T_elicit by the round count.

Three disciplines keep time-to-consensus bounded:

1. **Cap the rounds.** Deliberation returns diminish sharply after round two while latency compounds linearly. Set `max_rounds=2`; round three requires a human or an explicit budget override.
2. **Early-stop on dominance.** If a Condorcet winner exists after round one *and* its margin over the runner-up clears a threshold, declare victory. Most disagreements are shallow — in our eval sets, 60–70% — and you should not pay deliberation cost on questions that were never truly contested.
3. **Time-box and fall back.** Give the swarm a deadline. On expiry, resolve with whatever ballots exist using normalized Borda so partial participation does not bias the tally. A guaranteed latency ceiling is what makes the consensus layer safe to place on the critical path.

Finally, measure it like an SLO: p50/p95 time-to-consensus per question type, per model, per swarm size. When p95 drifts, it is almost always one slow voter — the fix is a per-agent latency cap with abstention, not a faster tally.

—

## 6. Real-World Use Case: Diagnosis and Legal Interpretation

### 6.1 Differential diagnosis in a clinical reasoning swarm

Consider a diagnostic swarm: a radiograph agent, a lab-value agent, a clinical-history agent, and a literature agent, each returning a ranked differential over five candidate conditions. This structure breaks plurality: the radiograph agent is confident in *pneumonia*, the lab agent in *pulmonary embolism*, the others split further — yet *all five* rank *heart failure* second. Plurality inflates the false dichotomy; Borda surfaces the compromise candidate every specialist could live with, which is exactly the differential a prudent clinician wants to work next.

Now add weights: the radiograph agent’s confidence on imaging questions is historically well-calibrated, so its weight is high; the generalist’s is low. Critically, the swarm can flag *its own insufficiency* — if the winning differential’s calibrated probability sits below the escalation threshold τ, the correct behavior is to page a human radiologist, not emit a confident wrong answer. The voting layer just converted a silent misdiagnosis risk into an explicit, documented escalation event.

### 6.2 Statutory interpretation across jurisdictions

Legal interpretation is the adversarial mirror image. Here agents are not collaborators but *advocates*, each briefed with one jurisdiction’s precedent, one canon of construction, one reading of legislative intent. Because voters are deliberately polarized, expect Condorcet cycles: interpretation A beats B in the textualist camp while B beats A among purposivists, and no majority winner exists.

This is where the two-stage policy earns its keep. A Condorcet winner means the question is more settled than the advocates’ posturing suggested — return it. If the tally cycles, the system should *say so*: emit the Borda outcome *labeled with the cycle*, publish the pairwise margins, and escalate to a human review board. The cycle is not a bug; it is a formal proof that the question is genuinely contested — legally meaningful information in itself. In regulated settings, the audit trail — full ballots, weights, pairwise matrix, and the rule that produced the verdict — is often more valuable than the verdict. MCP’s typed, logged tool calls make capturing that transcript a default property of the architecture rather than an afterthought.

—

## 7. Conclusion: Make Disagreement a Feature

Non-determinism is not going anywhere; it is the price of models that generalize. The professional response is not to hide from disagreement but to give it a **decision procedure**: rank-ordered ballots, Borda for robust aggregation, Condorcet for majority legitimacy, calibrated confidence and situational expertise as weights, a deterministic Judge for resolution, bounded rounds for latency, and escalation as a first-class outcome. Done right, the consensus layer turns a swarm’s divergence from a reliability liability into a measurable source of robustness — with the audit log to prove it.

**At OnlineCloud we ship these patterns in production — MCP-native orchestration with typed tools, evaluated outcomes, and consensus layers wired into Oracle, MS SQL, and Postgres.** If “which answer wins” in your multi-agent workload is still decided by vibes or a hardcoded if-statement, that is exactly the conversation we want to have.


—

## Ready to wire consensus into your agent stack?

We are a US-based Agentic AI & MCP engineering studio. Senior engineers scope it, build it, and operate it — no offshore handoffs, no decks, no quarters-long timelines. Most engagements ship to production in **6–10 weeks**.

– **Email:** contact@onlinecloud.dev
– **Web:** https://www.onlinecloud.dev/
– **Support:** Dedicated USA-based support · Free initial consultation · Response within 24 hours

*Tell us which agents in your system are arguing — we will show you the voting layer that settles them.*

# Designing a Self-Healing Agent Loop: Automated Error Recovery and Retries

*When the code path is generated per-run by a stochastic model, your retry policy can’t be a static snippet. Here is how an SRE makes an agent classify failures, recover from them, and stop amplifying outages.*


## 1. Introduction: The Fragility of Autonomous Agents in a Brittle API World

Every engineer who has run an agent against real APIs knows the death scene: the tool times out, and the agent calls it again with the same arguments, again and again, while the upstream stays down. Nobody wrote a retry loop; the model *became* it.

A deterministic service survives production entropy because its error handling was written once, in code: backoff, jitter, idempotency keys, a circuit breaker. An agent has none of that by default. Its “error handling” is improvised at runtime, under token pressure, by a model that treats every failure as a puzzle rather than a signal to stop. **In an agentic system, the retry policy is generated per-run by the thing that is already failing.**

Production APIs punish this: p99 spikes, 429 rate limits, 5xx during deploys, 2 a.m. maintenance resets. The failure shapes are consistent: the **retry spiral** (identical calls until a ceiling or a human intervenes), the **confident drift** (after two errors the model stops calling tools and invents answers), and the **cascade** (one poisoned result invalidates every downstream step). These failures are behavioral, not technical — the system returns success while doing the wrong thing, or quietly burns budget.

The fix is not to prompt the model to “be more careful.” It is to move error handling out of the context window and into the execution machinery around the model: **classify the failure, pick a recovery strategy, and enforce limits the model cannot override.** That is the same architectural move the Model Context Protocol made for tooling: taking invariants out of prose and putting them into structure. A self-healing loop is what you get when the runtime cares about failure as much as the model cares about the task.

## 2. Error Classification: Model Errors vs. Transient API Failures

You cannot heal what you cannot classify — and you cannot classify with a model. Asking an LLM “what went wrong?” on a 500 costs tokens and produces prose instead of policy. **Classification belongs at the tool boundary, in the runtime**, where every tool wrapper returns a structured error envelope before the model ever sees the failure:

“`json
{
“error”: {
“class”: “transient”,
“code”: “RATE_LIMITED”,
“retryable”: true,
“retry_after_ms”: 30000,
“tool”: “crm.account.update”,
“hint”: “honor Retry-After; do not regenerate arguments”
}
}
“`

Three classes cover nearly every production failure, and each maps to one family of recovery:

– **Transient API failures** — 429s, 5xx, timeouts, resets. A retry *can* succeed: the request was fine, the infrastructure was not. Recovery is deterministic — exponential backoff with jitter, honor `Retry-After`, retry the identical payload under an idempotency key. Never ask the model to “improve” a request that was never the problem.
– **Contract failures** — 400/422 schema rejections, missing resources, expired auth. Also deterministic, in the opposite direction: retrying an identical payload is pure waste. Recovery means repairing the plan — fix the arguments, refresh the credential — then re-executing; a contract failure that recurs after repair is a stop-and-escalate signal.
– **Model errors** — wrong tool chosen, arguments that pass schema but violate business semantics, a plan that contradicts verified results. Not a network class; retrying is meaningless because the same model repeats the same decision. This is the expensive class, because detecting it needs a **verification step** — a read-only check that output matches the goal — not a status code.

Two rules keep it honest. Classify on code and envelope, never payload heuristics: a 500 with an `UPSTREAM_TIMEOUT` body is transient; a 200 with an empty result may be a model error. Treat the classifier as policy, not model judgment — version it with the tool schema, change it without shipping a prompt.

## 3. Recovery Patterns: Backtrack, Plan-Repair, and Tool-Switching

With errors classified, the executor dispatches to one of three patterns from a policy table. They are deliberately few: the runtime must reason about them, and the model must be constrained by them — let the model choose among *visible, eligible* actions; let the runtime decide what is visible and eligible.

**Backtrack — rewind, don’t replay.** Every completed step appends to a checkpoint: plan state, tool result, idempotency key. When an error invalidates the branch, the executor rewinds to the last validated checkpoint — you checkout a known-good state instead of performing surgery on a broken one. The SRE detail that makes this safe: **rewinding context does not unwind side effects.** An order created before the backtrack does not un-create itself. Every side-effecting tool must be idempotent (keyed by step) or compensated (a paired cancel call), or backtracking is just a fancier way to corrupt data.

**Plan-Repair — amend, don’t regenerate.** For contract failures and shallow model errors, the cheapest correct recovery is a surgical amendment of the *remaining* plan — fix the argument that failed validation, drop the step that contradicts verified results. Repair is bounded to two or three rounds with a token budget: an unbounded repair loop is the retry spiral in a nicer jacket. The model may only edit **unexecuted** steps (executed history is immutable, keeping the decision log replayable), and repaired steps pass the same schema and policy validation as the original.

**Tool-Switching — capability pluralism.** The same capability — update an account, enqueue a job — usually lives behind several tools: a synchronous REST tool, a queue-backed async variant, a read-only verification view. The loop needs a **fallback registry**: an ordered list of substitutes per tool, maintained by policy, not improvised mid-crisis. When the primary tool’s breaker is open or its retries are exhausted, the executor promotes the next tool in the registry. The model never “decides” to switch — the runtime simply filters the unavailable tool from its toolset. Switching becomes routing with deterministic preferability.

The dispatch loop tying the patterns together:

“`
while plan.has_steps() and run_budget.remaining():
result = executor.call(step.tool, step.args, idempotency_key=step.key)
if result.ok:
checkpoint(step, result); continue
error = classify(result)

if error.transient and step.attempts < max_retries: # backoff retry sleep(backoff(step.attempts, jitter=True)); step.attempts += 1; continue if breaker.is_open(step.tool): # tool-switching step.tool = policy.fallback_for(step.tool) if step.tool is None: escalate(error); break step.attempts = 0; continue if error.contract or error.model: # plan-repair plan, repaired = repair_plan(plan, error, repair_budget=2) if not repaired: # backtrack plan = last_valid_checkpoint().plan ``` The property that makes this self-healing rather than self-harming: **every path terminates** — retries have a max, repairs have a budget, backtracks land on validated state, and exhaustion ends in a structured escalation: a human handoff, never a silent success. ## 4. Circuit Breakers: Preventing Agentic Retry Storms on Downed Upstreams A single agent retrying a downed API is a nuisance; a fleet of them is an outage amplifier. When the upstream goes down, every concurrent run discovers it at once and starts its own retry campaign against a service that cannot recover under the load they generate. No individual agent misbehaves, yet the aggregate is a self-inflicted DDoS with a token bill attached: the **agentic retry storm**. The arithmetic is brutal. An upstream sheds load with 429s and a 30-second `Retry-After`; a disciplined client retries every 30 seconds, while a model improvising its own policy retries in two — fifteen times the request rate. Across a hundred concurrent runs, that is a 1,500× amplification of the exact load the outage was designed to shed. The breaker makes disciplined backoff an *enforced property of the runtime*, not a suggestion. Agentic breakers follow the classic tri-state pattern with specifics bolted on: - **Per-tool and per-upstream.** State is keyed by tool name and underlying endpoint so one failing tool doesn't poison unrelated ones. Open on five consecutive transient errors or a sustained error rate above ~40%. - **Open, half-open, closed.** While open, calls are refused immediately and the **planner never sees the tool** — it is filtered from the toolset, so the model spends zero tokens proposing calls the breaker rejects. After backoff, a half-open breaker admits one probe: success closes it, failure re-opens it longer. - **Fleet-wide, not per-pod.** In-process state means every instance trips its own breaker while the fleet keeps hammering. Share state via Redis or etcd so the platform presents one breaker per tool, and stagger half-open probes with jitter. - **Break on the right class.** Open on *transient* failures only. Contract and model errors will never succeed on retry; tripping on them only masks a broken schema or a drifting model behind "service unavailable." The same logic extends to the model layer: when a provider degrades in one region, break to a secondary model or region. The breaker does not care whether the thing being hammered is an ERP endpoint or an inference API — and because it enforces limits the model cannot see past, a behavioral problem becomes a solved infrastructure problem. ## 5. Observability: Tracking Recovery Success Rates and "Time to Heal" A self-healing loop that is not instrumented is just a loop with extra steps — you cannot tell whether it is healing errors or converting them into confident wrong answers. Recovery observability adds two metrics to decision logging; both deserve SLOs. **Recovery success rate (RSR)** — the share of erroring steps that resolve to a verified successful outcome without human intervention, broken down by error class and healing pattern. RSR is the loop's health signal: a falling transient-class RSR tracks upstream fragility and miscalibrated retry budgets; a rising model-error RSR means the loop heals model mistakes so efficiently that your eval suite stops noticing the model is drifting. Track per class, per tool, per prompt version — alert on deltas, not absolutes. **Time to heal (TTH)** — wall-clock from first failed attempt on a step to that step completing successfully, or to human handoff. TTH is the direct cost of the uncertainty the loop absorbs and the best single number for comparing agent designs: a retry healed in 400 ms is noise; a model error needing four replans and a backtrack is a design finding. Track p50 and p95 and gate releases on them like latency. Every recovery decision emits an event that makes the healing replayable: ```json { "ts": "2026-09-06T09:14:02.118Z", "trace_id": "7c2e…", "step": 7, "tool": "crm.account.update", "error_class": "transient", "error_code": "UPSTREAM_TIMEOUT", "recovery": { "pattern": "retry_then_switch", "attempts": 2, "fallback_tool": "crm.account.update.async" }, "healed": true, "ttheal_ms": 5400, "cost_usd": 0.031 } ``` Build supporting panels around them — retry counts by error code, repair-budget exhaustion, breaker opens per tool — plus the chart most teams skip: **cost of recovery versus cost of starting over.** When a plan is fourteen steps deep and three are poisoned, healing can cost more than killing the run and restarting fresh. Then write SLOs that hold the loop accountable. The ones we run in production: **≥ 95% of transient errors heal automatically with p50 TTH under five seconds; ≥ 90% of contract errors heal via plan-repair; escalation under 2% of erroring steps; zero silent successes after an exhaustion path.** Alert on RSR falling below target or TTH p95 breaching — never on raw error counts, because a healthy healing loop is supposed to see errors. ## 6. The OnlineCloud Standard: Building Agents That Survive Production Entropy None of this is exotic, and that is the point. A self-healing agent is not a smarter model; it is a runtime that treats the model as the least reliable component and engineers around it. The discipline reduces to a short list — the one we hold every engagement to: **Typed tools with clean error semantics** — every wrapper classifies failures into the envelope before the model sees them. **Idempotency as a precondition** — without an idempotency key, retries and backtracking are off the table. **Recovery policy as data** — retry budgets, fallback registries, breaker thresholds, and repair caps live in versioned configuration, never improvised in a system prompt. **Limits the model cannot override** — breakers, per-run dollar ceilings, and human checkpoints on irreversible actions are enforced in the runtime. **Instrumented from day one** — RSR, TTH, and escalation rate ship with the first vertical slice, not the postmortem. One practice separates evaluated systems from demos: **failure injection.** Part of the eval suite that defines "done" is a chaos pass — kill the upstream mid-run, ratelimit the agent, return a schema-valid but poisoned response, rotate a credential — and the agent must heal within its SLOs. If it cannot survive a simulated outage in week three, it will not survive a real one in month six. That is the difference between an agent that demos well and one your operations team can sleep through: the second has already failed, in a controlled environment, and gotten better at it. Production entropy is not a bug to be eliminated; it is the environment. Upstreams degrade, schemas drift, credentials rotate, and models change their minds. The agents worth running are designed for that world — classifying errors, healing within budget, escalating cleanly when they cannot. At OnlineCloud, we build those agents: US-based senior engineers shipping evaluated, observable agentic systems against Oracle, MS SQL, Postgres, Snowflake, and the CRMs and APIs your teams use every day, typically in 6–10 weeks, with dashboards and on-call owned by the people who built it. If your agents are one 429 away from a thousand-dollar spiral, [contact the OnlineCloud team](https://www.onlinecloud.dev/) and bring the workload you are tired of babysitting — we will show you a loop that heals itself.

# Implementing SpeakableSpecification for Voice AI: Structured Data for AEO

*By the OnlineCloud Engineering Team*


Search isn’t a list of links anymore. It’s a conversation — and increasingly, a spoken one. From Google Assistant reading the morning brief aloud to ChatGPT, Perplexity, and Gemini synthesizing answers in real time, visibility has shifted from ranking on page one to *being the source an engine chooses to read back*. That shift is called **Answer Engine Optimization (AEO)** — and at the layer where web content meets spoken output, no schema.org vocabulary does more heavy lifting than **`SpeakableSpecification`**.

This deep-dive is for technical leads and SEO strategists who own documentation sites, knowledge bases, or content platforms: what Speakable actually is, how to implement it on technical content, how to verify it end to end, and why it’s the clearest early signal of readiness for the conversational-agent era.

—

## 1. Introduction: AEO and the Rise of Voice Search

Let’s be precise about what AEO is. Classic SEO optimizes for the *search engine results page*: title tags, backlinks, meta descriptions, all aimed at earning a click. AEO optimizes for the **answer itself** — the entity, sentence, or passage an engine extracts, attributes, and serves *without* a click. The customer isn’t browsing blue links; they’re asking “how do I reset my MCP server auth token?” into a speaker, a phone, or a chat bar, and an engine is choosing which document to paraphrase back to them.

The numbers have trended this way for years. Late-2010s forecasts predicted roughly half of all searches would be voice-driven by 2020; reality landed closer to a third of mobile queries, but the direction never reversed. Smart speakers passed one-third of US households. Then came the generative inflection point: Google rolled out AI Overviews above the classic results, ChatGPT and Perplexity turned “search” into “answer,” and every assistant vendor began treating your content as material for spoken and written responses.

Here’s the uncomfortable implication: **an engine will not read aloud content that isn’t marked as readable.** If your documentation is the best answer on the internet but you haven’t told engines *which part of the page to speak*, the engine will happily attribute the next-best source instead. That’s the gap `SpeakableSpecification` was built to close.

## 2. What Is SpeakableSpecification? The schema.org Contract for Spoken Content

`SpeakableSpecification` is a schema.org type, championed by Google, that answers one question for text-to-speech (TTS) systems: **which section of this page is safe and appropriate to read aloud?**

When Google announced speakable in 2018, the goal was to give Google Assistant a reliable signal for reading news and answers back to users on smart speakers and phones. Rather than making an assistant guess whether a page’s content stays coherent once stripped of navigation, ads, and cookie banners, the markup tells it explicitly: *start here, stop here, read this.* An assistant doesn’t consume the whole page — it consumes the **speakable region**, a contiguous block of primary content.

### The Anatomy of the Markup

`SpeakableSpecification` is deliberately tiny. It has two meaningful properties, and you supply **one** (Google recommends `cssSelector`):

– **`cssSelector`** — a CSS selector or list of selectors identifying the elements to read aloud. Give sections stable `id` attributes and reference them directly.
– **`xpath`** — an XPath expression pointing at the same kind of content, for teams that prefer XPath.

The `speakable` property hangs off your content type — `NewsArticle`, `TechArticle`, `Article`, `WebPage` — and accepts a single specification or an array, letting you define multiple readable regions per page.

### What Speakable Is Not

– It is **not** a visual rich result. It renders as *audio output* on Assistant surfaces, which is why rich-result testers that only report on visual enhancements often ignore it.
– It is **not** a way to make hidden text readable. Content that isn’t visible, indexable, and substantial won’t be spoken.
– On the **Alexa side**, arbitrary web pages aren’t ingested through speakable markup today; Alexa’s pathways run through skills, Flash Briefings, and content APIs. But schema.org is the shared vocabulary, and teams building for both ecosystems treat speakable as the *content-layer* contract — structuring sections so a skill, an Assistant action, or a future Alexa web-reader can consume the same TTS-ready blocks. Think of Speakable as the **accessibility contract for the AI age**: the discipline that makes content readable to a screen reader also makes it readable to an AI’s voice.

## 3. Technical Implementation: Adding Speakable to Technical Documentation

Documentation is arguably the *best* use case for speakable — even better than news. Tutorials, API guides, and troubleshooting flows are exactly what users ask assistants: “how do I connect this data source?”, “what’s the retry policy?”, “walk me through setup.”

### Step 1: Structure the page for speech

Before a line of JSON-LD, make the page TTS-friendly:

– Wrap the overview in `

`, and each procedure step in `

`, `

`, and so on.
– Keep each speakable unit small — a few sentences to a short paragraph. Assistants truncate long reads, and a five-minute monologue won’t be served anyway.
– Keep code samples inside the speakable region only if an assistant can meaningfully read them; otherwise exclude the code block and let the surrounding explanation carry the answer.

### Step 2: Inject the JSON-LD

Add a `