Designing a Cost-Aware Router for Model Cascades (Bedrock vs. Self-Hosted)

# Designing a Cost-Aware Router for Model Cascades: Bedrock vs. Self-Hosted

*A FinOps Architect’s guide to paying for reasoning only where reasoning is required.*


—

## 1. Introduction: The Economics of Frontier Reasoning vs. Steady-State Inference

Every LLM workload is really two workloads wearing the same API shape. There is the **reasoning tail** — the 10–20% of requests that genuinely need a frontier model to think: multi-step planning, adversarial code review, unstructured “figure out what I’m asking” prompts. And there is the **steady-state bulk** — the 80–90% of production traffic that is structurally repetitive: extraction from a known schema, classification, summarization of a fixed document type, routing a support ticket, rewriting a paragraph.

The gap is not semantic; it is **pricing architecture**. Frontier providers meter reasoning separately from output: every chain-of-thought token the model “thinks” is billed, putting reasoning-class models at $15–$75 per million output tokens plus a surcharge. Meanwhile, a self-hosted 8B–70B model on amortized GPU capacity delivers $0.10–$0.50 per million tokens — a **10–30× spread** — with sub-second time-to-first-token and data that never leaves your VPC.

The FinOps failure mode is treating these as one pool. Route every request to the frontier model because “it’s the best model” and you convert a 10–20% reasoning need into a 100% reasoning bill: the line item balloons, p95 latency climbs with queueing, and — counterintuitively — quality doesn’t improve. Frontier models are not measurably better at schema-constrained extraction than a tuned 8B model; they are just more expensive at it.

A model **cascade** fixes this. Put a router in front of the model tier, classify each prompt by complexity, and send the bulk to self-hosted inference on EKS while the reasoning tail goes to Bedrock. Done well, a cascade cuts GenAI inference spend by 60–75% at a bounded, measurable accuracy cost — typically under one point on your eval set. This post walks through the router pattern, the decision heuristics that make it trustworthy, the Go architecture we run in production, and a worked ROI model at 50k requests/day.

—

## 2. The Router Pattern: Classifying Prompts by Complexity and Routing by Cost/Latency

A cascade router is an intercepting layer between your application and your model endpoints. Every request arrives at one ingress and is dispatched to one of two tiers:

| Tier | Backend | Cost profile | Latency profile | Best for |
|—|—|—|—|—|
| **Frontier** | Bedrock (Opus/Sonnet class) | $3–$15+ per M tokens | 2–10s+ (reasoning) | Novel, ambiguous, multi-step reasoning |
| **Self-hosted** | EKS + vLLM/TGI (8B–70B) | ~$0.10–$0.50 per M tokens, amortized | 150–800ms | Extraction, classification, known-shape transforms |

The router performs three jobs, in order:

1. **Policy override.** Explicit client intent wins: an `X-Route: frontier` header, a `requires_reasoning: true` field, or a retry-after-failure escalation must always reach the frontier tier. The router is a cost optimizer, not a quality gate that can be silently skipped.
2. **Classification.** A cheap scoring step decides which tier the request belongs to (Section 3). It must cost under ~1% of the request’s downstream price — easy, since it’s a few milliseconds of CPU/GPU on a tiny model, not another LLM call.
3. **Dispatch with failover.** The chosen backend handles the request; on timeout, 5xx, or low-confidence output, the router escalates to the other tier exactly once and records the escalation as a metric.

Two properties make the pattern production-grade rather than a demo:

**Cascading is asymmetric.** Sending a request to the frontier model is always allowed and always safe; serving it from the self-hosted tier is only safe when you can bound the classifier’s error. Hence the design principle: *when in doubt, escalate.* The router’s default on uncertainty is the frontier tier, and savings come from making the *confident* cheap calls fast and correct.

**Cost is a routing dimension, not just a bill.** Tier choice sets both price and latency, so routing is a two-objective optimization with a constraint: per-request accuracy stays above the product’s floor. You are not minimizing cost; you are minimizing cost *subject to an accuracy budget* — and everything downstream (thresholds, heuristics, dashboards) exists to keep that constraint observable and enforceable.

—

## 3. Decision Heuristics: Logprob Variance and Classification

The router’s entire value rests on one question: *can I cheaply predict that the self-hosted model will answer acceptably before I let it answer?* Two families of heuristics answer it, and production routers use both in sequence.

### 3.1 Logprob variance (a cheap “does the model know?” probe)

When a model is confident, its sampled outputs converge; when it is guessing, they diverge. You can exploit this without touching the frontier tier: sample **K = 4 generations at low temperature** from the *self-hosted* model (nearly free at $0.10–$0.30/M) and measure agreement across the samples.

The cleanest signal is **semantic entropy**: cluster the K outputs by normalized semantic similarity (or exact-match on forced-extracted answers) and compute the variance of the model’s token logprobs around the majority answer. Low variance + high agreement → the model has the answer → serve it. High variance → the model is improvising → escalate.

“`text
logprob_variance = E[ (log p(token_i) – mean_logprob)^2 ]
agreement = (K – minority_count) / K

route = self-hosted if logprob_variance < τ_var AND agreement > 0.75
route = frontier otherwise
“`

The probe is **self-calibrating per request** — no labeled data, because it measures the model’s own epistemic state. In practice it catches the classic failure modes: adversarial phrasings, out-of-vocabulary entity names, ambiguous instructions with two plausible readings. Its cost is 3 extra cheap generations (30–90ms of added latency), amortized against a 10–30× price differential.

### 3.2 Learned classification

Probing multiplies cheap inference by K; for the 60–70% of traffic that is *obviously* steady-state — “extract the invoice number,” “classify this ticket,” “summarize this changelog” — you want a microsecond decision instead. That is the **small classifier** run before any generation:

– **Features:** prompt length, reasoning verbs (“prove,” “compare,” “design,” “why”), number of explicit constraints, entity density, structured-output requests, instruction complexity, domain markers.
– **Model:** a fine-tuned DeBERTa-v3-small (or gradient-boosted model over the feature vector) trained on 10–20k labeled exemplars from your traffic mix — single-digit milliseconds on CPU, emitting a score in [0, 1] plus calibrated confidence.
– **Decision:** below a low threshold → confidently cheap → self-hosted, skip the probe. Above a high threshold → reasoning needed → frontier. In the middle (the *uncertainty region*) → run the logprob probe, or default to frontier if latency-bound.

The two-stage design keeps the router **graceful under distribution shift**: when traffic drifts (a new product ships, a new template lands), the classifier’s scores shift and the probe stage absorbs the uncertainty instead of silently misrouting. You detect the drift on the dashboard (Section 6) and retrain.

### 3.3 Calibration and the accuracy budget

None of this works without a **held-out eval set** — a golden set of 1–3k prompts with reference answers, run nightly against both tiers. You tune τ_var and the classifier thresholds on a single axis: *the measured accuracy delta between the cascade and the all-frontier baseline*, held under a committed budget (e.g., Δ ≤ 0.5–1.0 points). Thresholds are not tuned to maximize savings; they are tuned to spend the accuracy budget, and savings is whatever falls out. That inversion of priorities is what keeps a cascade from becoming a silent quality regression users notice three weeks later.

—

## 4. Architecture: A Go Router in Front of Bedrock and EKS

Go is the right substrate here: it is the lingua franca of ingress proxies (the router *is* a proxy), it streams HTTP without the GC pauses that break SSE, and it deploys as a single static binary on EKS beside the inference tier.

“`
┌─────────────────────────────────────────┐
│ Go Cascade Router │
App traffic ───────► │ policy override → classifier → probe │
│ dispatch · retry · escalate · metrics │
└───────────────┬───────────┬─────────────┘
│ │
┌───────────────▼──┐ ┌────▼──────────────┐
│ EKS (self-hosted) │ │ Bedrock (frontier)│
│ vLLM · 8B-70B │ │ Opus/Sonnet class │
└───────────────────┘ └───────────────────┘
“`

The router is a plain `net/http` server whose decision core is deliberately small and pure — unit-testable without network:

“`go
type Route string

const (
RouteSelfHosted Route = “eks”
RouteFrontier Route = “bedrock”
)

type Router struct {
classifier *Classifier // feature-based scorer
prober *ConsistencyProber // logprob variance probe
cfg Config // thresholds, timeouts, retries
}

// Decide returns the tier for a request. It never returns an error
// for an ambiguous request — ambiguity escalates to frontier.
func (r *Router) Decide(ctx context.Context, req *Request) Route {
// 1. Policy override — explicit intent always wins.
if req.Header.Get(“X-Route”) == “frontier” {
return RouteFrontier
}
// 2. Cheap classifier first.
score, conf := r.classifier.Score(req.Messages)
if conf >= r.cfg.Confident && score <= r.cfg.SelfHostedThreshold { return RouteSelfHosted } if conf >= r.cfg.Confident && score >= r.cfg.FrontierThreshold {
return RouteFrontier
}
// 3. Uncertainty band — probe the cheap model’s own confidence.
if variance := r.prober.LogprobVariance(ctx, req, 4); variance < r.cfg.VarianceThreshold { return RouteSelfHosted } return RouteFrontier // when in doubt, escalate } ``` The dispatch layer wraps that decision in the concerns that keep the system alive: - **Streaming.** Both tiers are proxied with `http.Flusher` passthrough. The classifier runs *before* streaming begins, so the added latency is one ~5ms round-trip, not a serialized LLM call. - **Failover.** Every upstream call gets a deadline (frontier: 120s, self-hosted: 30s). On timeout, connection error, or a 429/5xx burst, the router retries once on the *other* tier and stamps `x-cascade-escalated: true` so callers and dashboards both see it. - **Bedrock specifics.** Calls use Bedrock's streaming invoke API with IAM SigV4 signing and `contentBlockDelta` decoding; the EKS tier is a vLLM `OpenAI-compatible /v1/chat/completions` endpoint on an internal service. One shared request/response schema keeps the router thin. - **Deployability.** The router ships as its own Deployment behind the service mesh, scales on CPU (the classifier is the only compute), and exposes `/-/healthz` plus a `/-/routes` endpoint dumping live threshold state for on-call engineers. The highest-leverage operational detail is **making the decision observable on the request itself**: every response carries `x-cascade-route`, `x-cascade-score`, and `x-cascade-escalated`, so application teams can build product behavior on top of routing (forcing frontier for a user segment, surfacing "answered by fast model" badges) without touching router code. --- ## 5. ROI Calculation: Monthly Savings at 50k Requests/Day A worked model with stated assumptions, so you can re-run it with your own rates: **Assumptions** - Traffic: 50,000 requests/day, average 2,000 input tokens and 500 output tokens each. - Frontier (Bedrock, Sonnet-class blended): **$3/M input, $15/M output**. - Self-hosted: 8B model on vLLM, fleet of 6× A10G at **$0.60/GPU-hr effective** (savings plans/spot), all-in for the bulk tier. - Cascade accuracy budget: Δ ≤ 0.01 on the golden eval set. **Baseline — everything to the frontier model:** - Input: 100M tokens/day × $3/M = $300/day · Output: 25M tokens/day × $15/M = $375/day - **API spend ≈ $675/day ≈ $20,250/month** — with no GPU cost at all, because there is no self-hosted tier. **Cascade — 85% confidently self-hosted, 15% frontier (probes and escalations absorbed):** - Frontier tier: 7,500 req/day → 15M input ($45) + 3.75M output ($56) ≈ $101/day ≈ **$3,030/month** - Self-hosted tier: 42,500 req/day on the 6× A10G fleet → 6 × $0.60 × 24h ≈ **$86/day ≈ $2,590/month** (classifier, probes, router CPU included) - **Cascade total ≈ $5,620/month ≈ 72% below baseline → ~$14,630/month saved.** | Scenario | Monthly cost | vs. all-frontier | |---|---|---| | All traffic → frontier | ~$20,250 | — | | Naive 50/50 static split | ~$15,300 (half frontier spend + larger GPU fleet) | −24% | | **Classifier-driven cascade (85/15)** | **~$5,620** | **−72%** | Two findings worth internalizing. First, **a naive static split is a false economy**: 50% of traffic to each tier still burns ~$10k/month on frontier tokens *and* forces GPU over-provisioning for requests the classifier would have kept at home — the savings live in the *routing decision*, not the split ratio. Second, **the GPU fleet is the floor, not the ceiling**: below ~15–20k self-hosted requests/day, fixed fleet cost dominates and a pure-API cascade (small model on Bedrock's cheap tier, no EKS) can win on paper. Model the fixed floor against per-token savings, then decide whether EKS earns its place. Annualized, that is **~$175k/year** at 50k req/day — the router's dev cost (a few engineer-weeks plus the classifier training set) pays back in its first month, and the nightly eval keeps the accuracy budget honest so savings arrive without waiting for user complaints. --- ## 6. Monitoring: Cascade Hit-Rates and Accuracy Trade-offs A cascade you can't observe is a cost leak with a latency SLA. The stack has one north-star metric and three supporting views, all exported as Prometheus metrics from the Go router and rendered in Grafana, with cost data reconciled from CloudWatch/Bedrock usage and EKS node bills. **North star — accuracy delta (the constraint).** Every night a batch job replays the 2k-prompt golden set against (a) the frontier tier and (b) the live cascade, publishing `cascade_accuracy_delta`. The alert is blunt: if the trailing-7d Δ exceeds the 0.01 budget, page the owning team. This is the *only* metric that can veto a threshold change — savings gains are real only while Δ stays inside budget. **View 1 — Cascade hit-rate and routing telemetry.** Per-route counters and histograms labeled by route, model, and escalation flag: - `cascade_hit_rate = self_hosted_requests / total_requests` — the efficiency gauge; watch it per segment (product, prompt template) to catch segments silently degrading. - `cascade_escalation_rate` — how often router failover fired. A sustained rise usually means the self-hosted tier is degrading (slow TTFT, timeouts), *not* that the classifier is wrong. - `router_decision_us{stage="classifier|probe"}` and the *uncertainty-band share* — traffic landing between thresholds; a widening band over weeks means drift and a retraining trigger. **View 2 — Cost and latency per request, by tier.** `cost_per_request{route}` (token meters × contract rates, plus fleet $/hr spread across requests) alongside p50/p95 latency per tier. The money line: `monthly_spend = sum(rate(cost_per_request) * seconds_in_month)` compared against the all-frontier counterfactual — the view the FinOps review actually reads: *did the cascade save what the model said it would, this week?* **View 3 — Classifier health and drift.** The distribution of classifier scores per segment, the logprob-variance distribution of probe-stage requests, and per-threshold confusion on the nightly eval (`self-hosted-but-wrong` is the true cost of misrouting). Thresholds are only meaningful relative to the eval set they were calibrated on, so the dashboard pins current τ_var and cutoffs next to the eval Δ — one screen answering "is this router still doing its job?" **Operational loops worth wiring from day one:** - **Shadow mode for threshold changes.** Replay 48h of logged traffic through the proposed config before moving a threshold; diff the predicted Δ with zero production risk. - **Escalation as a quality canary.** A sustained rise in escalations *to* frontier is a leading indicator of self-hosted quality drift — investigate before users file bugs. - **Accuracy sampling in production.** Beyond the golden set, sample 0.5–1% of self-hosted responses for LLM-as-judge scoring. Golden sets go stale; production samples don't. --- ## Build the Cost-Aware Tier on OnlineCloud The pattern is proven: classify, probe, escalate, and police the accuracy budget — letting the frontier model do the thinking only where thinking is required. The remaining question is operational: you need managed Bedrock access, an EKS tier that scales your GPU fleet to the *cascade's* actual demand, and cost telemetry spanning both — without stitching five consoles together. That's what OnlineCloud delivers: unified access to frontier models and your own Kubernetes inference clusters, with per-cluster and per-route cost visibility out of the box. **Ship your first cascade this week — start free at [onlinecloud.dev](https://www.onlinecloud.dev/)**, and let the dashboards show you where the 72% went.