AWS AI Infrastructure: Optimizing Costs for Bedrock, EKS, and GPU Clusters

# AWS AI Infrastructure: Optimizing Costs for Bedrock, EKS, and GPU Clusters


*By the OnlineCloud Architecture Team — Cloud Architects & FinOps Leads*

—

## 1. Introduction: The Hidden Costs of AI Scaling

Every AI project starts the same way: a promising proof of concept, a small GPU instance, and a token bill that fits on a line item. Then production happens. Traffic grows, fine-tuning jobs queue up, and suddenly the AWS bill is doing things nobody on the team can explain. The model didn’t get worse — the infrastructure economics did.

The uncomfortable truth is that AI workloads break the cost models we spent a decade perfecting for conventional cloud applications. Traditional compute is predictable: you scale EC2 instances, you watch utilization, you rightsize. AI infrastructure is different. Tokens have no physical shape. GPU clusters idle at thousands of dollars per hour while engineers debug data loaders. A single unmonitored Bedrock integration can quietly burn more in a month than an entire microservices estate.

According to multiple enterprise surveys, infrastructure now accounts for the largest share of AI program spend, and most teams admit they don’t have visibility into where that money goes. The gap between “we’re using AI” and “we know what AI costs us” is where budgets die.

This post walks through the four decisions that determine most of your AI infrastructure spend — Bedrock capacity and token management, GPU cluster design on EKS, ongoing FinOps monitoring, and the Bedrock-versus-self-hosted ROI calculation — and closes with a real tuning engagement that cut admin overhead by 67%.

## 2. AWS Bedrock: Capacity Planning and Token-Based Cost Management

Bedrock is the easiest AI service on AWS to adopt and the easiest to overspend on. Because there is no infrastructure to provision, teams treat it like an API — and stop thinking about capacity entirely. That is a mistake, because Bedrock’s cost structure has two very different regimes: on-demand and provisioned throughput.

**On-demand is a metering problem.** You pay per token (input and output), and per-model pricing varies by an order of magnitude. Claude and GPT-class frontier models cost several times more per token than compact or open-weights models served through Bedrock. The first FinOps discipline is therefore *routing*: classify every workload by its accuracy ceiling and route to the cheapest model class that satisfies it. Summarization doesn’t need frontier reasoning. A routing layer between your application and Bedrock — keyed by task type, payload size, and latency budget — routinely cuts token spend 40–60% before any other optimization is applied.

**Token economics favor aggressive prompt hygiene.** Output tokens cost more than input tokens on most Bedrock models, but every token counts when you serve millions of requests. Measure three numbers per request: prompt tokens, completion tokens, and cache hits. Prompt caching on supported models can cut input-token cost by up to 90% for repeated prefixes like system prompts, tool schemas, and retrieved context. Teams that ignore caching pay full price for the same preamble on every single call.

**Capacity planning means knowing your ceiling.** On-demand Bedrock has account-level and region-level rate limits that are invisible until you hit them. When your marketing campaign or product launch generates a traffic spike, on-demand throttling becomes an availability incident, and the emergency response — panic-provisioning provisioned throughput — is the most expensive possible procurement. Plan capacity the way you plan for a holiday sale: model peak requests per second, estimate token throughput per minute, and reserve provisioned throughput *before* the spike, not during it. Provisioned throughput converts unpredictable token metering into a predictable hourly commitment; it is worth it when you have sustained baseline traffic, and it is a trap when you buy it to cover a one-day surge.

Finally, instrument Bedrock like any other production dependency: log model ID, token counts, latency, and cost per request to CloudWatch or your observability stack. You cannot manage what you do not measure, and Bedrock’s own console shows aggregates, not the per-feature, per-customer cost attribution you need.

## 3. GPU Clusters on EKS: Scaling H100/A100 Clusters for Fine-Tuning

When prompting hits a ceiling, fine-tuning takes over — and fine-tuning is where cloud bills go to become line items of their own. A single 8×H100 node (the p5.48xlarge class) runs at tens of dollars per hour on-demand; an A100-based p4d.24xlarge runs about half that. Run a multi-node training job for a week and you are spending the equivalent of a senior engineer’s monthly salary on one experiment. The discipline that keeps this sane is ruthless separation of *training infrastructure* from *serving infrastructure* — and treating both as ephemeral.

**Right-size the GPU, not the ego.** Most mid-market fine-tuning is LoRA or QLoRA on a single node: parameter-efficient methods that update a fraction of the weights and fit comfortably on 1–2 A100s or even a single H100. A 7B-parameter LoRA run does not need an eight-GPU cluster. Multi-node training with tensor and pipeline parallelism only pays off at the 30B+ parameter scale or with very large datasets — and even then, profiling data loaders, `torch.compile`, and FlashAttention first routinely doubles throughput before you buy the second node.

**EKS is the right control plane if you make it elastic.** The Kubernetes-native pattern that works is a dedicated GPU node group or Karpenter provisioner with two properties: GPU-specific instance types (g5, p4d, p5 families) and aggressive consolidation. Use Karpenter with `ttlSecondsAfterEmpty` or node-group scale-in policies so that idle GPU nodes are terminated in minutes, not days. The most common failure we see is a cluster that auto-scaled *up* perfectly and never scaled *down* — a “parked” H100 running at 0% utilization, billing at 100%.

**Spot and reserved capacity have precise jobs.** Spot instances for GPU are risky for training — an interruption mid-epoch wastes more than the discount saves — but excellent for hyperparameter sweeps, data preprocessing, evaluation runs, and CI GPU tests that can restart. On-demand or Savings Plans cover steady-state training. One-year Savings Plans on p4d/p5 families typically reduce GPU cost 20–40% versus on-demand, and they pair cleanly with a commitment to *not* running GPUs 24/7. If your utilization is 40%, the cheapest GPU is the one that is off.

**Job scheduling is a cost control.** Adopt a fair-scheduling queue (Kueue, Volcano, or plain priority classes) so that training jobs compete for a fixed pool of GPUs instead of each team silently provisioning its own. Queueing changes the conversation from “can we afford another node?” to “which job deserves the node we already have” — and it eliminates the duplicate-experiment problem where three engineers re-run the same baseline because nobody can see the cluster is busy.

## 4. FinOps Strategies: Monitoring Drift and Optimizing Inference Latency

FinOps for AI is not a monthly report; it is a feedback loop. Cost *drift* — the slow, compounding change in spend that no single alert catches — is the real enemy. It comes from everywhere: a teammate bumping `max_tokens`, a retraining job left on a cron schedule, a new model version that is 20% more expensive per token, a GPU node group whose minimum size crept from 1 to 4.

**Monitor drift, not just spend.** Set up a weekly cost-per-request trend per model, per feature, and per environment. Cost Explorer and CloudWatch are necessary but insufficient on their own — most drift hides in unit economics that shift by pennies per request while volume doubles. Budget alerts at 80% are too late when your unit cost has quietly risen 30%. Treat “cost per successful inference” as a first-class SLO, tracked with the same rigor as p95 latency, and review it in the same weekly meeting as accuracy.

**Inference latency and inference cost are the same number.** For self-hosted models, every optimization that cuts latency also cuts the GPU-seconds per request — and therefore the bill. Work the stack in this order:

1. **Batching.** Continuous batching (as implemented by vLLM, TensorRT-LLM, and TGI) can multiply throughput 5–10× over naive single-request serving by filling the “gaps” between tokens with other requests’ work.
2. **Quantization.** FP16 → FP8 on H100s (or INT8/INT4 with AWQ/GPTQ) shrinks memory bandwidth pressure and can double throughput on memory-bound decode with modest quality trade-offs — always validated against your eval set first.
3. **Speculative decoding.** A small draft model proposes tokens while the large model verifies them, cutting latency per request substantially on autoregressive generation.
4. **Model routing and cascades.** Try a small model first; escalate to a larger model only when confidence is low. This “cascade” pattern is the serving-side twin of the Bedrock routing discussed earlier and delivers comparable savings on self-hosted stacks.

**Autoscale on the right signal.** For GPU inference, the signal that matters is queue depth, not CPU utilization. Scale pods on requests-in-flight or token throughput via KEDA; a GPU at 30% CUDA utilization with a growing queue is an availability incident, while a GPU at 90% utilization with an empty queue is wasted money. And always set an absolute scale ceiling — an autoscaler that can add p4d nodes without bound is a billing incident waiting for a traffic spike to trigger it.

## 5. Bedrock vs. Self-Hosted: The ROI Calculation for Mid-Market SaaS

For a mid-market SaaS company, the managed-versus-self-hosted decision is the highest-leverage infrastructure choice you will make, and the industry default — “Bedrock for everything” or “GPU cluster for everything” — is almost always wrong. The correct answer is usually a hybrid, and the way to find it is a unit-economics calculation that any team can run in an afternoon.

**The managed cost per request (Bedrock):** tokens per request × blended token price, plus provisioned-throughput hourly costs amortized over expected volume. There is no engineering overhead, no GPU idle time, no VPC security review for model weights — it is a pure variable cost with near-zero fixed cost and near-zero operational risk. For bursty, unpredictable, or low-volume traffic, managed inference is almost impossible to beat, because you pay only for what you use.

**The self-hosted cost per request:** amortized GPU infrastructure (instance or Savings Plan cost, divided by achieved utilization), plus the engineering time to operate the stack — serving framework upgrades, autoscaling, observability, security patching, and the on-call rotation when an inference node dies at 2 a.m. Most mid-market teams understate this second term by ignoring engineering hours entirely.

**The crossover formula:** self-hosted wins when *(monthly GPU cost + operating engineer time) / requests per month* drops below the managed per-request price *at your utilization level*. Practically, for most mid-market SaaS workloads, that crossover sits somewhere in the range of tens of millions of tokens per day at sustained utilization — below that, managed inference wins on total cost of ownership; above it, self-hosting a quantized open-weights model can cut per-request cost by 3–10×.

But cost is not the only variable. Self-hosting wins decisively on **data residency and privacy** — weights and inference never leave your VPC, which matters for regulated workloads like healthcare and finance. Managed Bedrock wins on **pace**: you can ship a new model capability in days without provisioning anything. And there is a third option teams forget: *both*. Keep Bedrock as the burst and experimentation tier, self-host the steady-state workload, and route between them by cost and latency. In our experience this hybrid is the least common configuration and the most frequently the right one.

## 6. Case Study: Reducing Admin Time by 67% Through Infrastructure Tuning

A healthcare services client — a mid-market organization running clinical workflow automation on a mix of Bedrock models and a small EKS-hosted inference cluster — came to OnlineCloud with a symptom that will sound familiar: “the AI works, but it’s consuming our team.” Their platform engineers were spending most of their week on firefighting: throttling incidents at peak hours, GPU nodes that had not scaled down in weeks, cost reports nobody trusted, and a manual process for evaluating whether each new workload belonged on Bedrock or on their own cluster.

We ran a six-week infrastructure tuning engagement across the four disciplines in this post:

– **Bedrock capacity & token management.** We added a routing layer that classified each request by task and routed to the appropriate model tier, enabled prompt caching for the large shared system prompts, and right-sized their provisioned-throughput commitments to actual baseline traffic. Token spend dropped by over half within the first two weeks.
– **GPU cluster elasticity on EKS.** We rebuilt the GPU node group around Karpenter with strict scale-in policies and moved their fine-tuning jobs onto a fair-share queue. GPU utilization rose from roughly 30% to above 70%, and the “parked node” billing category disappeared.
– **FinOps instrumentation.** We stood up per-request cost telemetry, cost-per-inference dashboards, and a weekly unit-economics review — so cost drift is now caught in days, not quarters.
– **Bedrock vs. self-hosted rebalancing.** We ran the crossover calculation workload-by-workload, moved steady-state inference to the self-hosted cluster, and reserved Bedrock for burst and new-model experiments.

The measurable outcome: **administrative and infrastructure-management time fell by 67%.** Platform engineers stopped firefighting and returned to building. The same period saw inference cost per request decline by more than half, throttling incidents drop to zero, and the finance team get a cost forecast they could actually plan against. The architecture that was “working” was quietly consuming the team; the tuned architecture gave the team back.

That is the pattern behind every successful AI infrastructure engagement we run: **capacity planned before it is needed, GPU spend treated as ephemeral, unit economics monitored weekly, and managed-versus-self-hosted decided by arithmetic, not inertia.**

—

## Get a Second Pair of Eyes on Your AWS AI Bill

If your AI infrastructure costs are climbing faster than your feature velocity — or you simply want to know whether Bedrock, EKS, and your GPU strategy are wired for the cheapest path to production — talk to an OnlineCloud architect. We are a US-based engineering studio specializing in AWS for AI workloads: GPU capacity planning, Bedrock-versus-self-hosted decisions, VPC isolation for regulated data, and the FinOps work that keeps your monthly bill honest. No decks, no offshore handoffs — evaluated, observable systems shipped to production in 6–10 weeks.

**Contact the OnlineCloud architects today** — [onlinecloud.dev](https://www.onlinecloud.dev/) · [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) · Free initial consultation, response within 24 hours.