Scaling Inference for 10k+ Concurrent Agents on EKS: HPA, vLLM, and Spot Optimization

# Scaling Inference for 10k+ Concurrent Agents on EKS: HPA, vLLM, and Spot Optimization


Running a fleet of autonomous agents is not the same problem as running a chatbot. A chatbot waits for a human to type. An agent fan-out does not. A single orchestration request can expand into twenty tool calls, a dozen retrieval hops, and a reflection loop that re-prompts the model three times before it decides it is done. Ten thousand concurrent agents is therefore not ten thousand concurrent requests — it is closer to two hundred thousand model invocations per minute, each with different token profiles, different latency budgets, and different preemption risk. This is the resource-density challenge, and it is where most inference platforms quietly fall over.

This deep dive is the architecture we use at [OnlineCloud](https://www.onlinecloud.dev/) when we provision GPU inference for agentic workloads on Amazon EKS. We will cover cluster scaling with Karpenter, inference engines (vLLM and TensorRT-LLM), request-per-token autoscaling with KEDA and HPA, a fault-tolerant spot strategy that cut compute spend by roughly 70%, and the benchmark data that tells you when to trade latency for throughput.

## 1. The Resource-Density Challenge of Large Agent Fleets

An agent fleet has a fundamentally different load shape from a REST API. Three properties dominate capacity planning.

**Burst amplification.** A single plan step can multiply into N tool calls. If each tool call requires an LLM judgment, your steady-state QPS is a poor predictor of peak GPU demand. We typically see a 6×–15× amplification factor between ingress requests and actual inference calls at p99.

**Variable token economics.** An extraction call returns 80 tokens; a summarization call returns 900. Autoscaling on request count alone will either under-provision the long-context pods or waste capacity on the short ones. You need a metric that reflects *work*, not just *arrivals*.

**Cold-start intolerance.** Agents hold state — a conversation, a scratchpad, a partially completed tool chain. When a node dies mid-loop, the agent must be resumable, or the user sees a broken experience.

The density math is unforgiving. An H100 80GB running a 70B model in FP8 with a healthy KV-cache budget can hold on the order of 100–200 concurrent sequences at 4k context. To reach 10k concurrent sessions you are provisioning 60–100 GPU nodes at peak — which is exactly why the spot question is not optional. At on-demand prices that fleet is a five-figure monthly line item; the same fleet on spot is roughly 30% of that.

## 2. Scaling H100/A100 Clusters with EKS and Karpenter

We standardize on EKS with Karpenter for node provisioning. The managed control plane gives us the API surface, IAM integration for IRSA, and cluster autoscaler primitives we do not want to maintain. Karpenter gives us something the legacy Cluster Autoscaler cannot: sub-minute node provisioning driven by actual pod scheduling pressure, and heterogeneous instance selection across GPU families.

The core pattern is **NodePool diversity**. We never pin a workload to a single instance type. Instead we define NodePools with `requirements` that accept a set of compatible GPU SKUs (p5, p4d, p4de, and A100-backed g5 families) and let Karpenter pick the cheapest available capacity that satisfies the pod’s resource request.

“`yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: gpu-inference
spec:
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 5m
limits:
cpu: “2000”
nvidia.com/gpu: “256”
template:
spec:
requirements:
– key: karpenter.sh/capacity-type
operator: In
values: [“spot”, “on-demand”]
– key: node.kubernetes.io/instance-type
operator: In
values: [“p5.48xlarge”, “p4d.24xlarge”, “p4de.24xlarge”, “g5.48xlarge”]
taints:
– key: nvidia.com/gpu
effect: NoSchedule
“`

Three things matter here. First, **weigh pods by real GPU footprint** — a 70B model needs a whole node; a 7B router model can bin-pack. Second, **taint GPU nodes** so only GPU-aware workloads land there; system daemonsets should never consume a full-precision slot. Third, **enable consolidation** with `WhenEmptyOrUnderutilized`. Agent traffic is bursty; without consolidation you pay for idle H100s between waves.

For AWS for AI workloads specifically, we also split NodePools by latency class: an interactive pool with `consolidateAfter: 30m` (cheap to lose, expensive to wait for) and a batch/offline pool with `consolidateAfter: 1m`. Mixing the two in one pool forces you to choose the wrong consolidation policy for half your traffic.

## 3. Inference Engines: vLLM and TensorRT-LLM for Multi-Agent Throughput

The engine choice is the single largest lever on cost-per-token.

**vLLM** is our default for multi-tenant agent serving. PagedAttention eliminates the KV-cache fragmentation that kills naive implementations, and continuous batching means a decoding step is never blocked waiting for the slowest sequence. For agent fleets this matters because a reflection loop and a classification call can share a batch efficiently. Prefix caching is a second-order win: agent system prompts are long and identical across sessions, so we cache the shared prefix and pay prefill once instead of ten thousand times. Typical gain on a 2k-token system prompt: 40–60% reduction in prefill compute.

**TensorRT-LLM** is our choice when a model is fixed and latency is the binding constraint. Compiled kernels, in-flight batching, and FP8/INT4 quantization on H100 give us 20–40% lower time-to-first-token than vLLM for the same model at high concurrency. The trade-off is rigidity: recompilation on model or shape change, longer cold starts, and a heavier image. We run TensorRT-LLM behind a **router** that sends latency-critical traffic (the user-facing first token) to TRT-LLM pods and everything else — tool calls, batch summarization, evaluator passes — to vLLM pods.

“`python
# Router: latency-critical traffic to TRT-LLM, everything else to vLLM
ROUTES = {
“agent.first_token”: “trtllm-pool”,
“agent.tool_call”: “vllm-pool”,
“agent.evaluator”: “vllm-pool”,
“batch.summarize”: “vllm-pool”,
}
“`

Both engines are exposed through a single OpenAI-compatible gateway so agent code never hardcodes an engine. We front them with an L7 proxy that does token-aware load balancing rather than round-robin — routing by estimated prompt+max-tokens so a long-context request does not land on a pod already at its KV-cache ceiling.

For multi-agent throughput, the practical configuration that scaled us to 10k sessions was: vLLM with `–max-num-seqs 256`, chunked prefill enabled, and a **per-pod KV-cache headroom target of 15%**. Headroom is not waste — it is the buffer that absorbs the burst amplification described earlier without triggering preemption.

## 4. Auto-Scaling with KEDA and HPA on Request-per-Token Metrics

Standard HPA scales on CPU. For GPU inference that is close to useless: an H100 at 100% SM utilization and one at 20% may both report the same pod CPU, because the CPU is just feeding the GPU. We scale on **work**, using a custom metric we call *tokens in flight* — the sum of prefill and decode tokens outstanding across a pod’s active sequences.

The pipeline is: engine exposes Prometheus metrics (`vllm:num_requests_running`, `vllm:gpu_cache_usage_perc`, plus a derived token counter) → Prometheus scrapes → **KEDA’s prometheus scaler** or the Prometheus Adapter exposes them to the HPA → HPA drives the deployment.

“`yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-agents
spec:
scaleTargetRef:
name: vllm-deployment
minReplicaCount: 4
maxReplicaCount: 96
cooldownPeriod: 180
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
– type: Percent
value: 100
periodSeconds: 30
scaleDown:
stabilizationWindowSeconds: 600
policies:
– type: Pods
value: 4
periodSeconds: 60
triggers:
– type: prometheus
metadata:
serverAddress: http://prometheus.monitor:9090
query: |
sum(rate(vllm:tokens_in_flight[1m]))
threshold: “180000” # tokens/sec of aggregate capacity per replica
“`

Two behaviors make this work at fleet scale. **Asymmetric scaling windows**: scale up aggressively (no stabilization, 100%/30s) because an under-provisioned fleet drops requests; scale down slowly (600s window, 4 pods/60s) because thrashing a GPU node costs more than keeping it warm. **Threshold expressed in tokens per second per replica**, tuned to the model. A 70B model at 4k context sustains a very different token rate than a 7B router, so we run separate ScaledObjects per model pool rather than one global rule.

We pair this with **predictive pre-warming** for known diurnal patterns — KEDA’s cron trigger scales the floor up before the morning wave — and a queue-depth guardrail: when the KEDA query is unreliable, a fallback trigger on the gateway’s pending-request count prevents a silent undercount.

## 5. Spot Instances: A Fault-Tolerant Agent Loop for ~70% Cost Reduction

Spot is where the economics change. H100 spot capacity in us-east-1 has historically run 60–75% below on-demand, and A100 families similar. On a 60–100 node fleet that is the difference between a viable product and a broken unit economics story. The catch is the two-minute interruption notice.

We built fault tolerance into the **agent loop**, not bolted onto the infrastructure. The principles:

1. **Checkpoint every step.** Agent state — the plan, completed tool results, scratchpad — is persisted to Redis (or DynamoDB for durability) after each step, keyed by session. A preemption costs at most one step of rework.
2. **Make steps idempotent.** Every tool call carries an idempotency key, so a replayed step does not double-charge a payment API or re-issue a ticket.
3. **Drain, do not die.** A DaemonSet watches the instance metadata endpoint for the spot interruption notice and immediately cordons the node and flips the pod’s readiness probe. The gateway stops routing new sessions; in-flight sequences finish their current step and checkpoint.
4. **Spread the risk.** We configure `topologySpreadConstraints` so no single interruption event can take more than a fraction of a model pool, and we keep a **20% on-demand floor** per pool with spot as the burst capacity.

“`yaml
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
topologySpreadConstraints:
– maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels: { app: vllm }
containers:
– name: interruption-watcher
image: onlinecloud/spot-drain:1.4
env:
– name: DRAIN_GRACE_SECONDS
value: “90”
“`

With checkpointing, idempotent tools, and graceful drain in place, preemption became a non-event: our measured p99 session abort rate dropped below 0.05%, which is lower than the abort rate from ordinary pod OOM kills. Combined with Karpenter’s bin-packing and the consolidation policy, the blended compute cost landed at roughly **30% of the pure on-demand baseline — a 70% reduction**.

## 6. Benchmarking: Latency vs. Throughput at 10k Concurrent Sessions

We benchmark every pool before it ships, and we report three numbers, not one: **TTFT p50/p99** (time to first token), **inter-token latency p99** (the metric users feel while streaming), and **tokens/sec/GPU** (the cost metric). Optimizing one in isolation is how teams ship a fast demo and an expensive product.

At 10k concurrent sessions across the mixed fleet, our reference results were:

| Configuration | TTFT p99 | ITL p99 | Tokens/s/GPU | Relative $/1M tok |
|—|—|—|—|—|
| vLLM, no batching, on-demand | 180 ms | 42 ms | 1,900 | 1.00× |
| vLLM + continuous batching | 240 ms | 38 ms | 4,600 | 0.41× |
| vLLM + batching + prefix cache | 210 ms | 38 ms | 6,400 | 0.30× |
| TensorRT-LLM FP8 (latency path) | 140 ms | 19 ms | 3,800 | 0.33× |
| Blended (TRT-LLM + vLLM) on spot | 165 ms | 26 ms | 6,100 | 0.11× |

The headline is the trade curve. Adding continuous batching raises TTFT slightly at p99 — more work queued ahead of you — while cutting cost per token by more than half. Prefix caching claws most of the TTFT back because the shared agent system prompt no longer re-prefills. TensorRT-LLM wins outright on latency but carries a lower throughput ceiling per GPU, which is exactly why the blended, routed architecture beats either engine run alone. Spot does not affect latency at all once drain and checkpointing are correct; it only moves the cost column.

Two saturation signals tell you the fleet is about to degrade: `gpu_cache_usage_perc` climbing past ~85% (KV-cache pressure → preemption storms) and inter-token latency p99 diverging from p50 by more than 3× (batching is starving interactive sequences). We alert on both, and the KEDA threshold above is tuned so the fleet scales before either fires.

## The Takeaway

Ten thousand concurrent agents is a density problem before it is a scaling problem. Answer it with Karpenter’s heterogeneous, consolidating NodePools; serve it with vLLM for throughput and TensorRT-LLM for latency behind a single routed gateway; scale it on tokens-in-flight rather than CPU; and fund it with a spot strategy that treats preemption as an expected event rather than an incident. Done right, the fleet holds p99 latency under 200 ms while paying roughly 11 cents per million tokens instead of a dollar.

—

**Building an agent fleet and watching the GPU bill climb?** OnlineCloud designs and operates the full stack — EKS GPU capacity planning, vLLM/TensorRT-LLM serving, KEDA autoscaling, and the spot economics that make agentic AI affordable at scale. We are a US-based senior engineering studio, and the architect who scopes your workload is the engineer who ships it. [Book a free 30-minute architecture review](https://www.onlinecloud.dev/#contact) and we will map your concurrency targets to a costed cluster design in a single call.