Optimizing GPU Utilization for Multi-Agent Inference on EKS: Fractional H100s

# Optimizing GPU Utilization for Multi-Agent Inference on EKS: Fractional H100s


**By the OnlineCloud Infrastructure Team**

Every platform team that has shipped a multi-agent AI system on Kubernetes eventually hits the same wall. The agents work beautifully in staging, the demo lands, and then the finance team pulls the GPU bill. A cluster of dedicated `p4d`/`p5` nodes, each carved out for a single agent workload, quietly burns five figures a month while the GPUs sit at 12% utilization. The infrastructure looks healthy on a dashboard — no throttling, no OOM kills, comfortable headroom — but economically it is a disaster.

This is the **idle-time tax**: the silent cost of over-provisioning accelerators because we treat GPUs as indivisible, single-tenant resources. This deep dive walks through how we break that assumption on Amazon EKS using NVIDIA MIG, CUDA time-slicing, Karpenter, and KEDA — and how we cut per-agent inference cost by 60% without sacrificing p99 latency.

—

## 1. Introduction: The Idle-Time Tax of Dedicated GPU Nodes

Consider a typical agent topology. You are running an orchestration agent, a retrieval agent, a summarization agent, a code-execution agent, and a reflection/critic agent. Each is a separate Deployment, each pinned to its own node via `nodeSelector: nvidia.com/gpu.product=NVIDIA-H100-80GB`. That pattern feels safe: perfect isolation, predictable performance, no cross-tenant interference.

But look at the actual workload. Agent inference is **bursty and short-lived**. A retrieval step might occupy a GPU for 40 milliseconds; a tool-calling loop spends most of its wall-clock time waiting on an external API rather than the accelerator. Measured end-to-end, a busy agent on a dedicated H100 typically achieves 8–18% SM (Streaming Multiprocessor) utilization. The remaining 82–92% is idle silicon you are renting by the hour.

At on-demand `p5.48xlarge` pricing, that idle time is the dominant line item. The math is unforgiving:

– **8× H100 per node**, billed hourly regardless of utilization.
– **1 agent per GPU**, because Kubernetes historically exposes `nvidia.com/gpu` as an integer resource.
– **Result:** you pay for eight full accelerators to serve eight agents that collectively need roughly one.

The idle-time tax compounds across three dimensions. First, **capital waste** — you buy MIG-capable H100s and run them as if they were undivided V100s. Second, **scheduling inefficiency** — the Kubernetes scheduler cannot bin-pack work it cannot see, so nodes stay under-committed. Third, **scaling latency** — adding one more agent means provisioning an entire node, a 3–6 minute cold start on EKS with GPU AMIs.

The fix is to make the GPU a **fractionable, schedulable, elastically-provisioned resource**. That is exactly what fractional GPU architectures deliver.

—

## 2. Fractional GPUs: Implementing NVIDIA MIG and Time-Slicing on Amazon EKS

NVIDIA offers two distinct mechanisms for sharing an accelerator, and choosing between them is the single most consequential architectural decision you will make.

### MIG (Multi-Instance GPU)

MIG physically partitions an A100 or H100 at the hardware level into isolated GPU instances, each with its own dedicated SM slices, L2 cache segments, and memory channels. An H100-80GB can be sliced into up to **7 MIG instances** in profiles like `1g.10gb`, `2g.20gb`, `3g.40gb`, or `7g.80gb`. Isolation is enforced by hardware, so a noisy neighbor in one instance **cannot** steal your memory bandwidth — which makes MIG the right primitive for latency-sensitive, multi-tenant inference.

On EKS, enabling MIG requires a coordinated setup:

1. **Instance type selection.** MIG works on A100 (`p4d`, `p4de`) and H100 (`p5`). Confirm the node’s GPU SKU supports the profile you want; not every profile is available on every card.
2. **Node configuration.** Install the NVIDIA GPU Operator with the MIG manager enabled. The MIG manager reads a labeled config and applies partitions at boot.
3. **Device exposure.** Define a MIG strategy of `mixed` in the Operator’s `ClusterPolicy`, then advertise profiles as extended resources such as `nvidia.com/mig-3g.40gb: 2`.
4. **Scheduling.** Pods request exact MIG profiles by name. The scheduler now bin-packs distinct slices on one physical card.

### Time-Slicing

Time-slicing takes the opposite approach. Instead of partitioning hardware, the CUDA context scheduler interleaves kernel execution from multiple processes on the same full GPU, using a `replicas` count in the device plugin config:

“`yaml
# time-slicing-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: time-slicing-config
namespace: gpu-operator
data:
nvidia-t4: |-
version: v1
flags:
migStrategy: none
sharing:
timeSlicing:
resources:
– name: nvidia.com/gpu
replicas: 16
“`

This advertises 16 logical GPUs per physical card. Time-slicing has **no memory isolation** — all workloads share the same 80GB framebuffer — but it offers the highest density and the lowest setup friction. It is ideal for bursty, latency-tolerant agents where statistical multiplexing wins.

### The Hybrid Strategy

In practice, we run both. **MIG for latency-critical orchestrator and tool-calling agents**, where tail latency is a contractual SLA. **Time-slicing for batch summarization, embedding refresh, and background reflection agents**, where throughput per dollar matters more than any single request’s tail. A single H100 becomes a mixed-occupancy node: two `3g.40gb` MIG instances plus a time-sliced remainder for background work. This is the density foundation everything else builds on.

—

## 3. Resource Density: Architecting for 100+ Concurrent Agents per A100/H100 GPU

The headline number — **100+ concurrent agents on a single H100** — sounds aggressive until you decompose what an “agent” actually consumes.

A modern agent runtime is a thin orchestration loop around an inference call. Most agents are **not** continuously generating tokens; they alternate between LLM calls, retrieval, tool execution, and waiting. We model this with **concurrency headroom**:

– **Peak simultaneous inference:** typically 15–25% of registered agents.
– **Average tokens in flight:** dominated by KV-cache, not weights.
– **Idle fraction:** the variable we exploit.

Layer the technology stack to hit density:

1. **A serving engine built for batching.** vLLM with PagedAttention and continuous batching is non-negotiable. It raises effective throughput 5–10× over naive HuggingFace `generate()` by filling every decode step with work from many requests. On H100, FP8 weights plus PagedAttention let a 7B–13B model serve dozens of concurrent sequences within a single MIG slice’s memory budget.
2. **KV-cache budgeting.** Memory — not compute — is the binding constraint at high concurrency. Enable KV-cache quantization (FP8/INT8) and cap `max_model_len` per agent class. Trimming context from 32K to 8K triples the number of resident sequences for the same framebuffer.
3. **Right-sized model routing.** Do not route every agent through a 70B model. A tiered gateway sends simple classification and extraction to a 7B/8B model on the dense time-sliced pool, escalating only hard reasoning to a larger MIG-partitioned model. This alone can double effective agents-per-GPU.
4. **Aggressive autoscaling signals.** Density architectures fail silently when concurrency spikes. We scale on **queue depth and TTFT (time-to-first-token)**, not CPU — because CPU metrics are blind to GPU saturation.

With these levers, a partitioned H100 hosting two latency-critical MIG slices plus a 16-way time-sliced pool comfortably serves **100+ concurrently *resident* agents**, with a realistic peak of 20–30 active decodes. Dedicated-node thinking would have spread those agents across a dozen accelerators.

—

## 4. Scheduling: Using Karpenter and KEDA to Optimize Fractional GPU Workloads

Density is a static property; **elasticity** is what makes it economical. Two controllers do the heavy lifting on EKS.

### Karpenter for Just-in-Time GPU Nodes

Karpenter replaces the Cluster Autoscaler’s node-group model with direct, constraint-based provisioning. For fractional GPU workloads this matters for three reasons:

– **Profile-aware instance flexibility.** A `NodePool` can list multiple instance families and let Karpenter choose whichever satisfies a pending pod’s MIG or time-slice request. If `p4d.24xlarge` capacity is unavailable, Karpenter falls back to `p5.48xlarge` automatically.
– **Consolidation.** Karpenter continuously evaluates whether running pods could be packed onto fewer, cheaper nodes. Empty MIG slices are reclaimed and nodes terminated within seconds — directly attacking the idle-time tax.
– **Bootstrapping correctness.** GPU nodes need the Operator, MIG config, and drivers pre-baked into a launch template or custom AMI. Karpenter’s `EC2NodeClass` applies these as immutable node identity.

“`yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: gpu-fractional
spec:
template:
spec:
requirements:
– key: karpenter.k8s.aws/instance-family
operator: In
values: [“p4d”, “p5”]
– key: karpenter.sh/capacity-type
operator: In
values: [“spot”, “on-demand”]
nodeClassRef:
name: gpu-nodes
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 30s
limits:
nvidia.com/gpu: “32”
“`

### KEDA for Workload-Driven Scaling

Karpenter decides **which nodes exist**; KEDA decides **how many pods exist**. KEDA’s `ScaledObject` can drive replicas from a GPU-aware or application-level metric — a Prometheus query for pending inference queue length, or an SQS/Redis backlog for the agent orchestration queue.

“`yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: agent-inference
spec:
scaleTargetRef:
name: agent-server
minReplicaCount: 4
maxReplicaCount: 120
triggers:
– type: prometheus
metadata:
serverAddress: http://prometheus:9090
query: sum(inference_queue_depth{pool=”critical”})
threshold: “8”
“`

The division of labor is clean: KEDA raises pod count the instant backlog grows, and Karpenter materializes the fractional GPU capacity those pods demand — then consolidates it away when the burst subsides. Combining this with **Spot capacity for the time-sliced background pool** pushes the effective hourly rate down another 60–70%.

—

## 5. Performance: Managing p99 Latency in Shared-Memory Multi-Tenant Inference

Sharing is a latency risk, and p99 — not p50 — is what your SLA measures. Three failure modes dominate fractional GPU deployments, and each has a concrete mitigation.

**Memory-bandwidth contention.** In time-sliced pools, a large-batch tenant can starve neighbors’ memory bandwidth. Mitigation: enforce **MPS with strict priority** or move latency-critical agents onto **MIG**, where bandwidth is hardware-isolated. Never co-locate a latency-critical agent with a heavy batch job on the same time-sliced card without quota enforcement.

**Head-of-line blocking from long requests.** One agent generating a 4,000-token document can inflate TTFT for every queued neighbor. Mitigation: cap `max_tokens` per request class, enable **chunked prefill** and **preemption** in the serving engine, and route long-generation agents to a dedicated pool.

**CUDA context switching overhead.** Time-slicing’s context switches add microseconds-to-milliseconds of jitter. Under high replica counts, this jitter compounds at the tail. Mitigation: size `replicas` conservatively (start at 8, not 32), and watch `gpu_context_switch` metrics as you scale density.

Operationally, we instrument four signals per pool: **TTFT p99**, **inter-token latency p99**, **KV-cache utilization**, and **SM occupancy**. When TTFT p99 breaches budget, we shed load from the shared pool to a MIG-isolated pool rather than over-provisioning the whole cluster. The result is a system where density is **adaptive**: high utilization in steady state, automatic isolation under pressure.

—

## 6. ROI: Reducing Per-Agent Inference Costs by 60% Through Density Tuning

Here is the payback. Take a representative fleet of 200 resident agents on H100 capacity.

**Baseline (dedicated nodes):** one agent per GPU, 25 GPUs across four `p5.48xlarge` nodes, all on-demand. At roughly $98/node-hour for the node, that is **~$9,800 per hour** of cluster spend — and 85% of it is idle.

**Fractional architecture:** the same 200 agents run across two nodes. Latency-critical agents occupy MIG slices; background agents sit on a 16-way time-sliced pool backed by Spot. Two nodes at on-demand plus Spot pricing land near **~$2,800 per hour**. That is a **~70% reduction** in raw cluster cost, and after accounting for modest overhead — additional GPU Operator management, more sophisticated observability, engineering time — the **normalized per-agent inference cost falls by roughly 60%**.

Where the savings come from, ranked:

1. **Multi-tenancy (largest lever):** 8–16 agents per physical GPU instead of 1.
2. **Spot capacity for tolerant pools:** 60–70% off the time-sliced background tier.
3. **Karpenter consolidation:** eliminates idle nodes during off-peak hours instead of running 24/7 headroom.
4. **Model tiering:** cheap models absorb the majority of agent calls.

The secondary benefits are just as valuable: **faster scale-out** (pods, not nodes, absorb traffic spikes), **better blast-radius control** (MIG isolation contains a runaway agent), and **a greener footprint** — the same workload on one-third the silicon.

—

## Conclusion: Density Is an Architecture, Not a Setting

Fractional GPUs are not a checkbox you enable in the device plugin. They are an architectural commitment that spans **hardware partitioning (MIG), software multiplexing (time-slicing), serving-engine efficiency (continuous batching + KV-cache control), and elastic scheduling (Karpenter + KEDA)**. Get all four layers right and the idle-time tax disappears: your H100s stop being expensive single-tenant heaters and become dense, governed, cost-efficient inference factories.

The teams that win the next wave of agent platforms will not be the ones with the biggest GPU budget. They will be the ones who extract the most useful inference from every hour of silicon they rent — and the tooling to do it is available on Amazon EKS today.

—

## Ready to Cut Your GPU Bill by 60%?

Running multi-agent workloads on dedicated GPU nodes is the most expensive habit in cloud AI. **OnlineCloud** designs and operates fractional GPU platforms on EKS — from MIG profile strategy and time-slicing policy to Karpenter consolidation and p99 latency guardrails.

**Talk to our cloud infrastructure architects** and get a free fractional-GPU readiness assessment. We will model your current idle-time tax and show you exactly how much density you are leaving on the table.

👉 **[Book your assessment at onlinecloud.dev](https://www.onlinecloud.dev/)**