# Building an Agentic Ops Dashboard: Visualizing Swarm Health & Latency Waterfalls
*An SRE’s field guide to instrumenting multi-agent systems — from black-box oracles to observable swarms.*
—
## 1. The Black-Box Challenge of Multi-Agent Swarm Orchestration
A single LLM call is easy to reason about. You send a prompt, you get a response, you measure latency and tokens. It’s a function. You can put a timer around it.
A **swarm** is not a function. It’s a distributed system that happens to be made of language models.
The moment you move from one agent to a planner-executor-critic topology with shared memory and parallel tool calls, the mental model that served you for batch jobs and microservices quietly breaks. You have:
– **Non-deterministic control flow.** The same input can take three different plans on three different runs. There is no fixed DAG to pin a flame graph to.
– **Hidden fan-out.** A single “summarize this ticket” request can fan into 14 tool calls, two retries, and a reflection pass — none of which are visible in your APM.
– **Emergent failure modes.** Agents don’t crash. They *drift*. They loop. They hallucinate a tool argument and silently truncate a result. Your 200 OK is a lie.
– **Cost that scales with reasoning, not traffic.** Your bill is a function of token throughput and loop depth, which correlate with *how hard the model is thinking*, not how many users you have.
Amdahl’s law taught us that you can’t optimize what you can’t see. In agentic systems, the hard part is that the thing you need to see — the *reasoning* — is text, not a span. The naive response is to log everything and grep. That works until you have 40 agents, 30 tools, and 10,000 runs/day. Then you have a pile of JSON and no signal.
This post is the instrumentation playbook we use at OnlineCloud when we ship agentic systems into production. The goal is a dashboard an SRE can actually be on-call against — one that answers “is the swarm healthy, and if not, *which* loop is burning the budget?” in under ten seconds.
—
## 2. Metrics for Agents: Completion Rate, Cost-per-Run, Token Throughput
Forget vanity metrics. “Number of agent invocations” is the equivalent of counting HTTP requests — it tells you traffic, not health. Here are the three that matter, and how to define them so they’re actually actionable.
### Task Completion Rate (TCR)
TCR is the agentic analog of a success rate, but you must define “success” with an **eval**, not an exception handler. A run completes when a verifiable end-state is reached: the ticket is closed, the record is updated, the answer passes a rubric. Three sub-metrics matter:
– **First-pass completion** — succeeded with zero human intervention.
– **Escalation rate** — hit a human checkpoint (which is fine — but should be *intentional*, not a symptom of thrash).
– **Silent failure rate** — returned a confident, wrong answer. This is the metric that keeps SREs awake, because it looks identical to success in your logs.
Track TCR as a rolling window, segmented by agent role and by plan template. A 92% overall TCR that hides a 60% TCR on your “refund executor” is a P1 you won’t find otherwise.
### Cost-per-Run ($/run)
Cost is not one number. Break it into:
“`
cost_per_run = Σ (input_tokens × input_price + output_tokens × output_price)
+ tool_call_costs
+ retry_overhead
“`
Emit `cost_per_run` as a histogram, not a gauge — you care about **p95 and p99 cost**, because agentic cost is fat-tailed. One runaway reflection loop can spend more than a thousand healthy runs. Set a hard **cost ceiling per run** and treat breach as a first-class alert, not a billing footnote.
### Token Throughput
Split it into two directions, because they mean different things:
– **Input tokens/sec** — context pressure. Rising input tokens usually means memory retrieval is over-fetching or conversation history isn’t being trimmed.
– **Output tokens/sec** — reasoning effort. This is your proxy for “how much the model is thinking,” and it directly predicts latency and spend.
Track the **input:output ratio** as a derived gauge. A healthy plan-act-reflect loop sits in a predictable band. When the ratio inverts — output exploding while input stays flat — you’re watching an agent ruminate, and you have a few seconds before latency SLOs start screaming.
Export all three via Prometheus counters and histograms so they compose with your existing alerting and Grafana. **Do not build a bespoke metrics pipeline for agents.** Agents are a new workload, not a new discipline.
—
## 3. Latency Waterfalls: p99 of Reasoning vs. Tool Execution
Here is the single most useful visualization in agentic ops: the **latency waterfall** that separates *reasoning time* from *tool execution time*.
Every agent step is one of two things:
1. **A model call** — reasoning, planning, reflecting. Latency is dominated by token count and model throughput.
2. **A tool call** — an MCP tool hitting a database, an API, a vector store. Latency is dominated by the downstream system.
When you collapse them into one “step duration,” you can’t tell whether your p99 regressed because the model got slower or because Oracle did. The waterfall forces the distinction:
“`
Run #4f2a ─────────────────────────────────────────► t = 8.4s (p99)
plan ████████ 1.2s [model]
tool: crm ░░░░░░░░░░░░░░░░░░ 2.9s [tool] ← p99 offender
reflect ██████ 0.9s [model]
tool: kb ░░░░ 0.4s [tool]
act ████████████ 2.6s [model]
tool: sql ░░░░░░░░░░░ 0.4s [tool]
“`
Two rules make this diagram honest:
– **Instrument each segment separately.** A model span and a tool span must never share a parent measurement that hides their boundary.
– **Attribute the waterfall to a plan template, not just a run.** Aggregate the p99 waterfall across all runs of the same plan. That’s how you learn “the *Refund Dispute* plan’s 95th-percentile tool time is Oracle” — which is a DBA ticket, not a prompt-engineering ticket.
The payoff is routing failures to the right team. In practice, once we separated reasoning and tool latency at OnlineCloud, roughly half of what teams had been calling “the agent is slow” turned out to be a downstream query that had nothing to do with the model at all. The other half was prompt bloat — context creeping up until prefill dominated. The waterfall tells you which half you’re in before you spend a sprint guessing.
—
## 4. Dashboard UI: A Real-Time Swarm Health Monitor in React + Tailwind
The dashboard has one job: make swarm health legible at a glance. We build it with React, Tailwind, and a streaming transport (SSE or WebSocket) that subscribes to the same metric stream your alerting uses.
Three panels, top to bottom:
**1. Swarm health strip** — TCR, $/run p95, and output-tokens/sec as large, color-coded tiles. Red-line thresholds are explicit, not vibes.
**2. Plan waterfall** — a horizontal, per-plan latency waterfall (the diagram above, made interactive). Click a bar to drill into the individual run’s span tree.
**3. Live event log** — a streaming tail of agent steps tagged with `run_id`, `agent_role`, and `step_type`, filterable by failure class.
“`jsx
// SwarmTile.jsx — one health tile, zero chart libraries
export function SwarmTile({ label, value, unit, status }) {
const tone = {
ok: “bg-emerald-500/10 text-emerald-400 ring-emerald-500/30”,
warn: “bg-amber-500/10 text-amber-400 ring-amber-500/30”,
crit: “bg-rose-500/10 text-rose-400 ring-rose-500/30”,
}[status];
return (
{label}
{value}{unit}
);
}
“`
“`jsx
// useSwarmMetrics.js — subscribe to the live stream
import { useEffect, useState } from “react”;
export function useSwarmMetrics(url) {
const [snapshot, setSnapshot] = useState({ runs: [], metrics: {} });
useEffect(() => {
const es = new EventSource(url); // SSE keeps it dependency-free
es.onmessage = (e) => setSnapshot(JSON.parse(e.data));
es.onerror = () => es.close();
return () => es.close();
}, [url]);
return snapshot;
}
“`
Two UX principles keep it on-call friendly. First, **color means status, not decoration** — reserve the red/amber palette exclusively for breached thresholds so a glance is a triage. Second, **every tile drills down to a run**. A dashboard you can’t descend from is a dashboard you’ll learn to ignore. The whole point is that a red TCR tile is two clicks from the exact span that failed.
Keep the client dumb: it renders a server-computed snapshot. Do the aggregation in the metrics layer, not in the browser. Your dashboard should never be the thing that goes down under load.
—
## 5. Observability: OpenTelemetry Traces for Plan-Act-Reflect Loops
Metrics tell you *that* the swarm is unhealthy. Traces tell you *why*. OpenTelemetry is the right substrate because agents are just another distributed workload — and you already have the collector, the backend, and the retention policy.
Model an agentic run as a trace, and each phase as a span:
“`python
from opentelemetry import trace
tracer = trace.get_tracer(“swarm.agent”)
def run_agent(goal, plan):
with tracer.start_as_current_span(“agent.run”) as run:
run.set_attribute(“agent.goal”, goal)
run.set_attribute(“agent.plan_template”, plan.name)
for i, step in enumerate(plan.steps):
with tracer.start_as_current_span(f”step.{step.kind}”) as span:
span.set_attribute(“step.index”, i)
span.set_attribute(“step.kind”, step.kind) # plan | act | reflect
if step.kind == “plan” or step.kind == “reflect”:
usage = call_model(step) # a model span
span.set_attribute(“llm.input_tokens”, usage.input)
span.set_attribute(“llm.output_tokens”, usage.output)
span.set_attribute(“llm.cost_usd”, usage.cost)
with tracer.start_as_current_span(f”tool.{step.tool}”) as tspan:
result = call_tool(step) # an MCP tool span
tspan.set_attribute(“mcp.tool”, step.tool)
tspan.set_attribute(“mcp.status”, result.status)
“`
Four conventions make these traces earn their keep:
– **Spans mirror the loop.** `plan → act → reflect` is a span hierarchy, so a runaway loop is visible as abnormal span *depth*, not just duration.
– **Cost lives on the span.** Attach `llm.cost_usd` to every model span. Now your tracing backend doubles as a cost-explorer: sort the slowest traces by dollars, not milliseconds.
– **Tool spans carry the MCP contract.** Tag `mcp.tool` and `mcp.status` so a failing tool is attributable to a specific typed surface, not “the agent.”
– **Propagate the run ID everywhere.** One `run_id` flows through every span, log line, and metric. When the on-call pager fires, it should carry a run ID that opens straight into the waterfall.
With this in place, the three metric panels from Section 4 become *derived views* over the same trace data. You stop maintaining a parallel observability stack and start using the one you already trust.
—
## 6. The OnlineCloud Protocol: Audited, Observable Agent Delivery in 6–10 Weeks
Everything above is the *internal* architecture of a system you can operate. But an agentic dashboard is only as good as the delivery discipline behind the agents it monitors — and that discipline is exactly where most builds stall.
OnlineCloud is a US-based engineering studio that ships **evaluated, observable agent systems** into production in **6–10 weeks, not quarters**. The protocol is deliberately boring, because boring is what survives an on-call rotation:
– **MCP-native architecture, audited end-to-end.** Tools, prompts, and resources are typed surfaces with clean error semantics and audit logs out of the box — so every tool call your dashboard renders is attributable to a real contract.
– **Eval-first, not demo-first.** A discovery sprint writes the eval that defines “done” *before* code. Your TCR metric has a definition because the eval did.
– **Observability built in, not bolted on.** Cost ceilings, plan/act/reflect traces, and p99 latency waterfalls are shipped as part of the system — the same instrumentation this post describes.
– **Senior engineers only.** The architect who scopes the work is the engineer pushing commits. No offshore handoffs, no decks in place of dashboards.
– **Short loops, honest measurement.** Weekly demos with working code, and shared dashboards that instrument accuracy, latency, and $/request.
The result is an agentic system that behaves like the rest of your infrastructure: measurable, alertable, and safe to put on-call against. You get a swarm you can *see* — and a team that hands you the pager with the runbooks already written.
—
## Ship the dashboard, not the black box
Multi-agent swarms are distributed systems, and the SRE playbook still applies — it just needs three new primitives: **agent-native metrics** (TCR, $/run, token throughput), **latency waterfalls** that separate reasoning from tool time, and **OpenTelemetry traces** that make plan-act-reflect loops debuggable. Build those three and you’ve turned a black box into an observable service.
**[Talk to OnlineCloud →](https://www.onlinecloud.dev/)** — ship an audited, observable agent into production in 6–10 weeks. *Free initial consultation · Response within 24h · contact@onlinecloud.dev*
—
*Written by the OnlineCloud engineering team. OnlineCloud is an agentic AI and MCP engineering studio. We don’t sell decks — we ship systems you can watch.*
Leave a Reply