AI Agent Evaluation: The Metrics that Define ‘Done’ in Enterprise Production

# AI Agent Evaluation: The Metrics that Define “Done” in Enterprise Production


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. The Evaluation Crisis: Why Generic Benchmarks Fail

Every engagement starts the same way: a head of engineering tells us their pilot agent resolved 90% of test tickets in the demo. Six weeks later, the same agent is closed out of the steering committee because it hallucinated a refund policy in front of a customer, or turned a two-second lookup into a twelve-dollar crawl through every tool we gave it.

The pilot was never bad. The problem is that **”works great” was measured against the wrong yardstick**.

Generic public benchmarks — MMLU, HumanEval, GSM8K, and their agentic descendants — are the wrong instruments. They measure latent capability on curated, self-contained tasks, but enterprise agents do not take exams. They compose tools against live database schemas, navigate auth boundaries, recover from malformed APIs, and decide when to stop and ask a human. A benchmark that cannot see your order schema, your refund policy, or your row-level access rules cannot tell you whether the agent touching them is safe to deploy.

There is a second failure: benchmarks measure outputs, but agent behavior is a **process**. Two agents can produce the same correct answer through radically different paths — three well-scoped tool calls versus thirty brute-force retries. An accuracy number cannot tell them apart, or tell you which one you trust with production data.

The industry’s answer is not a better public benchmark. It is a **custom evaluation suite that encodes your workflows, your policies, and your definition of done** — checked into the repo with the agent code, run on every change, and taken as seriously as your unit tests. This post covers the datasets, metrics, and operating rhythm behind that suite.

—

## 2. Building Labeled Datasets for Custom Workflows

You cannot evaluate what you cannot describe, and you cannot describe enterprise workflows with multiple-choice questions. A production eval suite starts with a **labeled dataset drawn from the workflows the agent will actually run** — not paraphrases of them, the workflows themselves.

The gold standard is **logged real traffic**. Your CRM, ticketing system, or order platform already holds thousands of historical interactions: customers asking where orders are, nurses checking protocols, analysts pulling forecasts. Each is a natural task input. The work is labeling the expected outcome, and the discipline is defining what “correct” means per task *before* the model answers.

For a retrieval-augmented order-status agent, a task record might look like this:

– **Task**: “Where is my order #20481 and why is it late?”
– **Ground truth**: order status, carrier, exception reason, and the exact fields the answer must cite
– **Pass criteria**: status and reason present and accurate, no fabricated ETA
– **Trap**: the order belongs to a different account than the caller — correct behavior is to refuse and escalate

That last field is where enterprise evals earn their keep. Public benchmarks measure whether a model *can* answer; production evals measure whether it *should*. A meaningful eval set is built from a taxonomy of real failure modes: ambiguous inputs, policy edge cases, multi-entity requests, out-of-scope asks, and the injection attempts your security team will run against you in week three whether you ask them to or not.

On scale: start with **150–400 labeled tasks per core workflow**, not 10,000. Evals are expensive to label and more expensive to get wrong; the first-pass goal is taxonomy coverage, not statistical volume. Human-in-the-loop corrections (Section 5) grow the set continuously — it is a living asset that compounds, not a one-time project.

Label quality is the silent killer. Without a written rubric, two senior engineers will disagree on 5–10% of edge cases, and your eval quietly measures labeler variance instead of agent quality. Write the rubric, adjudicate disagreements, and store every label with a confidence. A label needing a three-paragraph justification is a taxonomy gap — fix the taxonomy, not the label.

One more non-negotiable: **the eval set lives in the repo, versioned with the code.** An eval suite you cannot run in CI is a slide deck. When the agent changes, the suite runs; when the suite changes, it ships with a rationale. That is the difference between evaluation as an artifact and evaluation as a practice.

—

## 3. Quantifying Accuracy vs. Latency vs. Cost

Once you have a labeled set, resist the temptation to collapse everything into one number. Production agents live at the intersection of three constraints, and you optimize one only at the expense of the others:

– **Accuracy**: the fraction of eval tasks whose outcome meets the pass criteria
– **Latency**: end-to-end time from request to final answer, tool round-trips included
– **Cost**: dollars per resolved request — tokens, tool calls, and human review

Report a **three-axis scorecard per workflow**, not a blended metric. Every workflow we ship gets a row like: `Accuracy 94.2% | Latency p50 3.1s / p95 8.7s | $0.042/request`. The blended “quality score” that marketing departments love obscures exactly the trade-offs engineering decisions involve, so we refuse to produce one.

The trade-offs cut both ways. **Retrieval** is the classic lever: a wider context or an extra retrieval pass buys accuracy on ambiguous queries while lifting latency and token cost. **Model class** is another: a frontier API may buy two accuracy points at 4x the latency and 20x the marginal cost of a fine-tuned small model — which often wins on the metric that matters for tier-1 deflection, cost per resolved ticket.

That is why we report **cost per resolved request**, not cost per token. Tokens are an engineering unit; dollars per resolved request is a business unit — the number your CFO asks about in month two. Latency gets the same treatment: report the user-visible path. An agent that answers in 400ms but makes the customer wait through a fake “thinking” animation has saved no one time.

Two practices keep the scorecard honest. **Regression-gate every change**: nothing that drops accuracy below the prior release, or lifts p95 past the SLA, ships to the canary. And track the axes jointly over time, not as snapshots. When accuracy stalls at 93% while cost per request keeps climbing, the agent is spending more to guess instead of being taught — the next investment belongs in the dataset or the tools, not a bigger model.

—

## 4. Trajectory Scoring: Measuring the Plan, Not Just the Output

Here is the failure generic benchmarks cannot even see: an agent that reaches the right answer through the wrong journey — tools called in the wrong order, idempotent writes retried, data probed that should never have been touched — before stumbling onto the correct output by luck. Output-only evaluation scores it 100% and ships a liability.

**Trajectory scoring** evaluates the *path*: the sequence of observations, tool calls, and intermediate decisions between request and final answer. For enterprise agents, we score three dimensions:

1. **Tool-call validity** — Did calls match their schemas? Did the agent retry calls that should have succeeded, and respect boundaries — read-only tools for untrusted content, human approval before irreversible actions?
2. **Plan fidelity** — Did the executed steps match the stated plan, including mandated checks such as verifying caller identity before revealing order details?
3. **Efficiency** — Tool calls versus the expert minimum, and how much consumed context was irrelevant to the outcome.

Scoring ranges from cheap heuristics to model-based judges. On the cheap end: **deterministic assertions** on the trace — “no tool called twice with identical arguments,” “refund never precedes verify-identity,” “calls stay within 3x of the reference trajectory.” These compile to pass/fail checks that run in CI on every eval task and catch most process regressions for near-zero cost.

On the expensive end, **a judge model or small process reward model** scores each step against the rubric. Treat it as a sampling layer, not a replacement: run deterministic checks on 100% of tasks, judge-based scoring on a stratified sample — especially where output accuracy and trajectory quality diverge.

Invest because **trajectory quality predicts generalization in a way output accuracy does not**. An agent that solved 200 training tasks through brute force collapses when the distribution shifts; one that solved them through minimal, policy-compliant paths holds up. We have watched output accuracy sit flat at 95% while trajectory scores climbed from 60 to 90 — and it was the trajectory score that predicted which agents survived the production canary. Score the plan; the outputs take care of themselves.

—

## 5. Human-in-the-Loop as a Data Source for Evals

The most underused evaluation asset in most organizations is the humans already supervising their agents. Every correction, approval, or override is a free labeled data point — a statement about what the agent should have done. Teams that treat human review as a cost center to be eliminated are discarding the highest-signal evaluation data they will ever produce.

Design the loop to capture it. When a supervisor edits an agent’s draft, approves a high-impact action, or re-routes a mishandled task, log the original output and the correction as a **preference pair**. Over weeks, these accumulate into a dataset encoding your policies, risk tolerance, and customers — something no vendor benchmark can replicate.

Three mechanics make it work:

1. **Capture at the point of action.** The edit form is the labeling interface. A “why did you change this?” dropdown — wrong data, wrong tone, policy violation — structures corrections without burdening reviewers.
2. **Adjudicate disagreement.** When a human correction contradicts the eval’s ground truth, log it and adjudicate weekly. Rising disagreement usually means the rubric has drifted from policy or real traffic.
3. **Feed corrections back into the eval set.** Adjudicated corrections become new labeled tasks, growing the suite toward the long tail your initial set missed — every production week adds coverage.

The same loop feeds **model improvement**, not just measurement: preference pairs are exactly the data used for supervised fine-tuning and direct preference optimization. The flywheel repeats — deployment produces corrections, corrections become eval tasks and training data, the retrained model resolves more tasks autonomously, and review concentrates on the genuinely hard 5%. Human-in-the-loop is not the consolation prize for imperfect agents; it is the data engine that makes agents better.

—

## 6. The Roadmap: 6–10 Week Delivery Against an Eval Suite

An eval suite is not a phase of the project; it is the spine it is built around. This is the roadmap we run on every agent engagement — the same one we would run with your team.

**Week 1 — Discovery sprint, eval first.** Fixed scope, one week: map the data and workflows the agent will touch, choose the model class against your latency, residency, and cost constraints, and — most importantly — **write the eval that defines “done.”** That means 150–400 labeled tasks drawn from real traffic, the pass-criteria rubric, trajectory assertions, and the three-axis scorecard, all checked into the repo. Week one’s deliverable is not a slide deck; it is a test suite that fails. If you cannot define success measurably in week one, you will argue about it in week nine.

**Weeks 2–4 — Build in vertical slices against the eval.** Ship one usable workflow end-to-end every one to two weeks — real data, real users, wired through row-level access behind your network boundary — each with its own eval tasks and regression gates. We iterate against the eval, not against opinions: a demo scoring 90% on the suite ships; one scoring 70% goes back for another pass.

**Weeks 5–7 — Harden and widen the loop.** Instrument human review as a data source, promote the first production corrections into the eval set, and stress the agent on the adversarial taxonomy — injection attempts, out-of-scope requests, ambiguous identity. Audit paths go in — every tool invocation logged with actor, outcome, and dollar cost — and the scorecard starts reporting on production-shaped traffic rather than the curated suite.

**Weeks 8–10 — Canary, gate, and operate.** Run a canary on the lowest-risk workflow, gated on eval regression and the thresholds set in week one. Expand only when it holds its numbers, with a rollback path at every step. Then we operate — dashboards the business reads, retraining on accumulating corrections, drift monitoring on inputs and trajectory quality, on-call owned by the engineers who built it — and the system sharpens month over month as the eval set grows with it.

The non-negotiable outcome is not an agent that demos well. It is one whose **accuracy, latency, cost, and trajectory quality are known quantities, measured continuously, and improving on a schedule you can defend to a CFO.** That is what “done” means.

—

## Conclusion: Evaluation Is the Product

The teams winning with agents are not the ones with the cleverest prompts or the largest models. They are the ones who built what benchmarks cannot provide: a labeled suite of their own workflows, a scorecard that separates accuracy from latency from cost, trajectory scoring that catches bad process before it becomes bad production behavior, and a human loop that turns every correction into a better agent.

That work is unglamorous — which is why most vendors skip it, and why agents built on generic benchmarks fail in production while agents built against an eval suite hold up. The eval suite is not a testing artifact; it is the specification of the system, written in the only language that cannot lie: the language of measured outcomes.

If you are ready to stop demoing agents and start shipping ones you can measure — if you want an eval suite that defines “done” before a line of agent code is written — **talk to the OnlineCloud architects**. We are a US-based Agentic AI & MCP engineering studio; the architect who scopes your project is the engineer who writes the code. No decks, no offshore handoffs, no quarters-long timelines — evaluated, observable systems in production within 6–10 weeks.

**Contact the OnlineCloud architects** at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/) to start the conversation.