Building an Agentic Eval Harness for RAG: Recall, Precision, and Grounding

# Building an Agentic Eval Harness for RAG: Recall, Precision, and Grounding


*By the Applied Evaluation team at [OnlineCloud](https://www.onlinecloud.dev/) — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: Why Generic LLM Evals Fail to Score Production RAG Systems

Most RAG systems are evaluated with instruments built for a different organism. A team ships a retrieval-augmented agent, runs it against MMLU or a chatbot leaderboard, sees a healthy-looking number, and concludes the system works. Then it meets production: a user asks a multi-hop question whose answer is split across three documents, the retriever surfaces two of them, the generator confidently invents the third fact, and the response reads perfectly while being wrong. Nothing in the benchmark caught it, because nothing in the benchmark was looking.

The failure is structural, and an evaluation scientist should name it precisely. **Generic LLM evals score a completion; a RAG agent produces a trajectory.** A single generation eval asks “is this string a good answer?” A production RAG system asks a chain of coupled questions: did retrieval surface the evidence, did ranking put it in the context window, did the model actually condition on it, and did the final claim trace back to a source the user can inspect? Collapsing that chain into one accuracy number destroys exactly the signal you need to debug it.

There are four specific breakdowns that generic evals cannot see.

**They ignore retrieval entirely.** A completion benchmark hands the model a self-contained prompt. A RAG system’s dominant failure mode is upstream — the answer was never in the context. If you only grade the output, you cannot tell “the model is weak” from “the retriever is weak,” and those two failures require opposite fixes: a better generator versus a better index, chunking strategy, or embedding model.

**They are single-hop and uncited.** Real enterprise questions traverse schemas, policies, and documents. Eval sets built from clean, self-contained items systematically overestimate performance on the ambiguous, multi-document, multi-turn queries that dominate production traffic.

**They are non-deterministic under a naive comparison.** Sampling temperature, retrieval index churn, and reranker version changes mean two runs can diverge while both being acceptable. A harness that diffs against one golden string reports false regressions and burns engineering trust.

**They score answers, not grounding.** The single most expensive RAG failure is a fluent, unsupported claim — a fabricated refund window, a hallucinated policy clause. Generic accuracy does not penalize it, so it never shows up as a regression until it reaches a customer.

The correct response is not a better public benchmark. It is a **RAG-native eval harness**: a golden set of trajectories, a metric suite that decomposes retrieval from generation, a calibrated judge, and a CI/CD gate that blocks release on grounding regressions. This post is a working account of how we build one.

—

## 2. The Golden Set: Authoring Multi-Modal Test Trajectories for Retrieval Validation

Evaluation is bounded by its data. A generic QA set gives you inputs and answers; a RAG harness needs **trajectories** — the full path from query through retrieval, ranking, conditioning, and citation. Authoring them is the highest-leverage engineering work in the project, and it starts from your own logs, not from a public corpus.

**Source from real traffic.** Mine the queries your users actually send, the documents your index actually holds, and the questions your support team actually answers. Public sets teach the model the distribution of the internet; your eval must teach the harness the distribution of your business. Start with **150–400 trajectories per core workflow** — the goal is taxonomy coverage, not statistical volume — and grow the set as production corrections accumulate.

**Label the trajectory, not just the answer.** Each record should encode the evidence path. A well-formed trajectory carries:

– **Query and intent class** — single-hop lookup, multi-hop synthesis, comparison, temporal (“as of Q2 2025”), or unanswerable/out-of-scope.
– **Relevant document IDs** with graded relevance (2 = must retrieve, 1 = helpful, 0 = distractor) so retrieval metrics have ground truth.
– **Distractor documents** — topically similar but non-answering passages that a naive retriever will pull. These are where precision is won or lost.
– **Required citations** — the spans a correct answer must trace back to. This is the anchor for grounding scores.
– **Pass criteria and traps** — including the *should-not-answer* cases: the policy the system must refuse, the identity mismatch it must catch, the question whose correct answer is “I don’t have that.”

**Make it multi-modal.** Production retrieval is increasingly not text-only. Chunked PDFs with tables and figures, screenshots, voice transcripts, and structured rows from Oracle or MS SQL all enter the same context window. A trajectory therefore carries chunk *types*, not just strings: a scanned invoice, a table region, an audio transcript with speaker turns. Grade extraction of *load-bearing* fields (IDs, totals, dates) separately from surrounding prose — a misread digit in a payment amount and a misread digit in a footnote are not the same error. For every modality boundary, hold out document **templates**, not just documents; template leakage produces systems that ace the benchmark and fail on the first unseen vendor format.

**Write the rubric before the labels.** Without a written pass/fail rubric, two strong engineers disagree on 5–10% of edge cases, and your harness silently measures annotator variance instead of system quality. Adjudicate disagreements, store a confidence with every label, and treat any label that needs a three-paragraph justification as a taxonomy gap to fix rather than a judgment call to record.

Finally, **version the golden set in the same repository as the system.** An eval suite you cannot run in CI is a slide deck. When the retriever changes, the set runs; when the set changes, it ships with a rationale. That discipline is the difference between evaluation as an artifact and evaluation as a practice.

—

## 3. Metrics for RAG: Defining Recall@K, Context Precision, and Factual Grounding

With trajectories in hand, the metric suite must separate the retrieval stage from the generation stage. Blending them produces numbers that move without explaining why.

**Recall@K (did retrieval find the evidence?).** Recall@K is the fraction of trajectories for which at least one ground-truth-relevant chunk appears in the top-K retrieved passages. It is the ceiling on everything downstream: if the evidence is not in context, no generator can be faithful to it. Report it per intent class, because a single aggregate hides the multi-hop failures that matter most. Enterprises typically gate around **Recall@10 ≥ 0.90** for core workflows, with multi-hop classes tracked separately and expected lower until the retriever is strengthened.

**Context Precision.** Recall alone rewards a retriever that dumps everything into the window. **Context Precision@K** measures the proportion of retrieved passages that are actually relevant and correctly ranked — it is precision of the *context*, not of the final text. Use rank-aware variants (mean average precision or normalized discounted cumulative gain) so that pushing the relevant chunk from position 8 to position 2 earns credit. Precision is where cost lives: every irrelevant passage is tokens you pay for and noise the generator must ignore. A system with high recall and low precision is not “safe,” it is expensive and distractible. Target **Context Precision@5 ≥ 0.7** as a working bar, tuned to your context budget.

**Factual Grounding (is every claim supported?).** This is the metric that protects the brand. Grounding scores the final answer against the retrieved evidence, not against the *world*. Decompose each response into atomic claims, then classify each as **supported** (entailed by a cited passage), **partially supported**, or **unsupported/hallucinated**. Report a **Groundedness Rate** = supported claims ÷ total claims, plus a hard **citation-validity gate**: every citation must resolve to a passage that actually entails the claim it is attached to. The gate is binary and non-negotiable — an unsupported claim carrying a real-looking citation is worse than no citation at all, because it manufactures false confidence.

Layer two derivative metrics on top. **Answer Relevance** checks that the response addresses the *asked* question rather than an adjacent one it retrieved well. **Abstention Calibration** checks that the system refuses exactly when the evidence is absent — measure the false-answer rate on unanswerable trajectories as carefully as you measure accuracy on answerable ones. A system that never abstains scores beautifully on answerable sets and catastrophically in production.

The reporting unit is a vector, not a scalar: **(Recall@K, Context Precision@K, Groundedness, Relevance, Abstention)**. A change that lifts groundedness 4 points but drops recall 10 points is a regression the scalar would have hidden — and the vector makes the trade-off explicit to the team.

—

## 4. LLM-as-a-Judge: Implementing a Calibrated Grading Loop for Agent Responses

Faithfulness and relevance are not string-matchable. Human grading does not scale to every CI run. The pragmatic answer is an **LLM-as-a-judge** — with the discipline to keep it honest.

**Use a judge on a narrow, well-specified question.** A judge asked “is this a good answer?” produces noise. A judge asked “does passage P entail claim C? answer yes/no/partial, quote the supporting span” produces a signal you can validate. Decompose the grading task until each judgment is near-binary, then aggregate. Structured output (typed JSON, spans, verdict enums) beats free-text rationales for reliability and for downstream parsing.

**Calibrate before you trust.** A judge is a model, and it has biases you must measure:

– **Agreement with human labels.** Assemble a stratified human-graded sample of 100–200 items and report judge–human agreement (Cohen’s κ or F1 per verdict class). Do not deploy a judge below your agreed threshold on the classes that gate release.
– **Position and verbosity bias.** LLMs favor the first option in a pairwise comparison and favor longer answers. Randomize order, blind the judge to which system produced which output, and test whether length correlates with a “pass” verdict independent of correctness.
– **Self-preference.** A judge tends to reward generations from its own model family. Use a judge that is **not** the generator, or at minimum verify the effect does not shift verdicts.
– **Rubric anchoring.** Few-shot the judge with ratified examples from *your* rubric, including border cases and explicit “partial” examples. Store the rubric in version control alongside the prompt.

**Treat the judge as a component under test.** Track judge cost per eval run and its latency; cache verdicts by content hash so unchanged trajectories are not re-graded. When you upgrade the judge model, re-run the human-agreement calibration — a judge change is a metric change, and it must not silently move your release gates. Where stakes are highest (safety, money, regulated speech), route the judge’s borderline verdicts to a **human adjudication queue** and feed those decisions back into the few-shot set. The loop compounds: every human correction makes the judge sharper, and the golden set grows with it.

—

## 5. Observability: Tracking Eval Metrics in CI/CD Pipelines Before Production Release

An eval harness that runs only on a laptop is a research toy. The point of building it is to make **every change** answerable to the same scorecard before it reaches a user.

**Check the harness in with the code.** Eval trajectories, rubrics, judge prompts, and thresholds live in the repo. A pull request that touches retrieval, chunking, prompting, or the generator model triggers the suite automatically. No gate, no merge.

**Gate on the vector, with explicit tolerances.** CI should fail — or flag for human review — when any gated metric moves beyond its tolerance. Set thresholds in week one, in writing: for example, block on any drop in the citation-validity gate or any groundedness regression greater than two points, and flag recall regressions above three points on core intent classes. Tolerance is not laxity; it is the honest acknowledgment that retrieval and sampling carry variance, so you run the suite with a fixed seed where possible and report confidence intervals rather than a single run.

**Separate the offline gate from the online watch.** CI proves the change is not a regression on the golden set. Production observability proves the golden set still describes reality. Instrument the live system with the same taxonomy: log retrieved doc IDs, ranks, context size, generated claims, citation resolutions, latency waterfall, and per-request cost. Then monitor **drift** on inputs (query distribution, document churn) and on trajectory quality (recall sampled via explicit feedback, citation click-through, thumbs-down clustering). When live recall diverges from offline recall, your eval set is stale — that is a first-class alert, not a footnote.

**Make it legible.** Dashboards should serve two audiences: engineers needing per-stage diagnostics (was it the index, the reranker, the prompt, or the model?), and stakeholders needing the outcome line (accuracy, groundedness, latency, $/request). Version every metric with the model, index, and prompt revision that produced it, so a number can always be traced to a configuration. A metric you cannot attribute is a metric you cannot act on.

The payoff is cultural as much as technical: when the suite is the specification, “done” stops being an opinion and becomes a threshold the team agreed to in week one and defends at every release.

—

## 6. The OnlineCloud Protocol: 6–10 Week Delivery Against a Verified Eval Suite

This is the protocol we run on every RAG engagement, and it is designed so that the eval suite is the spine of delivery rather than a phase tacked onto the end.

**Week 1 — Discovery sprint, eval first.** One week, fixed scope. We map the data, the documents, and the workflows the system will touch; choose the model class against your latency, residency, and cost constraints; and — most importantly — **author the eval that defines “done.”** That means 150–400 trajectories from real traffic, the pass-criteria rubric, the retrieval and grounding metric definitions, and the CI thresholds, all checked into the repo. The week-one deliverable is not a deck; it is a test suite that fails.

**Weeks 2–4 — Build in vertical slices against the eval.** We ship one usable retrieval-and-answer workflow end to end every one to two weeks — real data, real users, row-level access enforced behind your boundary — each with its own trajectory set and regression gate. We iterate against the eval, not against opinions: a slice that clears recall, precision, and grounding thresholds ships; one that does not goes back for another pass.

**Weeks 5–7 — Calibrate the judge and harden the loop.** We stand up the graded loop, report judge–human agreement, route borderline verdicts to human adjudication, and stress the system on the adversarial taxonomy — unanswerable queries, distractor-heavy retrievals, injection attempts, and template-leakage probes. Audit paths go in: every retrieval and tool call logged with actor, sources, outcome, and dollar cost.

**Weeks 8–10 — Canary, gate, and operate.** We canary on the lowest-risk workflow, gated on eval regression and the thresholds set in week one, with a rollback path at every step. Then we hand over dashboards the business can read, the drift monitors, and the retraining cadence — and the eval set keeps compounding as production corrections flow back into it.

The non-negotiable outcome is not a system that demos well. It is one whose **recall, context precision, groundedness, latency, and cost are known quantities, measured continuously, and improving on a schedule you can defend to a CFO.** That is what “verified” means.

—

## Conclusion: The Harness Is the Product

Teams that win with RAG are not the ones with the cleverest prompt or the largest model. They are the ones who built what benchmarks cannot provide: a golden set of their own trajectories, metrics that separate retrieval from generation, a calibrated judge that turns subjective quality into a reproducible number, and a CI/CD gate that refuses to ship an unsupported claim. Evaluation is not the thing you do after the system is built. It is the specification you build it against.

If you are ready to stop demoing RAG and start shipping systems you can measure — with an eval suite that defines “done” before the first index is built — **talk to the OnlineCloud architects**. We are a US-based Agentic AI & MCP engineering studio; the architect who scopes your project is the engineer who writes the code. No decks, no offshore handoffs, no quarters-long timelines — evaluated, observable RAG systems in production within 6–10 weeks.

**[Book a free evaluation consultation →](https://www.onlinecloud.dev/)** or email [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) to start the conversation.

*Free initial consultation · USA-based support · Response within 24h.*