Evaluating Multi-Modal Agents: Benchmarking Vision, Voice, and Tool-Use

# Evaluating Multi-Modal Agents: Benchmarking Vision, Voice, and Tool-Use

*By the Applied AI Research team at [OnlineCloud](https://www.onlinecloud.dev/)*


The last two years of agent engineering got us comfortable with a narrow fiction: that an agent is a language model wrapped in a tool loop. We graded it on text — did it call the right function, did it pass the right arguments, did it stop before burning $40 of tokens. That fiction is now breaking under production load. The systems enterprises actually deploy have to look at a scanned invoice, hear a distressed caller, and decide — in one trajectory — whether the next action is an OCR pass, a retrieval call, or a human handoff.

Multi-modal agents are not “text agents with extra inputs.” They are systems whose *action space itself* is multi-modal. That changes what “correct” means, and it breaks most of the eval harnesses teams inherited from the LLM-only era. This post is a working account of how we think about scoring vision, voice, and cross-modal tool-use — and how we build gold sets that survive contact with a real production P95.

## 1. Introduction: The Complexity of Unified Multi-Modal Agent Trajectories

A classic LLM eval scores a single output against a single reference. A multi-modal agent produces a **trajectory**: an interleaved sequence of perceptions (frames, audio chunks, ticket metadata), internal reasoning steps, and tool invocations whose effects feed back into the next perception. The unit of evaluation is no longer a completion; it is a *policy rollout*.

Three properties make these trajectories hard to grade honestly.

**Non-determinism compounds across modalities.** Sampling temperature, frame sampling rate, VAD segmentation jitter, and tool latency all inject variance. Two runs of the “same” task can diverge at step 3 and end in different but equally valid terminal states. Any eval that compares a single golden transcript will report false failures.

**Perception errors propagate non-linearly.** A 95%-accurate OCR module embedded in a five-step reasoning chain does not yield a 95%-accurate agent. If a misread digit flows into a tool call that triggers a refund, the cost of a small perceptual error is enormous while the *accuracy metric* barely moves. We therefore score **error amplification** — the ratio of terminal-state failure rate to upstream perception error rate — as a first-class metric.

**Success is multi-objective.** A voice agent that resolves a caller’s request but conveys cold affect on a sensitive helpline has failed, even if the transcript-level task completed. We optimize on a Pareto front of task success, latency, safety, and affective quality, not a scalar.

The practical consequence: treat the agent as a stateful system under test, define trajectory-level acceptance criteria, and instrument every modality boundary. Everything below is about how.

## 2. Vision Evals: Scoring Spatial Reasoning and OCR Accuracy in Agents

Vision in an agent is rarely “describe this image.” It is *grounded extraction* — pull the value that drives the next action. We decompose vision scoring into three layers.

**Layer 1 — OCR fidelity, scored field-wise.** Character Error Rate (CER) is too coarse. Production documents have *load-bearing fields*: account numbers, totals, dates, IDs. We build a per-field confusion matrix and weight errors by downstream consequence. A misread “8” in a footnote is noise; a misread “8” in a payment amount is a P0. Metrics: per-field exact match, CER, and a **critical-field recall** gate that must hit 100% before any agent touching money ships.

**Layer 2 — Spatial reasoning, scored relationally.** Modern documents are layouts, not strings. “The total is bottom-right of the last line item” is a *spatial* claim, and OCR that reads the right text in the wrong region will corrupt downstream joins. We evaluate spatial reasoning with three tasks:

– **Region grounding:** given a natural-language referent (“the shaded cell in the Q3 column”), does the agent return a bounding box that overlaps ground truth at IoU ≥ 0.5?
– **Relational queries:** “which line item is directly above the subtotal?” — scored as exact relation match, not text similarity.
– **Structural reconstruction:** can the agent emit a table/JSON graph that preserves row-column adjacency? We diff against a canonical structure with a graph-edit-distance score.

**Layer 3 — Visual robustness under degradation.** Real inputs are phone photos: glare, skew, motion blur, JPEG artifacts, half-occluded receipts. We maintain a *degradation ladder* — clean scan, mild blur, 15° rotation, 40% shadow, low-light — and report the **accuracy retention curve** rather than a single number. A model that scores 98% clean and 61% at 15° rotation is a liability; the retention curve exposes it instantly.

Two practical cautions. First, never let the vision model self-report confidence as your only grounding signal; correlate it against actual per-field error to build a calibration curve, then threshold on *empirical* correctness, not vibes. Second, hold out document *templates* — not just documents — from training-adjacent data. Template leakage produces agents that look brilliant on benchmarks and fail on the first new vendor invoice.

## 3. Voice Evals: Measuring VAD Latencies and Emotional Prosody in Synthesis

Voice adds a timing dimension that text evals have no vocabulary for. Two voice agents can produce identical transcripts and feel entirely different to a caller. We score four things.

**VAD and endpointing latency.** Voice Activity Detection determines when the agent believes the user stopped speaking. Score the **false-trigger rate** (agent barges in mid-sentence) and the **endpointing delay** (silence after true end-of-turn). A 200 ms delay feels natural; 800 ms feels like a dropped call. We measure *distributionally* — P50/P90/P99 of endpointing latency across a scripted corpus with intentional pauses, fillers (“um…”), and mid-thought silences. The P99 is where turn-taking collapses, so it gets a dedicated budget.

**Overall conversational latency, decomposed.** Report a waterfall, not a total: VAD detection → ASR finalization → policy/LLM time-to-first-token → TTS time-to-first-byte → audio playback. This tells you exactly which stage to buy or optimize. Enterprises routinely blame “the model” for what is actually a 400 ms TTS cold start.

**Barge-in and interruption handling.** Does the agent stop cleanly when the caller speaks over it, then resume context rather than restarting? Score barge-in as a transition-accuracy problem: correct halt, correct re-anchoring, and no duplicated or dropped content on resume.

**Emotional prosody in synthesis.** This is the hardest and most consequential axis. On sensitive deployments — crisis lines, healthcare, financial hardship calls — tone is a safety feature. We score prosody three ways: (1) **affective matching** against a target emotion label (calm, warm, apologetic, urgent) using a held-out speech-emotion model as a judge; (2) **human Likert panels** on naturalness, empathy, and trust for a sampled stratified set; and (3) **prosodic consistency** — F0 variance, speaking rate, and pause distribution staying within bounds across a long session, catching the “drifts robotic by minute six” failure. We also explicitly test *avoidance*: does the model over-empathize into condescension, or flatten distress into cheerfulness? Both are scored as failures.

For any high-stakes voice deployment, prosody and safety are gated together. A perfect task completion delivered in a clinically flat tone to a caller in crisis is a regression, not a win.

## 4. Cross-Modal Logic: How Models Reason Between Vision and Tool-Call Intents

The genuinely novel evaluation problem is *cross-modal logic*: given what the agent sees and hears, which tool does it call, with what arguments, and why? This is where multi-modal agents either become useful or become dangerous.

Consider a support agent receiving a photo of a damaged product plus a voice complaint. The correct trajectory is a *fusion* of modalities: the image drives a product-identification or warranty-status lookup; the voice drives intent (refund vs. replacement) and urgency; the tool call must combine both. Failure modes cluster into four categories, each needing its own probe:

**Modality neglect.** The agent reads the image but issues a tool call that ignores it (generic “please return the item” template). Detected by ablation: mask one modality and confirm the tool-call arguments *change*. If masking the image leaves the call identical, the model isn’t actually fusing.

**Spurious fusion.** The opposite failure — the model hallucinates a correlation between a visual feature and an action. A red product photo triggering a “hazard” escalation with no textual basis. We test this with *counterfactual pairs*: same utterance, semantically-irrelevant visual perturbation. Any tool-call change is a spurious-fusion flag.

**Grounding drift across turns.** Vision context from turn 2 gets dropped by turn 5. We score **reference integrity**: does the final tool call still cite the correct entity, amount, or ID extracted early in the trajectory? This is a long-horizon attention test, and it fails far more often than teams expect.

**Conflicting-modality arbitration.** Image says one thing, voice says another (“this is the wrong item” over a photo of the right one). The correct behavior is often to *ask*, not to guess. We score arbitration as a three-way decision (trust vision / trust voice / escalate) against ground-truth policy, and reward calibrated escalation — the single most under-measured skill in agent design.

For all four we report **tool-call precision and recall at the argument level**, plus a *rationale-consistency* check: does the agent’s stated reason actually entail the call it made? An agent that acts correctly for the wrong reason will fail the next, slightly different input.

## 5. Implementation: Building a Multi-Modal Gold-Set for Production Testing

A gold set is not a dump of examples. It is a **specification of acceptable behavior**, versioned and maintained like code. Here is the pipeline we use.

**1. Define the task taxonomy from real traffic.** Sample production logs (or stakeholder interviews pre-launch) and cluster into intents × modality combinations: image-only, voice-only, image+voice, text+tool. Allocate gold-set volume proportionally to *business cost*, not frequency — a rare refund-fraud path earns more coverage than a high-volume FAQ.

**2. Author gold *trajectories*, not gold answers.** For each case, record: the expected tool calls and argument constraints, acceptable terminal states (often a set), hard-fail conditions, latency budgets, and modality-specific assertions (bounding box, prosody label). Encode this as structured JSON so it can be machine-scored.

**3. Build the degradation and counterfactual apparatus.** For every vision case, generate a degradation ladder. For every cross-modal case, generate spurious-fusion counterfactuals and modality ablations. This multiplies coverage without proportionally multiplying authoring cost.

**4. Layer the graders.** Deterministic checkers for schema, arguments, and latency; embedding/structural similarity for open-ended text; a calibrated model-judge for prosody and rationale consistency; and human adjudication for the strata that are either high-stakes or where model-judges disagree. Always validate the judge against human labels on a sample before trusting it — treat the judge as a model with its own eval.

**5. Score trajectories, not endpoints.** Report per-step success, terminal success, error-amplification ratio, and a Pareto view of success vs. latency vs. safety. A single scalar hides the tradeoff the business actually cares about.

**6. Wire it into CI and shadow deployments.** Run the gold set on every prompt, model, or tool-schema change. Track regressions with the same seriousness as unit-test failures, and gate releases on critical-field recall, barge-in transition accuracy, and safety strata — not on aggregate “score.”

Two disciplines keep this honest: version your gold set alongside the model, and keep a **frozen held-out split** you never tune against. Teams that only tune against their eval eventually benchmark themselves into a corner.

## 6. The Roadmap: Moving Toward ‘Omni-Agents’ in the Enterprise

The end state is the **omni-agent**: a single system that perceives text, image, audio, and structured data natively, maintains a unified world model, and acts through tools with the same fluency across modalities. We are not there. The roadmap runs through three gates.

**Gate 1 — Unified perception with calibrated confidence.** Models must expose *usable* uncertainty per modality. Today confidence is poorly calibrated and modality-specific, which is why we still hand-build arbitration logic. Progress here lets agents escalate to humans on genuine ambiguity instead of on hand-tuned heuristics.

**Gate 2 — Cross-modal memory.** Current agents lose visual and acoustic context over long horizons. Enterprise trajectories span days and hundreds of turns — a ticket, a phone call, an email thread. Durable, modality-aware memory is the missing substrate for agents that “remember the photo you sent last Tuesday.”

**Gate 3 — Governed autonomy.** Regulated enterprises need audit trails that reconstruct *why* a multi-modal decision was made: which frame, which audio segment, which tool, which policy. This is where [OnlineCloud](https://www.onlinecloud.dev/)’s MCP-native approach matters — typed tools, auditable transports, and row-level data policies make the trajectory itself an artifact you can review and replay.

The teams that win the next cycle will not be the ones with the flashiest demo. They will be the ones whose eval suites can answer, on demand and with evidence, “how does this system behave when the lighting is bad, the caller is upset, and the tool is down?” That question is answerable. It just requires treating multi-modal evaluation as the core engineering discipline it has become — not an afterthought bolted onto a text harness.

—

**Ready to put multi-modal agents into production with evals that actually hold up?** OnlineCloud is a US-based engineering studio building MCP-native agentic systems — vision, voice, and tool-use — shipped in 6–10 weeks with observability and eval suites baked in, not bolted on.

**[Talk to our engineers →](https://www.onlinecloud.dev/)** | contact@onlinecloud.dev