Building a Custom Evaluator for Code-Generation Agents: The Sandbox Pattern

# Building a Custom Evaluator for Code-Generation Agents: The Sandbox Pattern


*By the engineering team at [OnlineCloud](https://www.onlinecloud.dev/) — we build evaluated, observable agent systems that ship to production in 6–10 weeks.*

—

## Introduction: Why ROUGE and BLEU Are the Wrong Ruler for Code

If you have ever shipped an LLM feature that generates code, you have probably felt the same quiet panic: the model looks correct, the demo is smooth, and yet you have no idea whether the thing actually *works*. So teams reach for the metrics they already know — ROUGE, BLEU, METEOR, maybe BERTScore if they are feeling fancy. These are lexical overlap metrics from machine translation and summarization. They measure how similar a generated string is to a reference string. Code is not a string. Code is a specification that either executes correctly or does not.

Consider a concrete example. A user asks for a function that returns the nth Fibonacci number. The reference implementation uses an iterative loop with two accumulator variables. Your agent returns a memoized recursive version. The two functions are functionally identical — same inputs, same outputs, same complexity class in practice — but they share almost no surface tokens. ROUGE-L would score that pair somewhere in the 0.3–0.4 range and tell you your model is mediocre. It is not.

Now invert the example. The agent returns the reference implementation character-for-character, but with one token changed: `i <= n` becomes `i < n`. ROUGE-L is near 1.0 — a "perfect" score. The function now silently off-by-one's on every input. Your metric gave the broken code a gold medal. This is the fundamental failure mode: **lexical metrics are invariant to the only property that matters, which is behavior.** They reward paraphrasing failure and punish correct alternate solutions. For code generation they are worse than useless — they are actively misleading, because they create false confidence in CI dashboards. There are three deeper problems: 1. **The reference-bias problem.** ROUGE/BLEU assume a single canonical answer. Real code has an enormous space of correct solutions — different algorithms, different style conventions, different library choices. A metric that privileges one reference is measuring "did you copy the answer key," not "did you solve the problem." 2. **The non-compositional problem.** Meaning in code is compositional and scoped. A metric that sums token overlap cannot see that a renamed variable is harmless while a flipped comparison operator is fatal. It has no grammar, no scope, no types. 3. **The non-executable problem.** The metric never runs anything. Bugs are exactly the behaviors that survive static similarity checks. Off-by-one errors, null dereferences, resource leaks, race conditions — all of these produce near-perfect string similarity and catastrophic runtime behavior. The fix is not a better similarity metric. The fix is to stop comparing text and start **executing behavior**. That means a custom evaluator that treats the model's output as a candidate program, runs it in an isolated environment against a suite of tests, and scores it on what actually happened. This post is the engineering blueprint for that evaluator — the **Sandbox Pattern**. --- ## The Sandbox Pattern: Secure, Ephemeral Execution Environments The core idea of the Sandbox Pattern is deceptively simple: **never evaluate generated code on a machine you care about.** Instead, for every candidate program, spin up a disposable, resource-bounded, network-isolated execution environment, run the tests, harvest the results, and destroy the environment. The evaluator itself never trusts the artifact it is scoring. ### Why you cannot just use a subprocess The naive approach — `subprocess.run(["python", candidate.py])` on your eval worker — is the single most common mistake we see. It fails on every axis that matters: - **Filesystem escape.** Generated code runs with your worker's user privileges and can read `~/.aws/credentials`, `.env` files, or write a persistence hook into your CI runner. - **Network egress.** It can exfiltrate environment variables, call home, or scan your internal VPC. - **Resource exhaustion.** An infinite loop or a fork bomb takes down the worker, not the candidate. - **Resource leakage.** Leftover processes, open sockets, and temp files accumulate until the evaluator itself becomes unstable. - **Non-determinism.** The candidate mutates its own environment between runs, so scores are not reproducible. A container alone improves things but does not solve them. Docker shares the host kernel, and a kernel exploit in generated code — or a cleverly crafted `syscall` — escapes the container entirely. Containers provide isolation for trusted workloads; generated code is definitionally untrusted. ### Kernel-level isolation: gVisor and Firecracker To evaluate untrusted code, you need a real isolation boundary. There are two production-grade options, and they use different philosophies. **gVisor** inserts a user-space kernel between the untrusted program and the host. When the candidate program makes a syscall, gVisor's `Sentry` intercepts it and re-implements the syscall in memory, brokering only a tiny, vetted surface down to the real host kernel. The effect: the code *thinks* it is talking to Linux, but the host kernel sees only a narrow, audited set of operations. gVisor gives you strong syscall-level defense with startup times measured in tens of milliseconds — fast enough to run per-test-case if you need to. **Firecracker** takes a different route: microVMs. Each evaluation runs inside a lightweight KVM-based virtual machine with its own guest kernel, booted in roughly 125 milliseconds and consuming a few megabytes of memory. Because the guest has its own kernel, the isolation boundary is hardware-virtualization-grade — the same primitive AWS Lambda uses under the hood. Firecracker is the right choice when you need the strongest possible boundary, when candidates may be multi-language or need a full OS surface, or when regulatory constraints demand VM-level separation. A practical architecture uses both: **gVisor for fast, high-volume inner-loop scoring** during development, and **Firecracker microVMs for the outer, untrusted boundary** in CI and production. Some teams nest them — a Firecracker microVM running a gVisor sandbox — for defense in depth. ### The ephemeral lifecycle Whatever the isolation primitive, the lifecycle is always the same five phases: 1. **Provision.** Create a fresh, minimal root filesystem from a pre-built image. No secrets, no host mounts, no persistent volumes. 2. **Inject.** Write the candidate program and the test harness into an in-memory or tmpfs location. Mount read-only where possible. 3. **Constrain.** Apply hard limits *before* execution: CPU quota (e.g., 2 vCPU-seconds), memory ceiling (e.g., 512 MB), wall-clock timeout (e.g., 5s), process/thread cap, file-descriptor cap, and a blocked network namespace. 4. **Execute.** Run the tests, stream stdout/stderr to a bounded buffer, and collect exit codes. 5. **Destroy.** Tear down the environment unconditionally, discarding the entire filesystem. Nothing persists. The next candidate starts from a pristine image. The word that carries the most weight here is **ephemeral**. Cold-start-per-evaluation is a feature, not a cost, because it eliminates cross-contamination: candidate A cannot poison the environment that scores candidate B, and a flaky candidate cannot leave residue that corrupts future scores. ### Implementation sketch A minimal evaluator loop in Python, using a sandbox abstraction that maps to gVisor or Firecracker at runtime: ```python from dataclasses import dataclass from sandbox import Sandbox, Limits # gVisor/Firecracker-backed @dataclass class EvalResult: passed: int total: int timed_out: bool stderr: str def evaluate(candidate_src: str, tests: list[TestCase]) -> EvalResult:
limits = Limits(
cpu_seconds=2.0,
memory_mb=512,
wall_timeout_s=5.0,
max_procs=64,
network=”none”, # network namespace fully isolated
filesystem=”ephemeral”, # discarded after run
)
with Sandbox(image=”python:3.12-slim”, limits=limits) as sb:
sb.write(“/work/solution.py”, candidate_src)
sb.write(“/work/test_solution.py”, render_tests(tests))
proc = sb.run(
[“pytest”, “-q”, “/work/test_solution.py”, “–timeout=3”],
capture=True,
)
return EvalResult(
passed=proc.parsed.passed,
total=len(tests),
timed_out=proc.exit_code == 124,
stderr=proc.stderr[:4096], # bounded, never unbounded
)
“`

The `with` block guarantees teardown. The `Limits` object is the security contract. The `network=”none”` and `filesystem=”ephemeral”` flags are the two that prevent the most damage.

—

## Unit-Test Scoring: Measuring Behavior, Not Text

With a sandbox in place, the scoring function becomes almost trivial — and dramatically more honest. Instead of comparing strings, you run tests and count passes.

### The scoring formula

A practical composite score weights several signals:

“`python
def functional_correctness(result: EvalResult) -> float:
if result.total == 0:
return 0.0
base = result.passed / result.total # test-pass rate
if result.timed_out:
base *= 0.5 # penalize non-termination
return round(base, 4)
“`

The **test-pass rate** — `passed / total` — becomes your primary functional metric. It is monotonic in the right direction: fixing a bug increases it, breaking a feature decreases it. It is invariant to how the code is written and sensitive to what the code does. That is exactly the property ROUGE lacks.

### Designing the test suite

The test suite is now the most important artifact in your evaluator, because it *is* your specification. Three tiers work well:

– **Canonical cases.** Straightforward inputs and expected outputs. These catch the obvious failures and anchor the score.
– **Edge cases.** Empty inputs, single elements, maximum sizes, negative numbers, unicode, nulls. These are where LLMs fail most, and where lexical metrics are blindest.
– **Property-based cases.** Instead of fixed input/output pairs, assert invariants: sorting is idempotent, encode/decode round-trips, output length is bounded, a search function returns an element that compares equal to the target. Property tests generalize far beyond the examples you write, which makes them excellent at catching overfitting.

### Partial credit and weighting

Not all test cases are equal. A schema-validation failure is less severe than a crash on the happy path. Weight cases accordingly — core functionality at 1.0, edge cases at 1.5, performance-sensitive cases with their own budgets — so the score reflects *severity* as well as *frequency*.

### Why this resists gaming

A concern you should anticipate: can the model game the tests? Only if the tests are visible to the model and shallow. Mitigations:

1. **Hold out tests.** Score against a private suite the model never sees; keep a public subset for the model to self-check against.
2. **Randomize inputs.** Generate test parameters at evaluation time from a seed, so memorized answers do not survive.
3. **Adversarial cases.** Include inputs specifically designed to break shortcuts and hardcoded lookups.
4. **Cross-check properties.** Where possible, verify with an independent oracle rather than a fixed constant.

—

## Security: Preventing Escape from the Evaluator Sandbox

The uncomfortable truth about evaluating an adversary’s code — and a code-generation model can absolutely produce adversarial output, either through a hostile prompt or a data-poisoning attack — is that your evaluator is now an attack surface. Treat it as one.

### The threat model

Assume generated code will attempt, at minimum:

– **Credential theft** — reading environment variables, `~/.ssh`, cloud metadata endpoints.
– **Network exfiltration** — POSTing data to an attacker-controlled host.
– **Cryptojacking** — running a miner on your GPU or CPU budget.
– **Resource exhaustion** — infinite loops, fork bombs, memory bombs, zip bombs.
– **Sandbox escape** — syscall abuse, kernel exploits, container breakouts.
– **Supply-chain tampering** — `pip install` of a malicious package, or modifying the test harness itself.

### Defense-in-depth controls

**Network.** Default to no network at all. Run the candidate in a network namespace with no interfaces. If a task genuinely needs a package, pre-populate a local wheelhouse and serve it read-only *outside* the sandbox’s reachable surface — never give the candidate open egress. Block cloud metadata IPs (`169.254.169.254`) explicitly even when networking is nominally “on.”

**Filesystem.** Mount the root read-only. Give the candidate a single writable tmpfs mount, size-bounded, discarded on teardown. Never mount the Docker socket, the host `/proc`, or any credential path. Strip all secrets from the environment before execution — a sandbox with `AWS_ACCESS_KEY_ID` in its env is not a sandbox.

**Resources.** Enforce limits with the kernel, not with convention: cgroups v2 for CPU and memory, `RLIMIT_NPROC` for process count, `RLIMIT_NOFILE` for descriptors, and a hard wall-clock timeout enforced from *outside* the sandbox so a wedged process cannot block its own kill. Watch for the classic traps — `/dev/zero` reads, decompression bombs, symlink races.

**Least privilege.** Run the sandboxed process as an unprivileged, non-root user with a dropped capability set. Drop `CAP_SYS_ADMIN`, `CAP_NET_RAW`, and everything else not strictly required. Enable seccomp to allowlist only the syscalls the runtime actually needs — with gVisor, this happens by construction.

**Isolation hygiene.** One evaluation per sandbox. Never reuse an environment across candidates. Randomize sandbox IDs. Rotate the base image on a schedule and pin it by digest so a compromised registry tag cannot silently change what you run.

**Observability and kill switches.** Log every sandbox lifecycle event — provision, limit hit, teardown. Alert on anomalies: a candidate that spikes CPU for the full timeout, a burst of metadata-endpoint probes, a sudden jump in network attempts. Maintain a circuit breaker that disables evaluation if the escape rate crosses a threshold, and require human review before re-enabling.

The goal is not to make escape impossible — it is to make escape *expensive, noisy, and contained*. gVisor and Firecracker give you the boundary. cgroups and seccomp give you the constraints. Logging and alerting give you the tripwires. Layer all three.

—

## Metrics: Tracking Functional Correctness and Linter Pass Rate in CI

An evaluator is only as good as the signals it surfaces in your pipeline. Two metrics carry the most weight, and both belong on the same dashboard.

### Functional Correctness

**Functional Correctness** is the test-pass rate aggregated across your evaluation set — optionally weighted by case severity. Report it multiple ways, because a single number hides the distribution:

– **Mean pass rate** across tasks — the headline number.
– **Pass@k** — the fraction of tasks solved at least once in `k` attempts. Critical for agentic workflows where you sample multiple candidates and pick the best.
– **Median and p10** — to expose tasks where the model is consistently failing. A mean of 0.8 built from all-1.0s and a bimodal tail of 0.2s is a very different system from a uniform 0.8.
– **Segmented by task class** — algorithms, data transforms, API integration, SQL, concurrency. Aggregate numbers hide class-specific regressions.

Gate your CI on it. A pull request that changes a prompt template, swaps a model, or touches the agent loop must not merge if Functional Correctness drops by more than an agreed epsilon. Treat it exactly like test coverage: a metric that fails the build when it regresses.

### Linter Pass Rate

**Functional Correctness** answers “does it work.” **Linter Pass Rate** answers “is it shippable.” A function can pass every test and still be unacceptable — unused imports, shadowed variables, unhandled exceptions, missing type annotations, unsafe patterns flagged by `bandit` or `semgrep`.

Track linter pass rate as a first-class metric alongside correctness, reporting both the raw pass rate and the distribution of violation categories. Two reasons this matters:

1. **It catches sloppiness that tests miss.** A test exercises the happy path; a linter sees the `bare except:` sitting next to it.
2. **It correlates with downstream review cost.** Code that passes a linter reduces human review time, which is the real bottleneck in any agent-assisted workflow.

Run your standard stack — `ruff` or `flake8` for Python, `eslint` for JavaScript/TypeScript, `clippy` for Rust, plus a security scanner — inside the same sandbox, and record results in the same evaluation artifact. Combine the two into a single gated score, for example `0.7 * correctness + 0.3 * linter_pass_rate`, and version that formula alongside your eval suite so changes are reviewable.

### Instrumentation and reporting

Emit one structured record per evaluation — task ID, model/version, sandbox ID, test outcomes, lint outcomes, latency, resource peaks, timestamps. Store it durably and visualize trends over time. Without historical tracking you cannot tell whether a prompt change helped or whether the eval set drifted. The dashboard should answer one question at a glance: *is the system getting better or worse, for which task classes, at what cost?*

—

## Roadmap: A 6–10 Week Guide to Autonomous Coding Agents

This is the sequence we follow with clients. It front-loads the evaluator deliberately — you cannot safely automate what you cannot measure.

**Weeks 1–2 — Foundation and harness.** Stand up the sandbox infrastructure. Choose gVisor for the inner loop, Firecracker for the outer boundary. Build the five-phase lifecycle, wire in cgroups/seccomp limits, and ship a single-language, single-task evaluator end to end. Success criterion: a candidate program runs, gets scored on unit tests, and the sandbox is destroyed cleanly with zero residue.

**Weeks 3–4 — The test corpus and the linter gate.** Build the three-tier test suite (canonical, edge, property-based) for your target task class. Add the holdout split and input randomization. Integrate linters and security scanners into the sandbox. Establish baseline Functional Correctness and Linter Pass Rate for your current model and prompt. Success criterion: a reproducible, versioned eval suite with a baseline dashboard.

**Weeks 5–6 — CI integration and gating.** Wire the evaluator into CI as a required check. Define regression thresholds and fail the build on them. Add structured evaluation records and trend visualization. Run a friction sprint to cut eval latency — parallelism, image caching, warm sandbox pools. Success criterion: every prompt/model change is scored automatically and blocked on regression.

**Weeks 7–8 — The agent loop.** Now wrap the model in a plan/act/reflect loop with the evaluator as its reward signal. Add retry-on-test-failure with structured feedback (the failing test’s name and assertion, not the whole trace), and best-of-k selection ranked by score. Introduce cost ceilings and human checkpoints for high-risk changes. Success criterion: the agent self-corrects on failed tests and measurably outperforms single-shot generation.

**Weeks 9–10 — Hardening and autonomy.** Red-team your own sandbox: attempt escapes deliberately, probe the resource limits, and verify your tripwires fire. Expand the task corpus to the classes that matter for your domain. Turn on autonomous operation for low-risk task classes with the evaluator as the gate, keeping humans in the loop for the rest. Publish the metrics and the operating envelope. Success criterion: an agent that ships code you would merge, with an evaluation trail you can audit.

The through-line across all ten weeks is the same: **behavior is the ground truth, and the sandbox is how you observe it.** Lexical metrics gave you a ruler marked in the wrong units. Functional Correctness and Linter Pass Rate, measured inside an ephemeral, kernel-isolated sandbox, give you one that actually predicts whether the code is worth shipping.

—

## Build This With OnlineCloud

OnlineCloud is a US-based Agentic AI & MCP engineering studio. We design the sandbox evaluators, agent loops, and observability that turn code-generation prototypes into systems you can trust in production — and we ship them in 6–10 weeks, not quarters. Every engagement gets senior engineers, an eval suite that defines “done” before a line of agent code is written, and an architecture built to be audited end to end.

**Ready to build a code-generation agent that you can actually measure?** [Book a free consultation](https://www.onlinecloud.dev/) — we’ll map your task corpus, pick your isolation strategy, and define the eval that proves the thing works. Or reach us directly at **contact@onlinecloud.dev**; US-based support, response within 24 hours.

*OnlineCloud · Xlent Info tech. LLC · [www.onlinecloud.dev](https://www.onlinecloud.dev/)*