Implementing ‘Self-Correction’ Loops in Agentic Code Generation: Unit-Test Driven Repair

# Implementing “Self-Correction” Loops in Agentic Code Generation: Unit-Test Driven Repair


*By the engineering team at [OnlineCloud](https://www.onlinecloud.dev/) — an MCP-native studio shipping evaluated, observable agent systems.*

—

## 1. Introduction: Why One-Shot Code Generation Fails in Production

Every team that has shipped an LLM-backed coding feature has lived the same arc. The demo is electric: you paste a function signature, the model emits 40 lines of Python, and it runs. You wire it into a pipeline, point it at a Jira ticket, and two weeks later you are staring at a 3 a.m. incident caused by a generator that confidently returned a `datetime` where the downstream consumer expected a `str`.

One-shot generation fails in production for four reasons that have almost nothing to do with model quality.

**First, correctness is not a property of text — it is a property of execution.** A generated function is a hypothesis. Until it has been run against real inputs, covered by real assertions, and exercised against edge cases, it is prose with syntax highlighting. Models are trained to maximize the *plausibility* of the next token, not the *truth* of the program. Plausible code that references a function that does not exist, or that transposes two arguments, scores exactly as well during training as code that works.

**Second, context is lossy.** The prompt rarely contains the whole interface contract. Types are implicit, invariants live in a wiki, and the caller’s assumptions about nullability are undocumented. The model fills those gaps with the statistical average of GitHub — which is a reasonable prior and a terrible specification.

**Third, the error surface is combinatorial.** Off-by-one errors, empty-collection handling, timezone math, encoding, floating-point comparison, exceptions swallowed by a bare `except`. A single sampled response has to get *all* of them right simultaneously. The probability of a perfect first pass collapses exponentially with the number of independent decisions.

**Fourth — and this is the cultural failure — we treat generation as a write, when it should be a loop.** Human engineers do not write correct code in one pass. They write, run, read the traceback, and fix. Roughly 60–70% of a developer’s time is spent in exactly this read-run-repair cycle. The moment we give an agent the same loop, its functional correctness stops being a function of sampling luck and starts being a function of iteration budget.

The rest of this post walks through the architecture we use at OnlineCloud to build that loop: the Code-Test-Fix cycle, gVisor-backed sandbox execution, log-to-prompt error feedback, and a working LangGraph implementation.

—

## 2. The Self-Correction Loop: Implementing the Code-Test-Fix Cycle Autonomously

The mental model is a control system, not a pipeline. The agent is a controller; the test suite is the sensor; the sandbox is the plant.

“`
┌────────────────────────────────────────────────────────┐
│ │
▼ │
┌─────────┐ ┌──────────────┐ ┌────────────────┐ ┌──────────────┐
│ PLAN │───▶│ GENERATE │───▶│ EXECUTE │──▶│ EVALUATE │
│ (spec) │ │ (write code │ │ (gVisor sandbox│ │ (parse logs, │
│ │ │ + tests) │ │ lint + pytest)│ │ decide) │
└─────────┘ └──────────────┘ └────────────────┘ └──────┬───────┘
▲ │
│ pass? ──▶ DONE (emit artifact) │
│ │
└────────── REPAIR (feed structured errors back) ◀──────┘
fail
“`

Four design rules make this loop converge instead of thrash.

**Rule 1 — Tests are generated before the implementation is trusted.** The agent writes the function *and* a test file, but only the tests survive into the next turn if they were authored from the specification, not from the implementation. Tests derived by reading your own code are worthless — they assert what you wrote, not what was required.

**Rule 2 — Every iteration must be *strictly* monotonic or the loop halts.** Track a fitness signal: `tests_passed`, then linter errors, then type errors. If iteration *n+1* does not improve on iteration *n*, you have a hallucination spiral. Stop and escalate rather than burning tokens.

**Rule 3 — Budget is a first-class parameter.** Cap iterations (we default to 4–5), cap wall-clock per execution, cap tokens per repair prompt. Unbounded loops are how a $0.02 task becomes a $9.00 task.

**Rule 4 — The loop must leave an audit trail.** Every iteration’s code, logs, and diffs are persisted. When the agent finally passes, you want to know *what it changed and why* — both for review and as fine-tuning data.

The convergence math is optimistic: if a single pass has a ~55% chance of being fully correct and the loop has a ~70% chance of fixing a *detected* failure, four iterations push you past 90% functional correctness on a well-specified task. Section 6 puts real numbers on this.

—

## 3. Sandbox Execution: Running Generated Code and Unit Tests in Isolated Environments

You are about to execute arbitrary, model-authored code on your infrastructure. Treat that code as hostile, because eventually it will be — whether from a hallucinated `shutil.rmtree(‘/’)`, a runaway `while True`, a fork bomb, or a prompt-injected dependency that exfiltrates an env var.

Containers alone are not enough. The Docker daemon and its shared host kernel are a large attack surface, and a single kernel CVE is a full escape. For untrusted code execution we use **gVisor**.

### Why gVisor

gVisor (`runsc`) implements the Linux syscall surface in userspace. The containerized application talks to a *Sentry* process that emulates the kernel, with a separate *Gofer* mediating filesystem access. The untrusted workload’s syscalls never hit the host kernel directly, which removes the entire class of kernel-exploit escapes and dramatically shrinks the attack surface — while still giving you a normal container UX.

“`bash
# Run untrusted generated code under gVisor with hard limits
docker run –runtime=runsc \
–network=none \
–read-only \
–tmpfs /work:rw,size=64m,noexec,nosuid \
–memory=512m –cpus=1.0 –pids-limit=128 \
–security-opt=no-new-privileges \
–cap-drop=ALL \
-v “$PWD/candidate.py:/work/candidate.py:ro” \
-v “$PWD/test_candidate.py:/work/test_candidate.py:ro” \
sandbox-python:3.12 \
timeout -s KILL 30 pytest -q /work –tb=short
“`

The flags matter as much as the runtime:

– `–network=none` — no egress, so no exfiltration and no supply-chain fetch at runtime.
– `–read-only` + `tmpfs` — the root filesystem is immutable; scratch space is a small, non-executable tmpfs.
– `–pids-limit` — kills fork bombs before they take the host down.
– `–memory` / `–cpus` — a deterministic ceiling, not best-effort.
– `timeout -s KILL 30` — SIGKILL, not SIGTERM. Generated code with signal handlers cannot opt out.
– `–cap-drop=ALL` + `no-new-privileges` — strips the ability to re-escalate.

### Operational shape

At OnlineCloud we run a pool of pre-warmed sandboxes behind an internal service. A request looks like `{files, entrypoint, timeout_s, limits}`, and the response is `{exit_code, stdout, stderr, duration_ms, oom_killed, truncated}`. Pre-warming matters: cold-starting a language toolchain adds seconds, and multi-iteration loops multiply that. We keep N warm gVisor sandboxes per language and recycle each one after every job so no state leaks between candidates.

Isolation is also a *correctness* feature, not just security. A hermetic sandbox with a pinned toolchain version means a red test means the code is wrong — not that the environment drifted.

—

## 4. Error Feedback: Parsing Linter and Test-Runner Logs into Actionable Model Prompts

Here is where most self-correction loops quietly fail. Teams dump the raw 4,000-line pytest traceback into the next prompt, the model gets distracted by stack frames from its own test harness, and it “fixes” the wrong thing.

**Never forward raw logs. Distill them into a structured repair packet.**

Three parsers sit between the runner and the model:

1. **Test parser** — extracts, per test: node id, outcome, and the *assertion* line rather than the full trace. We use pytest’s `–json-report` plugin (or a `pytest_runtest_logreport` hook) to get structured results instead of scraping text.
2. **Linter parser** — Ruff / ESLint / mypy in JSON mode. Each diagnostic becomes `{file, line, col, code, message}`.
3. **Diff-aware locator** — maps each failure back to the exact span in the candidate file, so the model edits 3 lines instead of rewriting the module.

The distilled packet is small, deterministic, and ordered by severity:

“`python
# Structured feedback handed back to the model — NOT the raw traceback.
repair_packet = {
“iteration”: 2,
“summary”: “2 of 5 tests failing; 1 lint error; 0 type errors”,
“priority”: “tests”,
“failures”: [
{
“test”: “test_parse_duration_rejects_units”,
“line”: 17,
“assertion”: “assert parse_duration(‘5m’) == 300”,
“actual”: “returned 5”,
“hint”: “unit suffix is not being converted to seconds”,
}
],
“diagnostics”: [
{“file”: “candidate.py”, “line”: 9, “col”: 1, “code”: “F401”,
“message”: “‘math’ imported but unused”}
],
“instructions”: (
“Fix ONLY the failing assertions above. Preserve the public ”
“signatures. Do not modify the tests. Return the full file.”
),
“budget”: {“iteration”: 2, “max_iterations”: 5},
}
“`

Four principles govern the prompt discipline:

– **Assertion over traceback.** The assertion encodes intent; the traceback encodes plumbing. Give the model the former.
– **Actual vs. expected, both quoted.** Half of all repair failures come from the model guessing the wrong direction of a comparison.
– **Constrain the edit.** “Fix only these assertions; do not change the tests” prevents the classic degenerate strategy of *making the test pass by editing the test*.
– **Feed the loop state back in.** Showing `iteration: 2 / 5` measurably reduces repeated failed strategies and stops the model re-proposing the change that already failed.

The single highest-leverage line in the whole system is `instructions`. Treat the repair prompt as an API contract, not a chat message.

—

## 5. Implementation: A Self-Correcting Coding Agent in Python with LangGraph

Below is the skeleton we use, trimmed for clarity. The graph has five nodes with one conditional edge — the loop itself.

“`python
from __future__ import annotations
import json, subprocess, tempfile, textwrap
from typing import TypedDict, Literal
from langgraph.graph import StateGraph, START, END
from langchain_core.messages import SystemMessage, HumanMessage

MAX_ITERATIONS = 5

class AgentState(TypedDict):
spec: str
code: str
tests: str
iteration: int
status: Literal[“ok”, “fail”]
feedback: dict # the distilled repair packet
history: list[dict] # audit trail

# —————————————————————- GENERATE
def generate(state: AgentState) -> AgentState:
if state[“iteration”] == 0:
prompt = f”””You are a senior Python engineer.
Write (a) the implementation and (b) pytest tests for this spec:

SPEC: {state[‘spec’]}

Return STRICT JSON: {{“code”: “…”, “tests”: “…”}}.
Tests must assert the spec, not the implementation.”””
else:
prompt = f”””Your previous attempt failed. Iteration {state[‘iteration’]}.

REPAIR PACKET:
{json.dumps(state[‘feedback’], indent=2)}

PREVIOUS CODE:
{state[‘code’]}

Rules: fix ONLY the reported failures. Never edit the tests.
Return STRICT JSON: {{“code”: “…”, “tests”: “…”}}.”””

raw = call_model([SystemMessage(content=”You output only valid JSON.”),
HumanMessage(content=prompt)])
payload = json.loads(extract_json(raw))
return {**state, “code”: payload[“code”],
“tests”: state[“tests”] or payload[“tests”],
“iteration”: state[“iteration”] + 1}

# —————————————————————- EXECUTE
def execute(state: AgentState) -> AgentState:
“””Runs in a gVisor sandbox: no network, read-only root, hard limits.”””
result = run_in_sandbox( # returns exit_code/stdout/stderr
files={“candidate.py”: state[“code”], “test_candidate.py”: state[“tests”]},
entrypoint=”pytest -q –json-report –json-report-file=/work/report.json”,
timeout_s=30, memory=”512m”, network=False,
)
return {**state, “feedback”: parse_feedback(result)}

# —————————————————————- EVALUATE
def evaluate(state: AgentState) -> Literal[“repair”, “done”, “give_up”]:
packet = state[“feedback”]
if packet[“summary”].startswith(“0 failed”) and not packet[“diagnostics”]:
return “done”
if state[“iteration”] >= MAX_ITERATIONS:
return “give_up”
# monotonicity guard: no progress two turns in a row -> stop
h = state[“history”]
if len(h) >= 2 and h[-1][“passed”] <= h[-2]["passed"]: return "give_up" return "repair" # ---------------------------------------------------------------- FEEDBACK def parse_feedback(result: dict) -> dict:
report = json.loads(result[“stdout_report”]) # pytest –json-report
failures = [
{“test”: t[“nodeid”], “line”: t[“line”],
“assertion”: t[“assertion”], “actual”: t.get(“actual”)}
for t in report[“tests”] if t[“outcome”] == “failed”
]
diagnostics = run_ruff_json(result) # ruff –output-format json
passed = len(report[“tests”]) – len(failures)
return {“summary”: f”{len(failures)} failed / {passed} passed”,
“failures”: failures, “diagnostics”: diagnostics,
“instructions”: “Fix ONLY the failing assertions. Do not edit tests.”}

def run_in_sandbox(*a, **k) -> dict:
return subprocess.run( # simplified
[“docker”, “run”, “–runtime=runsc”, “–network=none”, “–read-only”,
“–memory=512m”, “–cpus=1.0”, “–pids-limit=128”, “–cap-drop=ALL”,
“sandbox-python:3.12”, “python”, “-m”, “runner”],
capture_output=True, text=True, timeout=k[“timeout_s”] + 5,
).__dict__

# —————————————————————- GRAPH
g = StateGraph(AgentState)
g.add_node(“generate”, generate)
g.add_node(“execute”, execute)
g.add_node(“repair”, lambda s: s) # repair == re-enter generate
g.add_edge(START, “generate”)
g.add_edge(“generate”, “execute”)
g.add_conditional_edges(“execute”, evaluate,
{“repair”: “generate”, “done”: END, “give_up”: END})
app = g.compile()
“`

The details that matter in production:

– The `monotonicity guard` in `evaluate` is what stops the infinite loop when the model fixates on a wrong hypothesis.
– `state[“tests”]` is only set on iteration 0 — a deliberate one-way latch so repair turns cannot rewrite the oracle.
– `run_in_sandbox` is the only place untrusted code is ever touched; it is a hard process boundary, and every call is logged with its resource limits.
– The `history` list is the audit trail and the future fine-tuning corpus.

Everything above is model-agnostic. Swap `call_model` for Claude, a GPT-4-class model, or an open-weights deployment running in your own VPC — the loop is the asset, not the model.

—

## 6. ROI: Improving Functional Correctness by 40% Through Iterative Refinement

Across the internal tasks we benchmarked — data-transform helpers, API clients, parsers, and business-rule validators — moving from one-shot generation to a bounded 5-iteration Code-Test-Fix loop with gVisor execution produced a **~40% improvement in functional correctness on the first shipped artifact**, measured as pass rate on a held-out test suite the agent never saw.

The breakdown matters more than the headline:

| Configuration | Held-out pass rate | Notes |
|—|—|—|
| One-shot, no execution | ~48% | “Looks right” only |
| One-shot + sandboxed tests | ~61% | Detection alone helps |
| Full loop (≤5 iters), raw logs | ~70% | Noisy repair signal |
| Full loop + distilled repair packets | **~84%** | ≈40% relative gain |

Three lessons from the data:

1. **Detection beats generation.** ~13 of the 40 points come from *running* the code, before any repair happens. You cannot fix what you cannot see fail.
2. **Feedback quality dominates iteration count.** Going from raw logs to distilled repair packets was worth more than doubling the iteration budget. Signal quality beats brute force.
3. **The guard rails pay for themselves.** The monotonicity guard and the iteration cap meant the median successful task converged in **2 iterations**; only 9% hit the cap. Latency rose modestly (p50 ~14s per task end-to-end) while human review time dropped sharply, because reviewers were handed a passing artifact *plus* the audit trail of how it got there.

Cost, honestly: token spend per task rose ~2.4× versus one-shot. That is the trade. You are buying correctness with iterations, and at ~$0.01–0.05 of extra inference per task, it is the cheapest QA engineer you will ever hire.

The strategic point is not the 40%. It is that the loop turns correctness from a *probability you sample* into a *property you engineer* — and that is the difference between a coding demo and a production system.

—

## Ship self-correcting agents with OnlineCloud

If your team is trying to get LLM-generated code into production and hitting the wall between “works in the notebook” and “passes CI,” this is exactly the kind of system we build. OnlineCloud is a US-based, senior-only engineering studio specializing in MCP-native agent architectures, sandboxed execution, and evaluated, observable AI systems — shipped into your VPC, wired to your existing stack, in 6–10 weeks rather than quarters.

**We don’t sell decks — we ship evaluated, observable systems.**

– Design a Code-Test-Fix loop tailored to your codebase and eval suite
– Stand up gVisor-backed sandbox execution with real resource limits and audit trails
– Build the log-distillation layer that makes repair prompts actually converge
– Instrument correctness, latency, and cost-per-task so you can prove the ROI

**[Book a free initial consultation →](https://www.onlinecloud.dev/)** — reach us at **contact@onlinecloud.dev**. Response within 24 hours.

*Note: the benchmark figures above are from OnlineCloud’s internal task suites and will vary with model choice, task complexity, and specification quality. We will reproduce the measurement on your data before you commit to anything.*