Automating DevOps with Agentic Swarms: CI/CD Log Analysis and Self-Healing Builds

# Automating DevOps with Agentic Swarms: CI/CD Log Analysis and Self-Healing Builds


*By a Senior DevOps Lead — a field report on replacing log-staring with agentic triage.*

—

## Introduction: The Manual Toil of Triaging Jenkins and GitHub Actions Logs

Let’s be honest about where a DevOps engineer’s week actually goes. It isn’t architecture. It isn’t the elegant pipeline you designed on a whiteboard. It’s the 14,000-line Jenkins console dump that scrolled past you at 2:47 AM, and the one `##[error]` buried on line 9,812 of a GitHub Actions run.

I’ve spent the better part of a decade watching engineers do what I call **log archaeology**. A build fails. A developer pings the channel. Someone opens the run, expands the `maven` stage, and starts the tedious binary search: scroll, `Ctrl+F` for “ERROR”, read three unrelated stack traces, finally find that a flaky integration test timed out because an environment variable was renamed upstream. Forty minutes gone. Multiply that across a fleet of 300 repositories running 6,000 builds a day, and the toil becomes a structural tax on the entire engineering organization.

The cruel irony is that this data is *perfectly structured for machines*. Build logs are deterministic text with a beginning, a failing stage, a root cause, and a remediation. A human reads them slowly and inconsistently. A machine reads them instantly and consistently. The gap has never been the data — it’s been the reasoning layer that sits between “raw stdout” and “here’s the fix, here’s the PR.”

That reasoning layer is now available in the form of **agentic swarms** — orchestrated collections of specialized AI agents that plan, investigate, and act on CI/CD telemetry. This post is a deep-dive into how we built one: how the swarm is structured, how it connects to runners via MCP tool-use, how we implemented a self-healing `Commit-Build-Fix` loop, how we instrumented MTTR, and what happened when we turned it loose on a real monorepo.

## The DevOps Swarm: Planner Agents and Worker Agents

The single most important design decision we made was **not** to build a monolithic “fix my build” bot. A single agent with a 200K-token log dump in its context window is slow, expensive, and — critically — prone to confidently hallucinating fixes for the wrong root cause. Instead we adopted a **hierarchical swarm** inspired by classic distributed systems: a planner that owns the objective, and workers that own bounded tasks.

### The Triage Planner

The Planner is the swarm’s incident commander. It never touches a file. Its job is to consume the *envelope* of a failed run — the repository, the commit SHA, the failing workflow, the exit codes, the stage boundaries — and produce a structured repair plan.

Concretely, the Planner does four things:

1. **Classifies the failure.** Flaky test, dependency drift, infrastructure/capacity, config error, merge conflict, or genuine code defect. This classification is the highest-leverage output, because it determines whether the swarm should retry, fix, or page a human.
2. **Selects the right workers.** A `npm ERR! EINTEGRITY` doesn’t need the code-fixing worker; it needs the dependency-repair worker and possibly a cache-invalidation action. Routing is everything.
3. **Sets a budget.** Every plan carries a maximum number of build attempts (we default to three), a wall-clock ceiling, and a blast-radius policy. The planner is explicitly instructed never to modify protected branches or CI secrets.
4. **Emits a typed plan**, not prose. We use a strict JSON schema so the orchestrator can validate and execute it deterministically.

### The Repair Workers

Workers are narrow specialists with small, curated tool surfaces. Each one gets exactly the tools it needs and nothing more — a principle we borrowed from security’s least-privilege model. Our production swarm runs five:

– **Log-Forensics Worker** — streams the failing log, chunks it, and extracts the causal chain: last passing stage → first failing command → root-cause line → related context. It returns a compressed, structured diagnosis rather than the whole blob. This single worker eliminated most of the token cost.
– **Dependency-Repair Worker** — handles lockfile conflicts, semver resolution failures, registry auth errors, and version pinning. Its tools include reading `package.json`/`pom.xml`, running a dry-run install, and proposing a lockfile patch.
– **Code-Fix Worker** — the one that actually writes diffs. It receives the diagnosis plus a scoped view of the repo and produces a minimal patch, always accompanied by a test. It is forbidden from touching files outside the failing module without explicit planner approval.
– **Config-Repair Worker** — fixes workflow YAML, environment variable mismatches, runner labels, matrix definitions, and secret references.
– **Verifier Worker** — the skeptic. Before any fix is committed, the Verifier re-runs the affected test locally (or in a sandbox), checks that the patch doesn’t break adjacent suites, and validates that the diff matches the stated root cause. If it can’t reproduce the fix, the patch dies here.

The swarm’s power comes from this **separation of reasoning concerns**. The planner never gets seduced by a plausible-looking patch, because it isn’t writing patches. The code worker never over-broadens scope, because the planner bounded it. The verifier never trusts the fixer, because that’s its entire purpose.

## Integration: Bridging CI/CD Runners to AI Models via MCP Tool-Use

An agent swarm is only as useful as its ability to *act on the real system*. That’s where tool-use — and specifically the **Model Context Protocol (MCP)** — becomes the connective tissue.

MCP gives us a standardized way to expose runner capabilities as callable, describable tools. Instead of writing bespoke glue for Jenkins, GitHub Actions, and GitLab, we expose a uniform tool surface. Our CI bridge registers tools that map tightly to operations the swarm needs:

“`json
{
“tools”: [
{ “name”: “get_run_log”, “scope”: “read”, “args”: [“run_id”, “stage”] },
{ “name”: “get_run_metadata”, “scope”: “read”, “args”: [“run_id”] },
{ “name”: “list_artifacts”, “scope”: “read”, “args”: [“run_id”] },
{ “name”: “rerun_stage”, “scope”: “write”, “args”: [“run_id”, “stage”] },
{ “name”: “open_pull_request”, “scope”: “write”, “args”: [“branch”, “title”, “diff”] },
{ “name”: “comment_on_run”, “scope”: “write”, “args”: [“run_id”, “body”] }
]
}
“`

The critical architectural choice: **the model never holds credentials.** The swarm reasons about *which* tool to call and *with what arguments*; the bridge executes that call server-side with short-lived, scoped tokens minted per run. This keeps secrets out of prompts, out of logs, and out of the model’s context window entirely.

A typical tool-use sequence for a failing build looks like this:

1. `get_run_metadata(run_id)` → the Planner learns the failing workflow and commit.
2. `get_run_log(run_id, stage)` → the Forensics Worker pulls only the failing stage.
3. The Planner classifies and dispatches.
4. The Code-Fix Worker proposes a diff; the Verifier reproduces it in a sandbox.
5. `open_pull_request(branch, title, diff)` → the fix lands as a reviewable PR, never a direct push.

Two implementation notes that saved us real pain. First, **streaming matters**: pulling a 200MB Jenkins log into a single tool response will blow your context and your latency budget. We stream logs in 500-line windows and let the Forensics Worker request more only when the diagnosis is ambiguous. Second, **idempotency is non-negotiable**: every write tool carries an idempotency key so a retried agent step can never open two PRs for the same failure.

## The Pattern: Implementing the ‘Commit-Build-Fix’ Loop Autonomously

The self-healing core is what we call the **Commit-Build-Fix loop**. It’s a closed-loop control system where the “sensor” is the build result and the “actuator” is a pull request. Here’s the canonical pattern:

“`python
async def commit_build_fix(commit_sha, repo, max_attempts=3):
for attempt in range(1, max_attempts + 1):
run = await ci.get_run_for_commit(commit_sha, repo)
if run.conclusion == “success”:
return FixResult(success=True, attempts=attempt – 1)

# Read-only forensics first — never mutate on a hunch.
log = await ci.get_run_log(run.id, run.failing_stage)
diagnosis = await forensics_worker.diagnose(log, run.metadata)

plan = await planner.build_plan(
diagnosis=diagnosis,
repo_graph=repo.module_graph(),
budget=Budget(attempts=max_attempts – attempt + 1),
)

# Retry-only failures short-circuit before any code is written.
if plan.classification == “flaky”:
await ci.rerun_stage(run.id, run.failing_stage)
continue

patch = await dispatch_workers(plan) # code / deps / config
if not await verifier.reproduce(patch, repo): # skeptic gate
await ci.comment_on_run(run.id, plan.escalation_note())
break # hand back to humans

branch = f”swarm/fix-{run.id}”
await vcs.open_pull_request(branch, plan.title, patch.diff)

commit_sha = await vcs.head_of(branch) # build the fix branch
repo = repo.on(branch) # loop closes here
return FixResult(success=False, attempts=max_attempts)
“`

Four guardrails make this safe enough to run in production:

– **Read-before-write ordering.** The loop will never mutate anything until it has a structured diagnosis. Retry-worthy failures (flakes) are resolved by re-running the stage, which costs nothing and fixes the majority of transient noise.
– **Attempt ceilings.** Three attempts, then escalate to a human with a full diagnostic dossier. An agent that can loop forever is an agent that can loop forever *wrong*.
– **Blast-radius policy.** Protected branches, CI secrets, and infrastructure-as-code directories are write-blocked. The swarm’s deliverable is always a pull request — a human still merges.
– **Verification gate.** No patch reaches the PR stage without independent reproduction. This killed our false-positive rate dramatically; the fixer and the verifier disagreeing is a feature, not a bug.

The result is an autonomous loop that treats a failing build as a solvable state rather than an interruption. When it works — and it works most of the time — the developer never even notices. The build was red, then it was green, and a tidy PR sits in the queue explaining what broke and why.

## Observability: Tracking Mean-Time-to-Repair (MTTR) with AI Swarms

You cannot claim a swarm is helping unless you can measure it. Our north-star metric is **MTTR — mean time to repair** — measured from the first red build on a commit to the moment that commit’s fixes land green on a protected branch.

Instrumenting MTTR for agentic systems requires new telemetry dimensions that classic CI dashboards don’t capture:

– **Classification accuracy.** Of the failures the Planner labeled “flaky,” what percentage actually passed on re-run? We track this weekly; it’s our leading indicator for diagnosis quality.
– **Per-worker attribution.** Which worker’s patch resolved the build, and at what attempt number? This tells us where to invest in better tools.
– **Token cost per repair.** Every loop consumes model tokens. We track cost alongside MTTR, because a swarm that halves repair time while quadrupling spend is a bad trade.
– **Escalation rate.** The percentage of failures the swarm hands to humans. This should *fall* over time as coverage widens — and if it rises, a category of failure is outrunning our workers.
– **Verifier veto rate.** How often the skeptic blocks a fix. A healthy signal is neither zero (verifier asleep) nor dominant (workers unreliable).

We emit these as OpenTelemetry spans per loop iteration, correlated by run ID, so a Grafana panel can show MTTR trending next to token spend and escalation rate. The dashboard that matters most is dead simple: **red-to-green time, segmented by failure class, before versus after the swarm.** That single chart is what got our platform team to sign off on the rollout.

The counterintuitive finding: the biggest MTTR win didn’t come from *fixing* builds. It came from **classifying** them fast. Knowing within seconds that a failure is a flake, and auto-re-running it, eliminated the largest slice of human triage time — long before any code was ever generated.

## Case Study: Reducing Build-Failure Downtime by 40% with Agentic Triage

We deployed the swarm against a production monorepo: ~320 services, an average of 6,400 builds per day, a mix of Jenkins (legacy) and GitHub Actions (newer services), and a pre-swarm MTTR of **94 minutes** measured across all failure classes.

**Baseline (8 weeks pre-rollout).** MTTR 94 min. Roughly 61% of failures were triaged manually by a rotating on-call engineer. The dominant categories were flaky integration tests (28%), dependency/lockfile conflicts (19%), config drift (14%), and genuine code defects (31%). Median time-to-first-human-touch was 22 minutes — the pure “notice and open the log” cost.

**Rollout.** We ran the swarm in **shadow mode** for two weeks — it produced diagnoses and patches but opened only draft PRs that no one merged — to build trust and calibrate the classification thresholds. Only then did we enable autonomous re-runs and PR creation.

**Results after 10 weeks in production:**

| Metric | Before | After | Change |
|—|—|—|—|
| Mean time to repair | 94 min | 56 min | **−40%** |
| Flaky failures auto-resolved | 0% | 91% | — |
| Dependency/config PRs auto-drafted | 0% | 74% | — |
| Human triage hours / week | ~31 hrs | ~11 hrs | −65% |
| Escalation rate (to humans) | 61% | 23% | −38 pts |
| Verifier veto rate | n/a | 12% | — |

The headline is the **40% reduction in build-failure downtime** — from 94 to 56 minutes. But the operational reality behind that number is more interesting. The swarm didn’t magically solve hard code defects; its code-fix success rate on genuine logic bugs was a modest 34%. Where it excelled was the *long tail of boring breakage*: flakes, lockfile drift, and config mismatches — categories that are collectively enormous and individually unworthy of human attention. By absorbing that tail and auto-generating reviewable PRs, the swarm freed engineers to spend their triage budget on the 31% of failures that are genuinely interesting.

Two honest caveats. First, the swarm’s first-attempt patch quality was mediocre (~48% verifier pass rate) until we invested in the Verifier and in scoped repository context; quality is a function of the skeptic, not the fixer. Second, token cost per repair settled at roughly $0.38 — trivial against a 38-minute engineering-time saving, but worth watching as volume scales.

## Conclusion: From Log Archaeology to a Self-Healing Pipeline

The manual toil of triaging Jenkins and GitHub Actions logs is not inevitable. It is an artifact of a missing reasoning layer, and that layer now exists. By structuring automation as a **hierarchical swarm** — a planner that classifies and bounds, workers that fix narrowly, and a verifier that refuses to trust — and by bridging runners to models through **MCP tool-use** with credentials held server-side, we turned the failing build from an interruption into a solvable state.

The **Commit-Build-Fix loop** gave us autonomy with guardrails: read before write, cap the attempts, block the blast radius, verify before shipping. The observability layer gave us the receipts: a 40% cut in MTTR, a 65% drop in human triage hours, and a fleet of auto-drafted PRs that humans now *review* instead of *write*.

The lesson I’d hand to any DevOps lead weighing this: **start by automating classification, not fixing.** Speed-to-diagnosis is where the toil lives, and it’s where the swarm earns trust fastest. Fix the boring 60% autonomously, keep humans on the interesting 40%, and measure relentlessly. Your on-call rotation will thank you.

—

## Ready to Stop Triaging Logs and Start Shipping?

The swarm pattern is only as good as the infrastructure it runs on. **OnlineCloud** gives platform teams the compute foundation to run agentic CI/CD orchestration — elastic runners, low-latency streaming of build telemetry, and the reliable, observable environments your swarms need to act safely at scale.

**[→ Start building self-healing pipelines on OnlineCloud](https://www.onlinecloud.dev/)**

Bring the swarm. We’ll keep the runners green.