Observability for AI Agents: Tracking Decision Logs, Cost Ceilings, and Token Drift

# Observability for AI Agents: Tracking Decision Logs, Cost Ceilings, and Token Drift

*What an SRE learns when the thing on call is a stochastic loop — and how to make agentic AI operable, budgetable, and auditable.*


## 1. The Black Box Problem: Why “It Worked in a Demo” Isn’t Enough

Every agent demo has the same shape: a prompt goes in, the model picks a tool, the tool returns, the model answers — and the room applauds. A demo is a single golden path, executed once, by the people who built it. It proves the model *can* do the task. It proves almost nothing about whether the system should run unattended against production data.

Production is the tail: the ambiguous request at 2 a.m., the tool that times out on the third retry, the user who phrases a refund request as a lawsuit threat. And an agent fails in ways a deterministic API never does. A classic service fails with a 5xx and a stack trace. An agent fails by doing the wrong thing confidently, or doing the right thing seventeen times in a loop, or burning $40 of tokens deciding it is confused. **The failure modes of an agentic system are not cheaper than its success modes — they are more expensive.**

That is the black box problem. Traditional observability assumes a request is reconstructible: deterministic code path, logs, a trace. Agents break that assumption three ways at once:

– **The input is underspecified.** Between the prompt and the answer, the agent made choices — which tool, which arguments, which plan — that exist nowhere in the request.
– **The code path is sampled, not fixed.** Two runs of the same prompt can take entirely different routes. “What did the system do?” has a per-run answer, not a single one.
– **Failure is behavioral, not technical.** No exception fires when an agent hallucinates an order status. The system returns 200 OK. It was wrong in a way only a downstream human notices.

“It worked in a demo” is how you *start* an agent project, not how you run one. The question that separates a prototype from a platform is not “can the model do it?” but **”can we see what it did, bound what it costs, and prove it still works next quarter?”** That is an observability problem — and the subject of this post.

## 2. Decision Logging: Replaying Agentic Choices for Full Transparency

If the problem is invisible behavior, the fix is not transcript logging — it is **decision logging**: capturing every meaningful choice in structured form so a run can be replayed like a trace.

The mental model comes from distributed tracing. A microservice emits a span per downstream call; an agent is the same shape with a nondeterministic planner on top. Each plan-act-reflect iteration is a span, and the *decision* is its payload. Every step should emit an event roughly like this:

“`json
{
“trace_id”: “8f3a1c…”,
“step”: 4,
“agent”: “order-status-closer”,
“model”: “claude-sonnet-4-20260514”,
“decision”: “call_tool”,
“tool”: “get_order_status”,
“rationale”: “customer asked for ETA; order id present”,
“prompt_version”: “v27”,
“tokens”: { “input”: 1840, “output”: 96 },
“latency_ms”: 842,
“cost_usd”: 0.0142,
“ts”: “2026-09-06T04:12:31.002Z”
}
“`

This buys what a transcript cannot. **Replayability:** with a trace ID, an engineer reconstructs why an agent escalated a customer or opened a case — weeks later. “Why did this happen?” becomes a query against structured logs, not a meeting. **Comparability:** because decisions are structured, you can diff run 4 against run 12 of the same request type and see where paths diverged — which is how you debug a regression that only shows on Tuesdays. **Auditability:** when an agent touches customer data or takes a side-effecting action, decision logs are the control — who authorized the run, which policy version was active, which tool fired, what it returned — without exposing raw transcripts to every auditor. This is why production Model Context Protocol (MCP) servers ship audit logs as a first-class feature: typed tools and auditable transports are the contract layer that makes decision logging possible instead of aspirational.

Two rules keep it honest. **Log the decision, not just the outcome** — including refusals and skipped tools; if you only log successful calls, the one event you need in the postmortem is the one you never recorded. **Keep payloads out of the hot path** — store full tool inputs and outputs in object storage keyed by trace ID; replay a run on demand rather than paying to store every token forever.

## 3. Cost Ceilings: Preventing the $1,000 Recursive Loop

Every team that has run an agent in production for more than a month has a recursive-loop story. The agent tries a tool, the tool errors, it retries with different arguments, the tool errors again — and because each failure looks like progress to the model, the loop never terminates on its own. Multiply a stuck loop by a pricey model by a growing context window, and one conversation quietly accrues a bill that outranks your entire CI fleet.

The math is the trap. Per-token pricing feels cheap, so teams reason in pennies and discover the real cost is **steps × tokens × price**, with every iteration feeding its own output back into an ever-longer context. A run that should have been three steps and $0.05 becomes forty steps and $12 — confidently failing the whole time. The $1,000 recursive loop needs no malice; it needs one tool with ambiguous error semantics and no ceiling.

Treat cost as a **hard resource constraint enforced at runtime**, the way you enforce memory limits on a container — not a line item discovered in the billing export. Production agents need ceilings at every layer:

– **Per-step budget** — hard token and wall-clock caps on any single model call.
– **Per-run ceiling** — max iterations (8–15 for well-scoped tasks) and a dollar cap per conversation. Exceeding either terminates *gracefully*: a structured handoff to a human, not a silent 500.
– **Per-tenant and org budgets** — one confused tenant cannot burn the shared quota; daily and monthly caps with alerts at 50%, 80%, and 100%.
– **Retry discipline** — repeated tool failures trigger backoff, then a circuit breaker that stops calling that tool for the run. A loop that failed three times will not succeed on the thirtieth.

The architectural principle: **check before you spend, not after.** Decrement the run’s cost ledger *before* each tool call; if the step would exceed the remaining budget, refuse and escalate. And pair ceilings with human checkpoints on anything irreversible — a refund, a record update, an email. The ceiling prevents the runaway; the checkpoint prevents the regret. An unbounded agent loop is just a process you forgot to limit.

## 4. Token Drift: Monitoring Accuracy and Model Performance Over Time

The observability failure that surprises even teams with solid logging and ironclad budgets: **the system silently gets worse, and nobody changed anything.** The agent that resolved 92% of order-status tickets in March resolves 84% in August. No code changed, no prompt changed. What drifted?

Token drift is the umbrella term for the slow, silent shifts that degrade LLM systems without a deploy event:

– **Input drift** — what users ask changes over time: new products, new edge cases, seasonal volume. The request distribution you eval’d in March is not August’s.
– **Output drift** — model behavior changes under you. Providers ship weight updates, deprecate versions, or change routing; your prompt was tuned against a snapshot that no longer exists.
– **Context drift** — the systems the agent reads drift too: schema changes, new status codes, data-quality rot in the CRM — tools return shapes the planner was never trained on.

Drift is invisible to classic SLOs: error rates stay flat, latency stays flat. The agent is *succeeding* at slightly the wrong thing, refusing slightly more often, taking nine steps where it used to take five. The symptoms hide in distributions, not averages.

The fix is a **golden eval set run on a schedule** — regression testing applied continuously. Keep 100–300 labeled production-like tasks with known-good answers, and run the agent against them nightly and on every intended change. Track completion rate, accuracy, and steps-to-success over time: a completion chart trending down while nothing else changed *is* the drift alarm. Gate every change on it — a model upgrade, prompt tweak, or provider switch that underperforms the incumbent does not ship, no matter how good the vendor’s benchmark card looks. And watch the distributions — output-length histograms, per-tool call frequency, refusal rates — because when those shift, the drift started before the score moved. Teams that skip this discover drift through escalated tickets. Teams that run it catch drift while it is still a line on a chart.

## 5. Dashboards for AI: Visualizing Accuracy, Latency, and ROI

A dashboard is the operating surface where an on-call engineer decides whether to page someone, a product owner decides whether to ship, and a CFO decides whether to renew. Agentic AI needs panels for all three.

**For the SRE on call**, the agent is a service with unusual failure modes: request volume, p50/p95 step latency, error and escalation rates, and a step-count distribution — a fat tail past ten steps is an incident, not a statistic. Watch per-model token consumption and budget burn against ceilings; alert on p95 latency, escalation spikes, and every ceiling hit, because ceiling hits are the signature of a loop gone wrong.

**For the product owner**, one question matters: *is it working?* Task-completion rate by intent, golden-set accuracy, escalation rate, steps-per-success. If completion drops or steps climb, quality is degrading before customers complain.

**For finance**, the dashboard is an ROI statement: cost per resolved task, cost per deflection versus the human-handling cost it replaces, monthly burn by model and tenant.

| Row | Audience | Panels |
|—|—|—|
| 1 | SRE | Request volume, p50/p95 step latency, error rate, ceiling hits |
| 2 | Product | Completion rate by intent, golden-set accuracy, escalation rate, steps-per-success |
| 3 | Finance | Cost per resolved task, $/request by model, budget burn vs. ceiling |
| 4 | Everyone | Model version mix, token drift indicators, decision-log anomalies |

Underneath it all: **an error budget with consequences.** If the agent’s SLO is 95% task completion on the golden set, a sustained breach triggers the same machinery as an availability breach — a review, a documented decision, a remediation plan. A dashboard with no alerting is a museum exhibit. OnlineCloud’s delivery philosophy is blunt on this: *we instrument outcomes — accuracy, latency, dollars per request — and share the dashboards.* Not slideware. The actual metric is whether the system is measurably cheaper and more accurate than the human process it replaced — this week, not launch week.

## 6. The 6–10 Week Operating Model: Ownership After Launch

Observability is not a phase that ends at go-live; it is what makes go-live survivable. Teams fail at agentic AI not during the build but in the first month of operations, when the builders have moved on and nobody owns the drift charts, the eval runs, or the cost alerts. The operating model has to be designed up front, with named ownership after launch.

A sane cadence for a well-scoped agent compresses to 6–10 weeks precisely because observability is built in from week one, not bolted on in week nine:

– **Week 1 — discovery.** Map the systems, pick the model class, and write the eval that defines “done.” If you cannot define correct in week one, you cannot measure drift in week twelve.
– **Weeks 2–7 — vertical slices.** Ship one usable slice every one to two weeks against real data — each instrumented with decision logging, cost ceilings, and its own dashboard from day one in front of users.
– **Weeks 8–10 — harden and hand over.** Load-test the failure paths, tune the ceilings, run the golden set, stand up the on-call rotation and runbooks. Launch is the *start* of operations, not the end of the project.

Then the model that keeps it honest after launch — and where most vendor relationships end and OnlineCloud’s begins: the team that built the system owns the dashboards, the retraining cadence, and the on-call. Concretely: a **weekly operating review** of drift charts, ceiling hits, escalations, and a sample of decision logs; a **model review board** — no prompt or model swap ships without passing the golden set; a **monthly cost review** reconciling burn against the ROI panels; and a real **on-call path** — when the completion chart trends down, someone with authority to roll back a prompt or escalate a model issue gets paged, not an email alias read on Tuesday.

The 6–10 week promise is not speed for its own sake. It is the consequence of instrumenting from day one: when the eval exists before the code, when decision logs ship with the first slice, when ceilings are in the architecture rather than the incident report — launch becomes an increment, not a cliff. Ownership after launch is what turns a shipped agent into a *trusted* one that gets sharper month over month instead of rotting quietly in a corner of the org chart.

## Conclusion: Ship Agents You Can See, Bound, and Prove

These three disciplines are one stance applied to the three things an agent can do that a normal service cannot: **decide, spend, and change.** Decision logging makes the deciding visible and replayable. Cost ceilings make the spending bounded. Drift monitoring makes the changing measurable and gateable. Dashboards and a real operating model turn all three into a routine that survives contact with production.

If you are standing up agentic AI against real systems — Oracle, MS SQL, Postgres, your CRM — and you want it to do more than impress a demo audience, talk to the OnlineCloud engineers. We scope the data, pick the model class, and write the eval that defines “done” in a one-week discovery sprint; we ship instrumented vertical slices every one to two weeks; and after launch we own the dashboards, the drift charts, and the on-call. That is the difference between a black box and a system your operations team can sleep through. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) to start the conversation — and bring the workload you are tired of babysitting.