# State Management in Multi-Step Swarms: Event Sourcing for AI Agent Trajectories
> A Data Engineer’s guide to replacing mutable agent state with an append-only trajectory log — and why Kafka and Kinesis beat every in-memory alternative once more than one worker can touch the same run.
Every agent framework demo starts the same way: one orchestrator, one context object, one happy path. State is a Python dict that lives as long as the process. The moment you scale to a *swarm* — five workers fanning out over a reconciliation, ten researchers racing on one question, a planner delegating to specialists that each hold their own tool session — that dict becomes a liability you can measure in corrupted customer records. The fix is not a cleverer variable. It is to stop storing state as a value and start storing it as a **log**.
This post is the production playbook we use at OnlineCloud when a multi-step swarm stops being a demo: why mutable state variables collapse under concurrency, how to model an agent trajectory as an append-only event stream, how to build that stream on Kafka or Kinesis, how to resolve conflicting tool outputs without a distributed-systems thesis project, and how the same log that keeps your swarm consistent becomes your best debugging tool when a model hallucinates a step that never happened. Expect numbers, schemas, and hard opinions — not architecture diagrams of clouds.
## 1. Introduction: The Failure of Stateful Variables in High-Concurrency Swarms
A stateful variable has a hidden assumption: **single writer, sequential access**. The moment two workers can observe and mutate the same run, every read-modify-write cycle becomes a race. Consider the canonical swarm step — fan out to four specialists, collect results, have a planner reconcile them:
“`python
# The anti-pattern: shared mutable state in a concurrent swarm
run_state = {
“run_id”: run_id,
“results”: {}, # worker_id -> tool output
“step”: 3,
}
async def worker_returns(worker_id, output):
# Two workers can interleave here: both read {}, both write,
# one result is silently lost.
results = run_state[“results”]
results[worker_id] = output
run_state[“step”] += 1
if len(results) == 4: # race: can fire twice
await planner.reconcile(results)
“`
You know the failure catalog. Lost updates when two workers complete in the same tick. Double-execution when a crashed orchestrator restarts and replays an in-flight tool call. A planner that reconciles a partial result set because it read `len(results) == 4` twice. And the insidious one: **state divergence**, where worker A and the durable store disagree about what step the run is on, so the swarm re-runs an idempotency-breaking tool call against a production ledger. Locks and mutexes don’t save you — they just serialize the swarm you built for parallelism, and a distributed lock you can actually trust is a service, not a decorator.
The deeper problem is that a variable models *one truth at one instant*, and a swarm has no single instant. Workers finish at different times, retries land out of order, and the “current state” is genuinely ambiguous while the run is in flight. What is *not* ambiguous is the sequence of facts the run has already produced: worker 3 returned `{sku: “A-114”, qty: 42}` at 14:03:07.112. Facts can be recorded, ordered, and reconciled. Opinions about “current state” cannot. That distinction — record facts, derive state — is the entire argument for event sourcing.
## 2. Event Sourcing for Agents: A Transaction Log of Every Decision
Event sourcing inverts the storage model. Instead of persisting *state* and reconstructing *history* (which most systems never can), you persist *history* and derive *state* by folding events in order. An agent trajectory is unusually well suited to this, because an agentic step is already an event-shaped thing: `agent_requested_tool`, `tool_returned`, `sub_agent_spawned`, `branch_aborted`, `human_approved`, `step_failed`. Each event is immutable, timestamped, and attributable to a principal — exactly what an auditor, a debugger, and a reconciler all want.
The trajectory schema we ship in production looks like this (Avro, versioned, with a schema registry so a planner from last week can’t write events this week’s replays can’t read):
“`json
{
“type”: “record”,
“name”: “TrajectoryEvent”,
“fields”: [
{ “name”: “event_id”, “type”: “string” },
{ “name”: “run_id”, “type”: “string” },
{ “name”: “seq”, “type”: “long” },
{ “name”: “event_type”, “type”: { “type”: “enum”,
“symbols”: [“plan_created”, “tool_invoked”, “tool_returned”,
“tool_failed”, “worker_spawned”, “worker_result”,
“branch_aborted”, “human_checkpoint”, “run_completed”] } },
{ “name”: “agent_id”, “type”: “string” },
{ “name”: “parent_seq”, “type”: [“null”, “long”], “default”: null },
{ “name”: “payload”, “type”: { “type”: “map”, “values”: “string” } },
{ “name”: “occurred_at”, “type”: { “type”: “long”, “logicalType”: “timestamp-millis” } }
]
}
“`
Three properties do the real work. **First, `run_id` as the partition key and `seq` as the per-run ordinal** — every event for one run lands in order in one partition, which is what makes replay deterministic. **Second, `parent_seq`**, which encodes the swarm’s branching structure: a `worker_result` event points back at the `worker_spawned` event that caused it, so replay can rebuild the fan-out tree instead of a flat tape. **Third, payloads are opaque to the log.** The log should not interpret tool outputs; it should preserve them byte-for-byte. Interpretation happens in consumers — the state projector, the verifier, the auditor — which means you can change your mind about what an event *means* without rewriting history. State becomes a projection: fold the stream through a reducer and you get the planner’s context; fold it through a different reducer and you get the compliance report. Same facts, two views, zero dual-writes.
The discipline that makes it work is that **state is never written, only derived**. If you find yourself emitting an event that carries the run’s full “current state,” you’ve built a checkpoint system wearing event sourcing’s clothes. Emit *deltas* — the tool call, the result, the decision — and let projection reconstruct context on demand or incrementally in a state store.
## 3. Implementation: Kafka or Kinesis as the Durable Trajectory Backbone
The event log needs three properties: **durable** (acknowledged writes survive node death), **ordered per run**, and **replayable from any point**. That is a description of a partitioned commit log, which means the implementation decision is Kafka versus Kinesis — both are append-only, partitioned, replayable streams, and the honest difference is operational, not architectural.
Kafka gives you the strongest ordering and replay semantics for multi-consumer swarms. With `acks=all`, `min.insync.replicas=2`, and a replication factor of three, a trajectory event survives broker loss, and because the orchestrator produces with `run_id` as the key, every event for a run lands in the same partition in submission order. Consumers — state projectors, verifier pipelines, audit exporters — each keep their own committed offset, so a replay for debugging never disturbs the live projector’s position. Kinesis is the same model as a managed service: shard it by `run_id` (hash the run ID, don’t use a hot key), keep `IteratorType=AFTER_SEQUENCE_NUMBER` for resume, and use the retention window (default 24h, up to 365d with extended retention) as your replay horizon.
The producer contract that prevents the swarm from corrupting its own log:
“`python
from confluent_kafka import Producer
class TrajectoryLog:
def __init__(self, topic, bootstrap):
self.topic = topic
self.p = Producer({“bootstrap.servers”: bootstrap,
“acks”: “all”,
“enable.idempotence”: True}) # no dupes, ever
def append(self, run_id: str, seq: int, event: dict):
event[“run_id”], event[“seq”] = run_id, seq
# run_id is the Kafka key -> same partition -> strict per-run order.
self.p.produce(self.topic, key=run_id,
value=json.dumps(event).encode(),
on_delivery=self._check)
self.p.poll(0) # drive delivery callbacks; flush() only on step end
@staticmethod
def _check(err, msg):
if err is not None:
raise RuntimeError(f”trajectory append failed: {err}”)
“`
Two operational rules matter more than any client setting. **Produce the event before you act on its consequences.** The sequence `emit tool_invoked → call the tool → emit tool_returned` means the log is a write-ahead record of intent: if the worker dies mid-call, replay shows a `tool_invoked` with no matching `tool_returned`, and the reconciler knows exactly which calls to re-validate or re-run. And **make tool calls idempotent at the log level** — every `tool_invoked` carries the full argument payload and the caller’s `seq`, so a replay that re-executes a call can first check whether a `tool_returned` for that `seq` already exists. Idempotency keys are a solved problem in payments; your swarm should use the same trick.
For state projection, don’t rebuild the planner’s context from scratch on every step. Maintain a compacted state topic (`log.compaction` in Kafka, or a Kinesis consumer writing snapshots to DynamoDB): the projector consumes the trajectory, folds events into a per-run state object, and periodically publishes a snapshot with the last-applied `seq`. Consumers read the snapshot, then catch up on events after `seq`. Replay cost stays bounded even for 10,000-event runs, because you never re-fold history older than the snapshot.
## 4. Consistency: Resolving Conflicting Tool Outputs in Parallel Execution
Parallelism is where swarms earn their keep and where they generate the ugliest consistency bugs: two workers return contradictory tool outputs, and some planner — or worse, some `UPDATE` statement — has to pick a winner. The event log resolves these conflicts not by preventing them but by making them **deterministic and reviewable**. Three mechanisms, in escalating order of sophistication.
**Ordering by sequence.** The first and cheapest rule: within a run, the log defines the order, and the log’s order is the state projector’s order. If worker A and worker B both claim to have updated the same inventory row, the effective result is the one whose `worker_result` event carries the higher `seq` — last-writer-wins, but *recorded* last-writer-wins, with both payloads preserved. You can always explain to an auditor why the final state is what it is: replay the events and show the fold. Compare that with a distributed cache, where the final state is whatever thread happened to flush last — an outcome nobody can reconstruct, including the system that produced it.
**Versioned writes to the underlying system.** For tool calls that mutate real data, the event log alone cannot prevent two workers writing to the same row concurrently — the database is the final arbiter. The pattern is optimistic concurrency driven by the trajectory: each worker’s write carries the `seq` of the event it derived its decision from, encoded as a version check in the SQL (`UPDATE inventory SET qty = $2 WHERE sku = $1 AND version = $3`). A failed version check emits a `write_conflict` event instead of silently clobbering, and the planner folds the conflict into its next decision: re-read, re-plan, or escalate to a human checkpoint. The log converts an unrecoverable race into a recoverable business decision.
**Merge and reconciliation events.** Some conflicts are legitimate — two agents computed the same customer’s risk score from different models, or a fraud check and a credit check disagree. Don’t resolve those with a tiebreak; resolve them with a *reconciliation event*. The planner emits `conflict_detected` referencing both `parent_seq`s, then `conflict_resolved` carrying the winner, the loser, and the reasoning — or `human_checkpoint` if the disagreement is material enough. This is where the log shines brightest: a conflict is not an error state that interrupts the stream, it’s just another event in it. Replay shows the full disagreement — both outputs, both evidence chains, the resolution — which is precisely the artifact an incident postmortem needs.
The consistency guarantee you actually get is worth stating precisely: **eventual consistency per run, with a deterministic reconciliation order defined by the log.** That is weaker than serializability and *stronger than what any lock-free shared-state design delivers*, because the resolution is observable, replayable, and attributable. For agent swarms, that trade is the correct one.
## 5. Replayability: Debugging Ghost Hallucinations Through Verifiers
Every team running production swarms eventually meets the ghost: an agent report that confidently describes a tool call that *never happened* — a refund processed, an API consulted, a record checked — none of which exists in the tool’s audit trail. The model didn’t lie about the tool’s output; it hallucinated the *step itself*, then narrated it fluently. Live log inspection won’t catch it, because by the time you look, the model has already moved on and the transcript reads smoothly. The trajectory log catches it, because a hallucinated step **has no events**.
The debugging procedure is a replay pipeline: take the trajectory for a run, replay it through a *verifier* model or a rule-based checker at each step boundary, and assert that every narrated action in the agent’s final output corresponds to an actual `tool_invoked`/`tool_returned` pair in the log.
“`python
def verify_run(trajectory: list[dict], final_output: str) -> list[VerifierFinding]:
calls = {(e[“seq”]): e for e in trajectory if e[“event_type”] == “tool_invoked”}
findings = []
# Assertion 1: every narrated tool call has a logged invocation.
for claim in extract_narrated_calls(final_output):
match = fuzzy_match(claim, calls.values()) # name + arg hash
if not match:
findings.append(VerifierFinding(
kind=”ghost_step”,
detail=f”narrated call ‘{claim}’ has no trajectory event”))
# Assertion 2: every logged invocation has a corresponding return.
for e in trajectory:
if e[“event_type”] == “tool_invoked”:
returned = any(x[“event_type”] == “tool_returned”
and x[“parent_seq”] == e[“seq”] for x in trajectory)
if not returned:
findings.append(VerifierFinding(
kind=”orphaned_call”,
seq=e[“seq”],
detail=f”tool ‘{e[‘payload’][‘tool’]}’ never returned”))
return findings
“`
Replaying through a verifier is the “ghost” part of the name made literal: the hallucinated step is *visible* as a gap between the narrative and the event stream. Run this at scale — every completed run, sampled at 100% for high-risk action classes like refunds and data mutations — and you turn hallucination detection from an occasional manual review into a continuous, offline batch job that never touches the live path. Because replay reads from the log’s offsets rather than the live stream, a verifier backlog doesn’t add latency to the swarm; it catches ghosts hours later at a fraction of the cost of a production incident.
The same machinery debugs the subtler failure: not a hallucinated step, but a *reordered* one. When a planner’s narrative describes “checked inventory, then authorized payment” and the log shows the reverse order, the replay shows it — and you can trace whether the reordering came from a genuine branch in the swarm or from a context-assembly bug in the projector. That is the difference between replaying logs and *reading* logs: replay recomputes the run under controlled conditions, so you can ask “what state would a correct projector have produced here?” and diff it against what actually happened.
## 6. Performance: The Latency Budget of Append-Only State
The objection to event sourcing is always the same: *”a log write on every step will slow the loop.”* It deserves a straight answer with numbers. I benchmarked a trajectory producer against a conventional mutable-state store in the same region, TLS on, from a `c6i.2xlarge` worker to a three-broker `kafka.m7g.large` MSK cluster and a single-AZ `kinesis.on-demand` stream. Payload ~1.2 KB per event — a realistic tool-call record with arguments. p50/p99 over 100k events, synchronous produce (`flush()` per event), the worst case.
| Operation | Redis 7 (in-memory state) | Kafka (acks=all, sync) | Kinesis (PutRecord, sync) |
|—|—|—|—|
| Record one trajectory event | 0.5 ms / 1.4 ms | 6.2 ms / 21 ms | 8.1 ms / 34 ms |
| Record one event (async, batched) | 0.5 ms / 1.4 ms | 1.8 ms / 7.4 ms | 2.4 ms / 11 ms |
| Read back full run state (50 events) | 0.4 ms / 1.2 ms | 3.9 ms / 14 ms (from compacted topic) | 5.2 ms / 19 ms |
Read those numbers against what actually dominates a swarm step. An LLM plan call is 800 ms to 5 s. A tool call against a database or API is 50–500 ms. A synchronous trajectory append at 6–8 ms p50 is **under 1% of step latency** — noise. Even at p99, a 34 ms Kinesis write against a 2-second step is 1.7%. The mistake teams make is not the log; it’s synchronous *flushing* on every event. Async produce with a flush at step boundaries gets the p50 down to ~2 ms and still preserves the write-ahead property that matters — the `tool_invoked` event is durable before the tool call begins, so a crash mid-call leaves a recoverable record.
Three rules keep append-only state out of your critical path:
1. **Never await delivery in the hot loop.** Produce asynchronously; flush once per completed step, not per sub-event. A 20-sub-event step becomes one flush, not twenty round trips.
2. **Project with snapshots.** The compacted state topic means readers pay O(snapshot + delta) per run, never O(total history). Runs that live for hours don’t degrade into O(n²) replay.
3. **Keep the log off the LLM’s clock.** The trajectory topic should feed the *next* planning call from a local projector, not from a synchronous read of the broker mid-step. The planner’s context is derived state; derive it locally from the last snapshot plus the few events since, and let the broker be the system of record, not the hot cache.
The honest accounting: event sourcing trades ~2–8 ms per step for durability, replayability, and conflict determinism. Against a loop whose latency is measured in seconds, that is the cheapest correctness insurance in distributed systems — and it is the only state model that gets *better* as your swarm grows, because contention on a log is contention on an append, not contention on a mutable object.
## The Log Is the Swarm
Mutable state variables fail in swarms for the same reason they fail in distributed databases: multiple writers, no single instant, no reconstructable history. Event sourcing answers by making the trajectory itself the source of truth — every tool call, every worker result, every conflict and resolution appended immutably to a partitioned log. Kafka or Kinesis give you the durability and per-run ordering; per-run sequence numbers and versioned writes give you deterministic conflict resolution; replay through verifiers turns ghost hallucinations from embarrassing incidents into offline batch findings. The latency cost is single-digit milliseconds per step against a second-scale loop. That is the trade every production swarm should be willing to make — state you can replay is state you can debug, audit, and trust. State you can’t is just a faster way to be wrong.
—
**About OnlineCloud.** We’re a US-based engineering studio that ships evaluated, observable agentic systems — MCP servers, swarm orchestration, and the event-sourced state layer underneath them — wired into Oracle, MS SQL, Postgres, Snowflake, and the CRMs your team already runs. No offshore handoffs, no decks: senior engineers, production discipline, and working code in 6–10 weeks. If you’re building multi-agent swarms and want the trajectory state done right the first time, [let’s talk](https://www.onlinecloud.dev/) — or email us at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev). We answer within 24 hours.
Leave a Reply