# Orchestrating Multi-Agent Swarms with Kafka: Event-Driven Reasoners
*At [OnlineCloud](https://www.onlinecloud.dev/), we build agentic systems on top of the boring, battle-tested infrastructure that already runs the world’s transaction pipelines. This post is a field note on replacing sequential agent loops with Kafka-backed reasoning swarms.*
—
## 1. From Sequential Loops to Event-Driven Swarms
The default mental model for an “agent” is a `while` loop. You hold a conversation history, you call a model, it emits a tool call, you execute the tool, you append the result, you loop. It is a single-threaded state machine with one very expensive `await`. It works beautifully for demos and falls over the moment you need concurrency, durability, or horizontal scale.
The problem is that a sequential loop conflates three things that deserve to be separated: **reasoning**, **tool execution**, and **state**. When they live inside one process, you get a system whose throughput is capped by its slowest tool call, whose state evaporates when the pod restarts, and whose retries are re-entrant nightmare fuel because the only way to “resume” is to replay the entire prompt.
An **event-driven swarm** breaks that monolith apart. Instead of one agent holding a loop, you deploy many narrow reasoners — planners, researchers, validators, executors — each subscribing to the events it cares about and publishing the events it produces. The loop does not disappear; it is *externalized* into the log. Kafka becomes the coordination substrate: the place where intent is announced, where work is claimed, and where the durable record of everything that happened lives.
The shift is philosophically the same one that took monoliths to microservices a decade ago. You trade local, in-process clarity for distributed, observable, replayable coordination. And just like microservices, the win only materializes if you get the topology and the semantics right.
## 2. Swarm Topology: Kafka Topics as Agent Reasoning Channels
The first design decision is the contract: **what is a topic, and what is an agent?** Our rule is simple. A topic represents a *class of reasoning event*, not a class of worker. Agents subscribe to topics; they do not message each other directly. This decouples producer from consumer and makes the whole topology introspectable with a single `kafka-topics –describe`.
A production swarm typically converges on five families of topics:
– **`swarm.tasks`** — the ingress log. A request arrives (a customer order, a support ticket, a compliance case) and is decomposed by a *planner agent* into a set of subgoals, each emitted as a task event keyed by `correlation_id`.
– **`swarm.reasoning.
– **`swarm.tool.calls` / `swarm.tool.results`** — the execution boundary. Tool-calling agents (the MCP servers we build for clients) consume call events and emit results, keeping side effects in a dedicated, auditable stream.
– **`swarm.critique`** — the adversarial channel. Validator agents subscribe to hypotheses and emit critiques, scores, or vetoes. This is where self-consistency and debate patterns live.
– **`swarm.state`** — a **compacted** topic. The latest known state for every `swarm_id`, replayed on demand to reconstruct where a multi-agent run currently stands.
Two invariants make this topology safe. First, **events are immutable and named in the past tense** (`TaskPlanned`, `HypothesisProposed`, `ToolInvoked`, `CritiqueIssued`) — an event log is a ledger of facts, not a command bus. Second, **ordering is per-key, never global**. Every event carries a partition key of `swarm_id` (or `task_id` where finer granularity is needed), so Kafka guarantees that all reasoning about the same unit of work is serialized within a partition while the swarm scales horizontally across partitions. Global ordering is a scalability trap; per-key ordering is the whole game.
Because agents are stateless consumers, you can scale a swarm by simply adding partitions and consumers. A slow `risk` agent does not block `pricing` reasoning — it creates *consumer lag*, which is a metric you can see and alarm on, rather than a hang you have to debug.
## 3. Implementation: Managing Multi-Agent State Transitions with Kafka Streams
Raw producers and consumers give you transport. **Kafka Streams** gives you the state machine. The interesting problem in any swarm is not “how do I send a message” — it is “how do I know that a task has moved from `planned` to `executing` to `converged`, reliably, exactly once, across many agents.”
We model each swarm as a topology of state transitions. The DSL reads like a declarative version of the loop we removed:
“`java
StreamsBuilder builder = new StreamsBuilder();
KStream
builder.stream(“swarm.tasks”, Consumed.with(Serdes.String(), swarmSerde));
// Fold every event for a swarm into its current lifecycle state.
KTable
.groupByKey(Grouped.with(Serdes.String(), swarmSerde))
.aggregate(
SwarmState::initial,
(swarmId, event, prior) -> prior.transition(event), // pure reducer
Materialized.
.withLoggingEnabled(Map.of(“cleanup.policy”, “compact”))
.withCachingDisabled())
.toStream();
// Route each transition to the channel the next reasoner watches.
state.filter((k, s) -> s.phase() == Phase.NEEDS_RESEARCH)
.mapValues(SwarmState::toTaskEvent)
.to(“swarm.reasoning.research”);
state.filter((k, s) -> s.phase() == Phase.NEEDS_CRITIQUE)
.mapValues(SwarmState::toHypothesisBatch)
.to(“swarm.critique”);
state.filter((k, s) -> s.phase() == Phase.CONVERGED)
.mapValues(SwarmState::toResult)
.to(“swarm.results”);
“`
The **aggregate is a pure reducer**: `prior.transition(event)` is a deterministic function of the previous state and the incoming fact. That is the operational superpower. Because the state store is changelog-backed and the function is pure, any swarm can be reconstructed by replaying its events — and any bug can be reproduced by replaying the exact log that caused it. In an LLM system, where non-determinism is everywhere, having a *deterministic shell around the non-deterministic core* is worth more than almost any other architectural decision.
Practical notes learned the hard way:
– **Idempotency keys are mandatory.** Model calls are not idempotent, so each tool invocation carries a `tool_call_id`; the executor dedupes against a windowed store. At-least-once delivery is a fact of life; exactly-once is a fiction you enforce at the edge.
– **Use interactive queries for the live view.** A dashboard asks the state store directly (`state.queryableStore(“swarm-state-store”).get(swarmId)`) instead of replaying the log — this is how you get a real-time “swarm monitor” without a separate database.
– **Bound the state.** TTL the state store per swarm; a stuck swarm should expire into `Phase.ABANDONED` rather than leak forever.
– **Keep transitions in the graph, not in the agents.** If an agent decides the next phase, you have re-created the monolith with worse ergonomics. The topology owns transitions; agents only produce facts.
## 4. Consistency: Resolving Conflicting Tool Outputs in Parallel Swarms
The moment you fan out, you fight for consistency. Ten research agents hit ten sources; three return different prices; two disagree on availability. This is the classic distributed-systems problem of reconciling concurrent writes, except the “writes” are probabilistic model outputs.
We layer three resolution strategies, from cheapest to most expensive:
**Deterministic merge (last-writer-wins with causality).** For facts that are genuinely single-valued — a stock count, an exchange rate — we pick a *winner by rule*, not by arrival time. Each tool result carries `(source_rank, observed_at, confidence)`. The reducer resolves ties lexicographically on that tuple. This is a vector-clock-flavored approach: we do not care *when* the message arrived, only the causal ordering of the observations it represents. Arrival order is a network artifact and must never be the tiebreaker.
**Quorum / voting.** For categorical outputs (a classification, a routing decision), we run redundant reasoners and take the majority, with a confidence-weighted tiebreak. Cheap, embarrassingly parallel, and it lifts reliability dramatically for high-variance classification. The `swarm.critique` topic is where the dissenting votes land so they remain inspectable rather than silently dropped.
**Adjudication (a designated resolver agent).** When conflicts are semantic rather than numeric — “is this transaction suspicious?” — no rule resolves them. We route the conflict to a single adjudicator agent, keyed by `swarm_id` so exactly one adjudicator sees each conflict. The adjudicator sees *all* competing hypotheses plus their provenance and emits a `ConflictResolved` event that flows back into the state machine.
The critical design principle is **conflicts are first-class events, not exceptions**. A naive system hides the conflict by having one agent overwrite another. A swarm treats the disagreement as data: it lands in `swarm.state` with full provenance, it is replayable, and it is measurable. The metric that matters here is the *conflict rate* per task type — when it climbs, your downstream tools have drifted or your prompts have regressed, and you find out before a customer does.
## 5. Performance: Async Swarms vs. Sequential Agent Reasoning
Sequential loops and async swarms differ in exactly one variable: **where the latency lives**. In a sequential loop, per-request latency is the *sum* of every step. In a swarm, it is the *longest dependency chain* plus coordination overhead. If a task has a fan-out of `N` independent tool calls, the sequential loop pays `Σtᵢ` while the swarm pays `max(tᵢ) + ε`.
We benchmarked a representative fulfillment task — validate inventory, price, check risk, reserve logistics — against both architectures on identical models and tools:
| Metric | Sequential loop | Kafka swarm | Delta |
|—|—|—|—|
| p50 end-to-end latency | 8.4 s | 3.0 s | **−64%** |
| p99 end-to-end latency | 27.1 s | 6.4 s | **−76%** |
| Throughput (tasks/s) | 3.1 | 22.0 | **+610%** |
| Failure blast radius | whole run | single partition key | contained |
| Resume after crash | replay whole prompt | replay from offset | O(events) |
Three findings are worth dwelling on.
First, **the swarm wins hardest at p99, not p50.** Sequential loops compound variance: every step’s tail latency adds to the total. Swarms run steps in parallel, so the tail is the tail of the *slowest* step, not the *sum* of all tails. If your SLO is a p99 and not a mean, the swarm is not a 2× win — it is an order-of-magnitude win.
Second, **coordination overhead is real but fixed.** Kafka produce/consume round-trips cost single-digit milliseconds. Against model inference at hundreds of milliseconds to seconds, the overhead is noise. The swarm is slower only when the task is a *strictly serial* dependency chain with no parallelism to exploit — and in that case, neither architecture is fast, you just have less cost to amortize.
Third, **throughput scales with partitions, not agents.** Adding consumers to a swarm scales only up to the partition count of the channel. Size partitions for peak, not for today, because repartitioning a live topic is an operational event you want to avoid. We typically provision 3–4× current peak and let consumer lag be the autoscaling signal.
The honest caveat: async swarms push complexity from the code into the topology. You now own consumer lag, dead-letter queues, partition skew, and idempotency. For a single tool call, a loop is correct. For a system where “reasoning” spans multiple domains, runs concurrently, and must survive restarts, the swarm pays for itself the first time a pod gets evicted mid-run.
## 6. Case Study: Logistics Fulfillment with Event-Driven Reasoning Swarms
A mid-market retailer ran fulfillment decisions through a monolithic sequential agent. Every order triggered one long chain of model calls, and every step had to finish before the next began. Peak-season volume drove p99 past 40 seconds and, worse, a single flaky carrier API would hang an entire order — the agent had no way to proceed, so it retried the *whole* chain.
We rebuilt it as a swarm on Managed Kafka. The planner decomposed each order into four independent reasoning tasks — **inventory**, **pricing**, **risk**, and **logistics** — each emitted to its own domain channel, all keyed by `order_id`. Independent analyst agents consumed them concurrently, calling MCP-wrapped tools against the ERP, the pricing service, and two carrier APIs. A validator agent subscribed to `swarm.critique`, and a single adjudicator — keyed per order — resolved the common case where two carriers disagreed on delivery ETA.
The results, measured over a full quarter including Black Friday:
– **p99 order-to-decision latency dropped from 41 s to 7.2 s**, tracking the slowest carrier call instead of the sum of all steps.
– **Throughput held at 3.4× the prior ceiling** during peak windows, absorbing a 2.1× traffic spike with no added model capacity.
– **Carrier API failures stopped propagating.** A timeout on one carrier now delays only the logistics task; the other three resolve normally, and the adjudicator picks a fallback carrier. The blast radius narrowed from “the order” to “one partition key.”
– **Recovery became trivial.** When a consumer pod was evicted during a deploy, the swarm resumed from the last committed offset. No prompt replay, no lost work.
The unexpected benefit was observability. Because every reasoning step is an event, the team could finally answer questions like *”why did this order route to the expensive carrier?”* by replaying its log. Debugging went from archaeology to a single query against the state store. The swarm did not just make the system faster — it made the system **legible**, which is the property that keeps an agentic system maintainable a year after launch.
—
## The Takeaway
Sequential agent loops are fine until they are not. The moment your reasoning spans multiple domains, needs to run concurrently, or has to survive a restart mid-thought, the loop becomes the bottleneck — and the log becomes the architecture. Treat reasoning as a stream of immutable facts, let Kafka own ordering and durability, let Kafka Streams own the state machine, and treat conflicts as data rather than bugs. That is what separates an agent demo from an agent *system*.
If you are building multi-agent infrastructure and want the topology designed by engineers who have shipped it — **[talk to the team at OnlineCloud →](https://www.onlinecloud.dev/)**. We build MCP-native, observable, event-driven agent systems that plug into the databases and tools you already run, and we ship them to production in weeks, not quarters.
Leave a Reply