Event-Driven Architecture

A comprehensive guide to building systems where components communicate through events rather than direct calls. Event-driven architecture (EDA) decouples producers from consumers, enabling independent scaling, temporal decoupling, and resilience to downstream failures. This skill covers four core pillars: event sourcing (storing state as a sequence of events), CQRS (separating read and write models), message brokers (the transport layer), and eventual consistency (the consistency model that makes it all work). Agents use this skill to design, implement, and troubleshoot event-driven systems at any scale.

---

When to use this skill

Trigger this skill when the user:

• Wants to implement event sourcing for an aggregate or service
• Needs to separate read and write models using CQRS
• Is choosing between Kafka, RabbitMQ, NATS, or other message brokers
• Asks about eventual consistency, compensation, or saga patterns
• Wants to design an event schema or event versioning strategy
• Needs to handle idempotency in event consumers
• Is debugging issues with message ordering, duplicate delivery, or consumer lag
• Asks about domain events, integration events, or event-carried state transfer

Do NOT trigger this skill for:

• Synchronous REST API design without an event component (use api-design)
• General system design questions about load balancers, caches, or CDNs (use system-design)

---

Key principles

1. Events are facts, not requests - An event records something that already happened (OrderPlaced, PaymentReceived). It is immutable. Commands request something to happen (PlaceOrder). Never conflate the two. Events use past tense; commands use imperative.

2. Design for at-least-once delivery - No message broker guarantees exactly-once delivery in all failure scenarios. Design every consumer to be idempotent. Use deduplication keys (event ID + consumer ID) or make operations naturally idempotent (SET over INCREMENT).

3. Own your events, share your contracts - The producing service owns the event schema. Consumers must not dictate what goes in an event. Publish a versioned schema contract (Avro, Protobuf, or JSON Schema) so consumers can evolve independently.

4. Separate the write model from the read model - CQRS lets you optimize writes for consistency and reads for query performance independently. The write side validates business rules; the read side denormalizes for fast lookups. They connect through events.

5. Embrace eventual consistency, but bound it - Eventual consistency is not "maybe consistent." Define SLAs for propagation delay (e.g., "read model updated within 2 seconds of write"). Monitor consumer lag. Alert when the bound is breached.

---

Core concepts

Events are immutable records of state changes. A domain event captures a meaningful business occurrence within a bounded context (OrderPlaced). An integration event crosses context boundaries and should carry only the data consumers need - not the entire aggregate state. Event-carried state transfer includes enough data in the event so consumers never need to call back to the producer.

Event sourcing stores the current state of an entity as a sequence of events rather than a single mutable row. To get current state, replay all events for that aggregate from the event store. Snapshots periodically checkpoint state to avoid replaying the full history. The event store is append-only - never update or delete events. This gives a complete audit trail and enables temporal queries ("what was the order state at 3pm yesterday?").

CQRS (Command Query Responsibility Segregation) splits a service into a command side that handles writes and a query side that handles reads. The command side validates invariants and emits events. The query side subscribes to those events and builds denormalized read models (projections) optimized for specific queries. CQRS does not require event sourcing, and event sourcing does not require CQRS - but they pair naturally because the event log is the bridge between the two sides.

Message brokers are the transport layer. They sit between producers and consumers and handle routing, delivery guarantees, and backpressure. Key broker categories: log-based (Kafka, Redpanda) retain ordered, replayable event logs; queue-based (RabbitMQ, SQS) deliver messages to consumers and remove them after acknowledgment. Choose log-based when you need replay, ordering, and multiple consumer groups. Choose queue-based for simple task distribution and routing flexibility.

Eventual consistency means that after a write, all read replicas and projections will converge to the same state - but not instantly. The gap between write and convergence is the propagation delay. Sagas coordinate multi-service transactions: each step emits an event, and failure triggers compensating events that undo prior steps (e.g., PaymentFailed triggers OrderCancelled). Prefer choreography (services react to events) over orchestration (a central coordinator sends commands) for loosely coupled systems.

---

Common tasks

Implement event sourcing for an aggregate

Store all state changes as events. Rebuild current state by replaying them.

Event store schema (PostgreSQL example):

CREATE TABLE events (
  event_id     UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_id UUID NOT NULL,
  aggregate_type VARCHAR(100) NOT NULL,
  event_type   VARCHAR(100) NOT NULL,
  event_data   JSONB NOT NULL,
  metadata     JSONB DEFAULT '{}',
  version      INTEGER NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (aggregate_id, version)
);

Aggregate reconstruction:

def load_aggregate(aggregate_id: str) -> Order:
    events = event_store.get_events(aggregate_id)
    order = Order()
    for event in events:
        order.apply(event)
    return order

Use the UNIQUE constraint on (aggregate_id, version) for optimistic concurrency. If two commands try to append at the same version, one fails - retry it.

---

Set up CQRS with separate read/write models

The command side validates and persists events. The query side projects events into denormalized views.

Command side: Receives commands, loads aggregate from event store, validates business rules, appends new events.

Query side: Subscribes to event stream, updates read-optimized projections (e.g., a materialized view in PostgreSQL, an Elasticsearch index, or a Redis hash).

Projection example:

class OrderSummaryProjection:
    def handle(self, event):
        if event.type == "OrderPlaced":
            db.upsert("order_summaries", {
                "order_id": event.data["order_id"],
                "customer": event.data["customer_name"],
                "total": event.data["total"],
                "status": "placed"
            })
        elif event.type == "OrderShipped":
            db.update("order_summaries",
                where={"order_id": event.data["order_id"]},
                set={"status": "shipped"})

Keep projections rebuildable. If a projection is corrupted, delete it and replay all events from the store to reconstruct it from scratch.

---

Choose a message broker

Requirement	Recommended broker
Ordered event log with replay	Kafka or Redpanda
Simple task queue with routing	RabbitMQ
Serverless / managed queue	AWS SQS + SNS
Low-latency pub/sub	NATS
Multi-protocol flexibility	RabbitMQ (AMQP, MQTT, STOMP)

Kafka specifics: Topics are partitioned. Order is guaranteed only within a partition. Use the aggregate ID as the partition key to ensure all events for one entity land on the same partition in order. Consumer groups enable parallel consumption - each partition is read by exactly one consumer in a group.

RabbitMQ specifics: Supports direct, fanout, topic, and header exchanges. Use dead-letter exchanges for failed messages. Prefetch count controls how many unacked messages a consumer holds - set it to prevent memory exhaustion.

---

Design a saga for distributed transactions

A saga is a sequence of local transactions coordinated through events. Each step has a compensating action that undoes it on failure.

Choreography-based saga (preferred for loose coupling):

OrderService  --OrderPlaced-->  PaymentService
PaymentService --PaymentSucceeded-->  InventoryService
InventoryService --InventoryReserved-->  ShippingService

On failure:
PaymentService --PaymentFailed-->  OrderService (compensate: cancel order)
InventoryService --InsufficientStock-->  PaymentService (compensate: refund)

Orchestration-based saga (use when coordination logic is complex): A central OrderSaga orchestrator sends commands to each service and tracks state. Easier to reason about, but the orchestrator is a single point of coupling.

Always define the compensating action for every step before implementing the happy path. If you cannot compensate a step, it must be the last step in the saga.

---

Handle idempotency in consumers

Duplicate messages are inevitable. Every consumer must handle them safely.

Strategy 1 - Deduplication table:

CREATE TABLE processed_events (
  event_id UUID PRIMARY KEY,
  processed_at TIMESTAMPTZ DEFAULT now()
);

Before processing, check if event_id exists. Use a transaction to atomically insert into processed_events and execute the business logic.

Strategy 2 - Natural idempotency: Use operations that produce the same result regardless of how many times they run. SET status = 'shipped' is idempotent. INCREMENT counter is not. Prefer SET-style operations where possible.

---

Design event schema and versioning

Schema structure:

{
  "event_id": "uuid",
  "event_type": "OrderPlaced",
  "aggregate_id": "uuid",
  "version": 1,
  "timestamp": "2026-03-14T10:00:00Z",
  "data": {
    "order_id": "uuid",
    "customer_id": "uuid",
    "items": [],
    "total": 4999
  },
  "metadata": {
    "correlation_id": "uuid",
    "causation_id": "uuid",
    "user_id": "uuid"
  }
}

Versioning strategies:

• Upcasting: Transform old events to the new schema at read time. The event store keeps the original; the reader converts on the fly.
• Schema registry: Use Confluent Schema Registry (Avro/Protobuf) or a custom registry for JSON Schema. Enforce backward compatibility on every schema change.
• Weak schema: Add new fields as optional with defaults. Never remove or rename fields in a non-breaking way.

Always include correlation_id and causation_id in metadata. Correlation ID traces the full business flow; causation ID links to the specific event that caused this one.

---

Anti-patterns / common mistakes

Mistake	Why it's wrong	What to do instead
Using events as remote procedure calls	Tight coupling disguised as events; consumers depend on producer behavior	Events describe what happened, not what should happen next
Giant events with full aggregate state	Consumers couple to the producer's internal model; any schema change breaks everyone	Include only the data consumers need; use event-carried state transfer selectively
No dead-letter queue	Poison messages block the entire consumer; one bad event stops all processing	Configure a DLQ on every queue; alert on DLQ depth; review and reprocess manually
Ordering across partitions	Kafka only guarantees order within a partition; assuming global order causes race conditions	Partition by aggregate ID; accept that cross-aggregate ordering requires explicit coordination
Skipping idempotency because "the broker handles it"	At-least-once is the realistic guarantee; exactly-once has caveats and performance costs	Build idempotency into every consumer with dedup tables or natural idempotency
Unbounded event store without snapshots	Aggregate reconstruction slows to a crawl as event count grows	Snapshot every N events (e.g., every 100); load from latest snapshot then replay remaining events

---

Gotchas

1. Kafka ordering is per-partition, not per-topic - If two events for the same aggregate land on different partitions (e.g., because the partition key was not the aggregate ID), consumers can process them out of order. Always partition by aggregate ID to guarantee ordering for a single entity.

2. Projections that fail partway through leave read models in a partially-updated state - If a projection crashes after updating one table but before updating a second, the read model is inconsistent. Either wrap projection updates in a transaction, or design projections to be idempotent and rebuildable from scratch on failure.

3. No dead-letter queue means one poison message blocks all consumers indefinitely - A malformed event that causes a consumer to throw on every retry will halt the entire consumer group for that partition. Configure a DLQ from day one and alert on DLQ depth. Never leave a consumer group blocked waiting for a poison message to magically self-heal.

4. Schema changes to events must be backward-compatible or all existing consumers break - Renaming a field, changing a field type, or removing a required field in an event schema breaks every consumer that uses it, including projections built from historical events replayed from the store. Add new fields as optional with defaults; never remove or rename fields without a versioning strategy.

5. The event store growing without snapshots causes unbounded aggregate reconstruction time - An aggregate with 10,000 events takes 10,000 event loads and applies to reconstruct current state. Plan snapshots before going to production: capture state every N events (e.g., every 100) and store alongside the event log.

---

References

For detailed content on specific sub-topics, read the relevant file from the references/ folder:

• references/event-sourcing-patterns.md - Advanced event sourcing patterns including snapshots, projections, temporal queries, and event store implementation details
• references/broker-comparison.md - Deep comparison of Kafka, RabbitMQ, NATS, SQS/SNS, and Pulsar with configuration examples and operational guidance

Only load a references file if the current task requires it - they are long and will consume context.