Event-Driven Architecture

Why Event-Driven Matters

Tight RPC coupling is invisible at five services and existential at fifty. Every synchronous dependency is a runtime dependency: if it’s down, you’re down. Every new consumer is a producer change. Every release becomes a coordinated dance because two services can never be deployed independently if one calls the other in the request path.

Event-driven architecture inverts the dependency. The producer publishes a fact about its own domain — it has no opinion about who reads it. Consumers subscribe at their own pace, scale independently, fail independently, and can be added or removed without the producer noticing.

What you trade for that decoupling

If you don’t need any of the things in the left column, don’t pay for the things in the right column. Event-driven is not a virtue — it’s a tool. Apply it where loose coupling and asynchronous processing actually buy you something.

Events vs Commands vs Messages

The vocabulary is small but precise:

An event says “this happened.” It cannot be rejected and cannot be wrong — the past is the past. The producer publishes it because it changed its own state and is informing the world. Other services consume it and decide what to do.

A command says “please do this.” It can be rejected. It is addressed to one specific receiver. The receiver owns the decision about whether the operation succeeds. Commands look like async RPC and are often the right primitive when you’re asking another service to take an action with side effects.

You can use the same broker for both — Kafka, RabbitMQ, NATS, AWS SNS+SQS, GCP Pub/Sub all transport bytes — but the architectural shape differs. Commands typically use a queue with one consumer group; events typically use a topic with many consumer groups, each independently positioned.

Choreography vs Orchestration

In choreography, each service subscribes to events it cares about and publishes events of its own. There is no central conductor; the workflow emerges from local reactions. This is event-driven in its purest form.

In orchestration, a coordinator service (or workflow engine like Temporal, AWS Step Functions, Camunda) holds the state machine. It calls services, waits for replies, handles retries and compensation, and is the one place in the system where the entire flow is visible.

Choose orchestration when the failure modes need to be reasoned about in one place — payments, refunds, KYC pipelines, anything you’ll get audited on. Choose choreography when the producer genuinely doesn’t care who’s downstream — analytics fan-out, search index updates, cache invalidations.

Most mature systems use both: choreography for the broad event spine, orchestration for the few critical workflows that span services and need explicit compensation.

An orchestrated workflow in Temporal

from datetime import timedelta
from temporalio import workflow, activity

@activity.defn
async def charge_card(order_id: str, amount: int) -> str:
    return await payment_client.charge(order_id, amount)

@activity.defn
async def reserve_inventory(order_id: str) -> None:
    await inventory_client.reserve(order_id)

@activity.defn
async def refund(order_id: str) -> None:
    await payment_client.refund(order_id)

@workflow.defn
class PlaceOrderWorkflow:
    @workflow.run
    async def run(self, order_id: str, amount: int) -> str:
        # Step 1: take the money. Retries, timeouts, and durable
        # state are handled by the workflow engine, not by us.
        auth = await workflow.execute_activity(
            charge_card, order_id, amount,
            start_to_close_timeout=timedelta(seconds=30),
        )
        try:
            await workflow.execute_activity(
                reserve_inventory, order_id,
                start_to_close_timeout=timedelta(seconds=10),
            )
        except Exception:
            # Compensation: the saga’s undo step
            await workflow.execute_activity(refund, order_id)
            raise
        return auth

Notice what the orchestrator buys you: the entire happy path and the compensation are readable in one file. A choreographed equivalent has the same logic spread across three services and a documentation page nobody updates.

Event Schemas and Evolution

The two relevant standards in production today:

• CloudEvents (CNCF spec) — a vendor-neutral envelope: id, source, type, specversion, time, data. Standardizes the metadata so cross-cutting tooling (tracing, routing, dead-letter handling) doesn’t need to know your payload.
• Schema registry (Confluent Schema Registry, AWS Glue Schema Registry, Apicurio) — stores Avro / Protobuf / JSON Schema definitions, assigns versions, and enforces compatibility rules at publish time.

A CloudEvent in JSON

{
  "specversion": "1.0",
  "type": "com.acme.orders.placed.v1",
  "source": "/orders-service",
  "id": "01HRB6ZQX9N7K8M2J4F5G6H8YT",
  "time": "2026-05-12T14:22:31.412Z",
  "subject": "order/9821",
  "datacontenttype": "application/json",
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "data": {
    "orderId": "9821",
    "customerId": "c-44012",
    "totalCents": 12999,
    "currency": "USD",
    "items": [
      { "sku": "BK-001", "qty": 2, "priceCents": 5499 },
      { "sku": "BK-014", "qty": 1, "priceCents": 2001 }
    ]
  }
}

The version is in the type name (...placed.v1). When the schema breaks compatibility, you publish to ...placed.v2 and run both topics in parallel until consumers migrate. The version is never silently overloaded.

Compatibility modes

In Avro and Protobuf, the rules are simple and enforced by the registry: adding an optional field with a default is fine; removing a required field breaks; renaming breaks; changing a type breaks. The registry rejects the publish before a single consumer ever sees the new shape.

The cost of breaking changes

Once an event has 50 consumers, a breaking change is a project, not a commit. You publish v2 alongside v1, write a deprecation notice, chase every team to migrate, run both topics in parallel for months, and only then turn off v1. If you skip the schema registry, the “migration” happens during an outage instead of a project plan.

Outbox Pattern and Reliable Publishing

Two writes to two systems in one request is impossible to make atomic without distributed transactions, and distributed transactions are not what you want. The outbox sidesteps the problem by making the broker publish a downstream consequence of a single local transaction.

The outbox table

CREATE TABLE outbox (
    id           UUID        PRIMARY KEY,
    aggregate    TEXT        NOT NULL,         -- e.g. 'order'
    aggregate_id TEXT        NOT NULL,         -- partition key for ordering
    event_type   TEXT        NOT NULL,         -- 'OrderPlaced.v1'
    payload      JSONB       NOT NULL,
    headers      JSONB       NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at TIMESTAMPTZ
);

CREATE INDEX outbox_unpublished_idx
    ON outbox (created_at)
    WHERE published_at IS NULL;

The producing transaction

def place_order(req: PlaceOrderRequest) -> OrderId:
    with db.transaction() as tx:
        order = orders.insert(tx, req)            # business write
        outbox.insert(                            # event write — same tx
            tx,
            aggregate="order",
            aggregate_id=order.id,
            event_type="OrderPlaced.v1",
            payload=order.to_event_payload(),
            headers={"traceparent": tracing.current_traceparent()},
        )
    # Either both rows commit, or neither does. No dual write.
    return order.id

The relay worker

def relay_loop():
    while True:
        batch = db.query("""
            SELECT id, aggregate_id, event_type, payload, headers
            FROM outbox
            WHERE published_at IS NULL
            ORDER BY created_at
            LIMIT 500
            FOR UPDATE SKIP LOCKED
        """)
        if not batch:
            time.sleep(0.1)
            continue

        for row in batch:
            producer.send(
                topic="orders.events",
                key=row.aggregate_id.encode(),     # preserves per-order ordering
                value=cloudevent_envelope(row),
                headers=[(k, v.encode()) for k, v in row.headers.items()],
            )
        producer.flush()
        db.execute(
            "UPDATE outbox SET published_at = now() WHERE id = ANY(%s)",
            [r.id for r in batch],
        )

The same row may be published twice if the worker crashes after send but before the UPDATE. That is fine, and it is why consumers must be idempotent. The contract is at-least-once delivery; consumers turn that into exactly-once effectively by deduplicating on the event id.

CDC instead of a relay worker

Debezium reads your database’s write-ahead log (Postgres logical decoding, MySQL binlog) and emits a Kafka record for every row change. You can point Debezium at the outbox table and skip the polling worker entirely. The broker stays in sync with the database with no extra moving parts on the application side.

Event Sourcing

Event sourcing is a much stronger commitment than “we publish events.” In an event-sourced system, the events are stored permanently and any read model is reconstructable by replay. You get a perfect audit log, the ability to rebuild caches and projections from scratch, and the ability to ask questions about the past you didn’t know to ask at the time.

You also pay for it. Every read either replays from the log or hits a projection that is itself eventually consistent. Schema changes apply to events written years ago, so you need an upcasting strategy. Snapshots help with replay performance but add another moving piece to maintain.

The honest take: event-driven architecture and event sourcing are different things, and most teams should have the first without the second. Publish events from your services using outbox + a normal DB. Reach for full event sourcing only when you have a domain that genuinely benefits from it, and a team that has done it before.

Operational Concerns

Once events are in production, the failure modes are different from RPC and you need a different operational vocabulary.

Ordering and partitioning

Kafka and Pub/Sub guarantee ordering within a partition, not across the topic. The partition is selected by the message key. If you want all events for a given order to arrive in order, key them by orderId — they hash to the same partition and a single consumer thread processes them sequentially.

Choose the partition key from the unit of consistency you actually need. customerId if customer-level ordering matters. tenantId for multi-tenant isolation. Random if you genuinely don’t care — which is rare.

Dead-letter queues and replay

A consumer that cannot process a message after N retries should not block the partition. Move it to a dead-letter queue (DLQ) — a separate topic for poison messages — and continue with the next message. A human or automation later inspects the DLQ, fixes the underlying issue, and replays the messages.

Replay is the underrated superpower of an event-driven system. Bug in your projection? Reset the consumer offset and rebuild from the start of the topic. New analytics use case? Spin up a consumer at offset zero and let it catch up. None of this is possible if you treated the broker as fire-and-forget.

Schema-version skew

Producers and consumers will run different schema versions in production at the same time, every single day. You don’t coordinate releases across teams — that’s the whole point of decoupling. So consumers must tolerate v1 and v2 events simultaneously during the migration window, and the schema registry’s compatibility rules are what makes that safe.

Idempotency keys end-to-end

The event id from the CloudEvents envelope is your idempotency key. Persist it on the consumer side in the same transaction as the business mutation. Combined with at-least-once delivery, this gives you the “exactly-once effectively” semantics that production systems actually run on. Anyone selling you literal exactly-once across heterogeneous systems is selling you a marketing slide.

The Kafka consumer that handles all of this

from confluent_kafka import Consumer, KafkaError, KafkaException
import json

consumer = Consumer({
    "bootstrap.servers": "kafka:9092",
    "group.id": "shipping-projector",
    "enable.auto.commit": False,    # commit only after we persist
    "auto.offset.reset": "earliest",
    "isolation.level": "read_committed",
})
consumer.subscribe(["orders.events"])

while True:
    msg = consumer.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        if msg.error().code() == KafkaError._PARTITION_EOF:
            continue
        raise KafkaException(msg.error())

    event = json.loads(msg.value())
    try:
        with db.transaction() as tx:
            # Idempotency: insert into processed_events first.
            # Unique-key violation => we’ve seen this event; skip.
            tx.execute(
                "INSERT INTO processed_events (consumer, event_id) VALUES (%s, %s)",
                ("shipping-projector", event["id"]),
            )
            apply_to_projection(tx, event)
        consumer.commit(msg)
    except UniqueViolation:
        consumer.commit(msg)              # duplicate; safe to skip
    except PoisonMessageError:
        send_to_dlq(event)                # move on; don’t block partition
        consumer.commit(msg)

Real-World Examples

LinkedIn (Kafka’s origin). By 2010, LinkedIn had a tangle of point-to-point integrations between dozens of systems — activity, search, recommendations, analytics. Each new pipeline meant new integrations and brittle ETL. They built Kafka as a unified, replayable log so producers wrote once and any downstream system could subscribe. Today LinkedIn processes trillions of events per day on Kafka and donated it to the Apache Software Foundation in 2011.

Uber’s dispatch. Driver locations, ride requests, ETA updates, surge pricing — all flow as events through Kafka topics keyed by city, driver, or trip. The dispatch decision is a stream-processing job. The synchronous version of that system would have collapsed under traffic; the event-driven version horizontally scales by adding partitions and consumer instances.

Netflix Keystone. Netflix runs one of the largest event pipelines in the world — trillions of events per day flowing into S3, Elasticsearch, Druid, and downstream services. The Keystone platform standardized event publishing across thousands of microservices so any team could declare a stream and let the platform handle delivery, retention, and routing.

Shopify. Shopify runs an internal event bus on Kafka with strict CloudEvents-style schemas and a registry. Every domain event — orders, products, customers — is published once and consumed by analytics, search, fulfillment, fraud detection, and partner webhooks. New consumers attach without producer changes; the producer team owns the schema and the broker, and that’s the whole interface.

Best Practices