The Outbox Pattern
Here is a problem that sounds like it should be a non-problem. You've written an order to the database. Now you want to tell the rest of the system about…
The shape of the whole thing
4 stages, in the order the chapter argues them. Read it top to bottom, or jump straight to the part you came for.
Introduction
Here is a problem that sounds like it should be a non-problem. You've written an order to the database. Now you want to tell the rest of the system about it, so you publish an event to Kafka. Two lines of code, adjacent on the screen. Done.
Except sometimes the publish fails. Or the process dies in the gap between the database commit and the Kafka send. Or Kafka blinks — briefly, just long enough — and the retry logic that looked bulletproof in review turns out not to survive a pod restart. The order exists. The event does not. Nothing downstream knows the order happened. The inventory service is holding stock for an order nobody will process. The recommendation engine spends the next three weeks suggesting the item the user already bought. The fraud score never updates. And none of this announces itself — it waits, politely, until a support ticket arrives and someone goes looking.
This is the dual-write problem, and it comes for every team that builds event-driven services with enough enthusiasm to ship fast and enough traffic to turn the edge cases into Tuesdays. Chapter 10 met it from the other side: there, the fix was to stop asking the application to publish at all and let CDC read the log. The Outbox pattern is the same instinct pointed at a slightly different target, and it's the one you reach for when you want the application's events — business facts, not row diffs — delivered with the log's reliability.
The move is almost insulting once you see it. Don't write to the database and then write to Kafka. Write to the database twice — the business data and the event — in the same transaction. Then let something else read the event back out of the database and publish it. That something else is usually CDC, which the last chapter covered at length. The application never speaks to Kafka. It writes SQL and goes home.
The name is older than any of this. An outbox is the tray on a desk: you write the letter, drop it in the tray, and someone else mails it. You've done your part. The mail can be slow, but it cannot be lost, because the letter is sitting in a physical place that survives you leaving the room — which is more than you can say for an in-flight Kafka producer when the pod gets OOM-killed. The database is the tray. CDC is the person who collects the mail. Kafka is the postal service. The letter goes out exactly as written, in the order you dropped it, regardless of what happens to you after you set it down.
Here's the ground we'll cover: why the naive approaches fail in ways that sail through review and surface in production, the mechanics of the outbox table and the CDC pipeline that drains it, where the pattern earns its operational rent and where it's just rent, how Uber leaned on it at ride-matching scale, and the questions a principal engineer should answer out loud before reaching for any of it.
The Problem: You Can't Commit to Two Systems at Once
Consider the purchase flow one more time. It's the clearest example and it never gets old.
User clicks Buy. OrderService opens a transaction, inserts a row into orders, commits. The write is durable; the purchase is real; the database said so. Now OrderService calls kafkaProducer.send("order-created", event). The client buffers the message, flushes it to the broker, waits for the acknowledgment, gets it, and returns 200 to the user. That's the happy path, and it works exactly as advertised. The problem is the seam between commit and send.
The process gets OOM-killed. A network partition isolates the pod from the broker for thirty seconds — longer than the producer timeout. The broker is mid-way through a leader election for that partition. A config mistake set acks=0, so the send "succeeds" from the producer's point of view while the broker never persists a thing. The deployment you pushed two minutes ago terminates the old pod before its in-flight send completes. None of these throw a clean exception the application can catch and handle. Most of them produce silence — and silence is the one failure your monitoring can't page you about. The OrderService is gone before it can retry. The order is in the database. The event is nowhere. The services that depend on order-created — inventory, loyalty, the confirmation email — proceed on the assumption that nothing was ordered, because as far as they can tell, nothing was.
The uncomfortable part is the arithmetic. A few thousand orders a day, a dual-write window of maybe fifty milliseconds, two deployments a day, each killing a handful of in-flight requests — multiply it out and you don't risk this failure, you schedule it. Not in theory. In production. On a Tuesday.
The common first response is, "We'll publish to Kafka before we return 200, and if the publish fails we roll back the database write." Tempting, and mostly wrong. Rolling back an order because Kafka is having a moment couples your checkout flow directly to Kafka's availability — so now a routine broker maintenance window takes down purchasing. You haven't solved the consistency problem. You've traded it for a worse-behaved one that fires more often and on someone else's schedule.
The second response is better: "Write the database first, and if the Kafka send fails, retry from the application." Fine, until you remember that retries need a living process to run them. The crash, the OOM kill, the deployment — process death is precisely the scenario the retry was supposed to cover, and it's the one scenario where there's no process left to do the retrying. You cannot retry from a pod that no longer exists.
Both answers treat the database and the broker as peers to be kept in sync. The Outbox pattern throws that framing out. The database is the source of truth. The broker is a derived view. The application's only job is to update the source of truth, once, atomically. Everything else is delivery — and delivery is allowed to be eventually consistent, because the truth is already safe.
The Outbox Table
The mechanics are unglamorous, which is the point. You add a table — same database, same schema, same transaction boundary as everything else — that holds pending events. When the application does a business write, it also inserts a row into this table, inside the same transaction. (Full schema and transactional insert: Appendix A.1.)
The table itself is boring on purpose: an auto-incrementing id for sequence, an aggregate_type and aggregate_id naming the domain object, an event_type for what happened, a payload (JSON, usually) for the event body, and a created_at. The interesting part is what the transaction boundary buys you. The insert into orders and the insert into outbox either both commit or both roll back. There is no state where the order exists without its event, or the event exists without its order. You did not coordinate two systems to get this. You wrote one transaction and let the database do the thing it has been quietly excellent at since before Kafka was a gleam in LinkedIn's eye.
That's what "atomic" means here — not fast, not single-step, but atomic in the sense that matters: the two writes live or die as a unit. Drop the connection mid-transaction, both roll back. Crash the application a microsecond after commit, the outbox row is sitting there, waiting. The event isn't lost. It's just undrained, which is a problem with a known solution rather than a mystery with a support ticket.
Something does have to drain it. That's where CDC walks back in.
CDC Processing the Outbox
Once the event is in the outbox table, you need it in Kafka — and the cleanest way, with the strongest guarantees, is to point your CDC pipeline at the outbox table and let it treat that table like any other.
Debezium watches the outbox table through the WAL, exactly as Chapter 10 described. A new row gets inserted inside a business transaction; Debezium picks it up from the log and publishes it to Kafka. No code in OrderService calls a producer. No code in OrderService knows where Kafka lives, or that Kafka exists.
orders table (business data), one into the outbox table (the event) — annotated "ACID: both or neither." The application touches nothing else. Center (read lane): Debezium reads the outbox table's WAL entries off to the side, never in the write's path, converts each row to an event, and emits to a Kafka topic; the arrow is labelled "at-least-once, in commit order." Right: Inventory, Email, and Fraud each consume from the topic independently, at their own pace, each with its own offset.Debezium ships a purpose-built Outbox Event Router — a Kafka Connect single message transform that understands the outbox table's shape and routes events to per-domain topics based on aggregate_type. An OrderCreated lands in orders.events; a PaymentProcessed lands in payments.events. You don't strictly need it — you can route at the consumer instead — but it earns its place the moment multiple aggregates share one outbox table, which they usually do.
Here's the subtlety that catches people, and it's a good one. Without the router, Debezium publishes the outbox row as what it literally is: a database change event. Your consumers receive a message that says "a row was inserted into the outbox table," with the actual event payload nested inside Debezium's change-event envelope — not a message that says "an order was created." The router flattens that: it lifts the payload column up to the Kafka message value and sets the message key to aggregate_id, so consumers see a clean business event instead of a CDC artifact. This matters more than it sounds, because your consumers are written by other teams who have no desire to learn that your event bus secretly runs through a database table, and every layer of CDC plumbing that leaks into their deserializer is a layer they'll eventually file a ticket about.
The other way to drain the outbox is to skip Debezium entirely and poll: a background job selects unprocessed rows, publishes them, marks them processed, repeats. No WAL access, no replication slot, no Connect cluster — and the same atomicity guarantee, because the event sits in the database until it's safely published, no matter what happens to the poller. (Full polling drainer: Appendix A.2.) What you pay for that simplicity is latency, bounded by the polling interval, plus the chore of running the job. If you need sub-second delivery, use CDC. If a few seconds of lag is fine — and for a great many systems it is, whatever the real-time aspirations in the design doc — polling is worth a serious look, because it removes an entire category of infrastructure for a small, well-understood reliability cost.
Cleaning Up After Yourself
Left alone, the outbox table grows forever, because nothing in the design deletes a row by accident. CDC or polling, you need a deletion strategy, and you need it on purpose.
With CDC, deletion is cleaner than it first appears. Once Debezium has read a row from the WAL and published it, the row has done its entire job. Delete it — from the application, from a sweep job, or via a short TTL — and you're fine, because Debezium already captured the insert; the delete event it later emits for your cleanup can simply be dropped by configuring its tombstone handling and a filter. The row is gone; the event is already in Kafka, unbothered.
With polling, you mark rows processed before deleting them so a restarted poller doesn't republish — a processed_at column and a WHERE processed_at IS NULL predicate do it — and you sweep on a separate schedule.
And here's the failure mode nobody reads about until they're standing in it: a high-write service whose outbox table is never cleaned up, accumulating quietly over months until it's a few hundred million rows, at which point the poller's WHERE processed_at IS NULL query is doing a full table scan, because the partial index on processed_at was never added. The table is now actively slowing the write path it shares a database with — every insert still has to maintain that enormous index — and you learn all of this from the database CPU graph at 3 AM, which is the worst available time to learn anything. Add the index. Schedule the cleanup. Do both before you ship, not after the graph wakes you up to explain it.
Tradeoffs
Outbox vs. Application Events
The short version: application events are fast and simple, the Outbox pattern is slower and more complex. True, and not the whole story.
Application events — write the row, then call the producer directly — work correctly in the common case. The broker's up, the send succeeds, the consumer gets the event. For a low-write service where crashes are rare and the business can shrug off the occasional missed event, this may genuinely be the right call. The Outbox pattern adds infrastructure (the table, plus a CDC pipeline or a poller) and latency (Debezium reads the WAL; there's a gap between commit and publish). That overhead is worth paying when correctness is non-negotiable, and pure waste when you're publishing audit-log events nobody consumes in real time.
So the question is not "is the Outbox pattern better?" Better isn't a property a pattern has; it's a property of a pattern measured against a requirement. The real question is: when my Kafka publish fails and my process dies in the same instant, what do I want to have happened? If the honest answer is "we lose the event, and that's survivable," you probably don't need the outbox. If the answer is "we have an order with no inventory reservation and a customer waiting on an email that will never send," you do.
Ordering Guarantees
This is where the Outbox pattern is quietly stronger than most teams notice until they need it.
Publish directly to Kafka from the application and your ordering guarantees are weak. Multiple threads publish concurrently. The producer's internal queue can reorder under back-pressure. Kafka preserves order within a partition keyed by order_id, sure — but if two events for the same order are produced moments apart by different threads, you have no guarantee they reach the partition in the order the business actually intended.
The outbox table is a database table, so it has a sequence, and the drain reads it in the order the database committed the writes — Debezium walks the WAL in commit order. Events for the same aggregate arrive in Kafka in the order they were committed, which is the order your business operations actually happened. For a lot of systems this solves a problem they don't have. For ledgers, inventory, and anything shaped like a state machine — where PaymentCaptured arriving before PaymentAuthorized isn't a glitch but a refund-window calculation starting from the wrong moment, and OrderDelivered before OrderShipped is a customer-service call waiting to happen — it's the entire ballgame. You get this ordering as a free consequence of writing through a single transaction log.
The caveat, named out loud because teams keep rediscovering it the expensive way: ordering holds within one outbox table on one database primary. Shard across instances and events from different shards have no ordering relationship to each other. That isn't an Outbox limitation so much as a law of distributed systems with better PR — but teams migrating from a monolith to a sharded fleet tend to meet it as a surprise in a postmortem rather than a line in a design doc, so consider this the line in the design doc.
Architecture: What It Actually Looks Like End-to-End
In production: OrderService receives a placement request, opens a transaction, writes the order to orders and a structured event to outbox, commits, returns 200. From the client's side, that's the whole request — done in one round trip, with no dependency on any broker being awake.
Meanwhile Debezium reads the outbox table's WAL entries and publishes to Kafka, within milliseconds to seconds of the commit depending on replication lag and flush intervals. Inventory, email, and fraud each consume OrderCreated from orders.events, update their own state, and run independently — a slow fraud evaluation doesn't hold up the confirmation email, because they aren't standing in each other's line.
OrderService → a transaction box containing two inserts side by side, orders and outbox, joined by a brace reading "same transaction." Middle (read lane): outbox → WAL → Debezium (Kafka Connect worker) → Outbox Event Router SMT → topic orders.events, annotated with key = order_id, value = OrderCreated payload. Bottom: three consumer groups branching off orders.events — Inventory (reserves stock), Email (sends confirmation), Fraud (updates risk model) — each with its own offset pointer.OrderService has no operational dependency on Kafka, Debezium, or any consumer; the outbox row is the only handoff; consumer independence means a failure in one consumer doesn't reach the others or the write path. Shared visual grammar with the earlier outbox figure: write lane vs. read lane, capturer aside from the write.What the diagram can't draw is the part principal engineers care about most: OrderService and the Debezium connector have no operational dependency on each other. Restart Debezium to change a connector setting without touching OrderService. Roll out a new InventoryService consumer without touching anything upstream. Each component evolves on its own schedule, for as long as the outbox event schema stays compatible — and that last clause is doing more work than it looks, which is a thread we pick up in the Principal Engineer section.
How Uber Handles This at Ride Scale
Uber's driver-matching system is a clean case study because the ordering requirement isn't a nice-to-have. It's correctness.
A trip generates a sequence: TripRequested, DriverMatched, TripStarted, TripCompleted, PaymentCaptured. Those events drive state machines in several systems at once — pricing, driver pay, the rider's trip history, fraud scoring, analytics. A TripCompleted that arrives before TripStarted produces a trip that ended before it began, which is the kind of state that makes a fraud model file a bug report against reality. A PaymentCaptured that lands before TripCompleted starts a refund-window clock at the wrong moment. None of these are exotic; they're just what out-of-order delivery looks like when the events mean something.
The problem compounds at Uber's volume — millions of trips a day, multiple microservices each keeping their own view of trip state. When Uber moved from direct Kafka publishing to an outbox approach, the specific thing they were killing was lost events during deployments. The trip service shipped dozens of times a week, and each deployment had a window — brief, but measurable — where in-flight publishes were dropped as pods cycled. At millions of trips a day, "brief and measurable" rounds up to hundreds of lost events per deployment cycle, which is a lot of trips quietly falling out of someone's downstream state.
The fix was the one this chapter describes: write the event to an outbox table inside the trip-state transaction, drain it via CDC. The deployment window stopped mattering. A pod dying mid-deploy left the outbox rows on disk; the new pod came up and Debezium resumed draining them with no application-level awareness that anything had been interrupted.
The detail worth stealing is the routing. Trips fire events in quick succession — DriverMatched and TripStarted can commit seconds apart — and Uber keys outbox events to Kafka partitions by trip_id. Every event for one trip lands in the same partition, in order, no matter how many consumers are running, so a consumer processing trip abc-123 sees its events in commit order every time, with zero extra coordination. This is not a clever trick. It's the boring consequence of using the aggregate's primary key as the partition key — and the boring consequence is the one that saves a sprint of debugging out-of-order processing, which teams usually only appreciate after they've spent that sprint.
Shopify's order pipeline lands in the same place at a different scale and under a different constraint. Shopify coordinates inventory across thousands of shop-specific databases, each with its own outbox, so their implementation carries a correlation_id that propagates a request identifier across service boundaries — letting them reconstruct the full event chain of an order even when events arrive from many sources with no shared wall-clock order. It's a one-column extension to the standard outbox schema, and it makes cross-service debugging dramatically less miserable, which is the highest praise a schema column can earn.
Technologies
Debezium is the primary CDC engine in most outbox implementations, and for a good reason: its first-class OutboxEventRouter SMT handles the extraction and routing without custom code. You tell it which columns map to aggregate_type, aggregate_id, event_type, and payload, and it takes it from there. If you're already running Debezium for other CDC work — which, after Chapter 10, you might be — adding outbox support is a connector config change, not a new piece of infrastructure to operate. (Full connector configuration: Appendix A.3.)
Polling-based draining is the right call when Debezium's operational overhead isn't justified by the use case. It's a background job that selects unprocessed rows ordered by id, publishes them, and stamps processed_at — no replication slot, no schema registry, no Connect cluster. The simplicity is real, not a consolation prize. The latency floor is the polling interval, typically one to five seconds; if downstream needs sub-second delivery this won't do, and if it doesn't, the simpler path is the better path more often than ambitious architects like to admit.
Spring Modulith on the JVM and MassTransit on .NET both ship outbox support as a first-class feature, and if you're in either stack, adopting the framework's mechanism almost always beats hand-rolling. The edge cases around idempotency, cleanup, and schema handling are already solved and already battle-tested. Rolling your own table and poller isn't hard, exactly — it's just more than it looks once you've added the cleanup job, the index, the deduplication, the monitoring, and the second poller you stood up for redundancy that now double-publishes everything until someone discovers SKIP LOCKED.
The application database needs no special configuration for any of this. The outbox table is just a table — and that's one of the pattern's most underrated properties. No new datastore, no new protocol, no new dependency in the request path. The consistency guarantee comes from the database you were already trusting with the business data, which means you're not adding a thing to trust so much as asking the thing you already trust to do slightly more.
The Principal Engineer Perspective
When the Outbox Pattern Is Worth the Complexity
The Outbox pattern adds three things you didn't have: a table (with its own schema, indexes, and cleanup obligations), a drain (a CDC pipeline or a polling job to operate, monitor, and debug), and a latency floor (the gap between commit and delivery). None of them are free, and pretending the first one is free — "it's just a table" — is how the other two arrive unbudgeted.
The pattern earns its keep when two conditions are both true: the event must not be lost, and the application can't afford to couple its availability to Kafka's. If only the first holds — losing events is unacceptable, but Kafka coupling is fine — a synchronous Kafka write with a circuit breaker and a dead-letter queue may be simpler. If only the second holds — you won't couple to Kafka, but the occasional lost event is survivable — direct publishing with retries may be simpler. The Outbox pattern is for the intersection, which describes a meaningful chunk of high-value financial and operational flows and a much smaller chunk of analytics and notifications. Knowing which set you're in is most of the decision.
The organizational cost is the one teams reliably underprice. The pattern requires that whoever is on call for the data pipeline understands the relationship between the outbox table and the CDC connector. When Debezium falls behind on a busy day — a batch job touches half a million rows and the outbox drains slower than it fills — someone has to recognize that downstream services are no longer real-time, diagnose the lag, and tell the affected teams before they notice on their own. That's not exotic knowledge, but it has to live somewhere. If the team that owns the service also owns the drain, fine. If CDC belongs to a platform team with a ticket queue and a business-hours SLA, you're going to have some genuinely memorable conversations at 2 AM about whose incident this actually is.
The Failure Mode That Surprised Us
It's not a crash. It's a schema migration — the most routine thing a database team does, turned into someone else's outage.
A developer adds an is_corporate_account boolean to the OrderCreated payload, updates OrderService to populate it, merges. The outbox starts producing events with the new field. Consumers written against the old schema — and not told about the change, because why would they be, nobody touched their code — meet a payload they didn't expect. If they validate strictly, they fail loudly. If they deserialize loosely, they silently drop the field. If they're strict but the field has a default, they're fine. Which outcome you get depends entirely on how each consumer was written, by different teams, at different times, and you will not know which until it happens.
The outbox payload is an API. It doesn't look like one — no OpenAPI spec, no version header, no SDK, just a JSON column traveling through a database table — which is exactly why people change it without the ceremony they'd give a REST endpoint. The consumers don't care that it doesn't look like an API. They break like it's an API. Every field in that payload is part of a contract, and changes to it need to be versioned and backward-compatible, or coordinated across consumers before the producing service ships. Same discipline as any public interface; it just doesn't feel like one, and the feeling is the trap.
The subtler version is worse, because nobody even thinks they changed the event. A developer runs ALTER TABLE orders ADD COLUMN corporate_account_id INT. They don't go near the outbox. But the code that assembles the OrderCreated payload reads from the full order record — so the new column is now in the JSON, automatically, silently. Nobody edited the outbox. Nobody believes they touched the event. The event changed anyway, because the payload is built from a row that just grew a column, and if your Avro schemas in Schema Registry don't validate JSONB contents at publish time — they frequently don't — nothing catches it until a consumer does, in production, on its own schedule.
So add schema validation to the outbox publisher. Write a test that fails when the event structure changes unexpectedly. Treat it as the API surface it is. The cost is an afternoon. The cost of skipping it is a downstream outage at a time, again, not of your choosing — there's a pattern to when these things fire, and the pattern is "never during business hours."
Questions Before You Commit
Before adopting the Outbox pattern for a new service, a few questions deserve answers out loud, not assumed.
Who owns the CDC pipeline that drains the outbox? If it's your team, its availability is part of your service's reliability, full stop. If it's a platform team, what's the SLA, and does it actually meet what your downstream consumers believe they're getting?
What lag between commit and delivery is acceptable? Debezium running normally adds milliseconds to low seconds. If a consumer SLA needs sub-100ms, account for it now. If it needs sub-10ms, the Outbox pattern — a database table plus a CDC pipeline — is probably not the architecture you want, and it's cheaper to learn that here than in a load test.
How will you handle schema evolution? If multiple teams consume your events, you need a versioning strategy before the first event ships, not after you've broken someone and learned their team's name during the incident.
What's the cleanup strategy for processed rows? Define it, build it, and test it under load before you're staring at a 200GB outbox table wondering why the database is at 90% CPU. This is the one everyone agrees with and half of teams still skip.
Exercises
Exercise 1: The Deployment Gap
Your OrderService uses direct Kafka publishing, no outbox. You average 500 orders per hour. A deployment takes 45 seconds, during which old pods are terminated before new pods are fully ready. Assume 200ms average between database commit and Kafka publish.
Estimate how many orders lose their event during a typical deployment. What would you need to know to make the estimate precise? (Hint: think about concurrent in-flight requests, not just timing.) Then design the outbox table for this service — columns, indexes, and the cleanup job. If your cleanup job is "we'll add it later," reread the section on the 3 AM CPU graph and try again.
Exercise 2: The Slow Drainer
Your outbox table has 2 million unprocessed rows. Debezium is running but lagging by four hours. OrderService is healthy; orders are being created normally.
What's the impact on downstream services right now? Which are most affected, and which can keep functioning without real-time events? What's your recovery plan — let Debezium catch up, or do something more active? And the question that matters most for next time: what monitoring would have caught this two hours earlier, while it was still boring?
Exercise 3: The Schema Surprise
A team adding a new payment method extends the OrderCreated payload with a payment_method_type field. They ship it. Two weeks later, a downstream fraud service starts throwing errors.
Trace the failure end to end. What exactly went wrong, and why did it take two weeks? What process change would have prevented it? Then write the schema-compatibility check you'd add to the outbox publisher's unit tests — the one that would have turned this incident into a failed build instead.
Connections to Other Chapters
← Chapter 10 (CDC). The Outbox pattern is where CDC stops being a replication tool and becomes a delivery guarantee. Chapter 10 used CDC to capture changes without touching the application. The Outbox pattern gives the application a way to intend those changes — to write events into the database deliberately so CDC picks them up and delivers them reliably. They compose cleanly: CDC is the mechanism, the outbox is the discipline.
→ Chapter 12 (Saga Pattern). Sagas — multi-step distributed transactions across services — almost always need the Outbox pattern for their event publishing. A saga step commits a business write and publishes the event that triggers the next step; without the outbox, a crash between those two operations strands the saga with no way forward. Chapter 12 assumes the outbox as the publishing primitive for saga steps, which makes this chapter the prerequisite rather than the sequel.
← Chapter 9 (Event Sourcing). Event sourcing stores every state change as an event and makes that log the source of truth. The Outbox pattern is a lighter cousin: the outbox is a log of intent, not a full event store, but it plays the same structural role — making events durable, first-class records instead of fire-and-forget messages. Teams who want event sourcing's reliability without its full commitment often find themselves, somewhat to their surprise, building an outbox.
The intuition to carry out of this chapter: the dual-write problem is not a discipline problem, and you will not fix it by being more careful. It's a property of writing to two systems that don't share a transaction boundary — no quantity of code review legislates that away. The Outbox pattern doesn't solve distributed coordination; it refuses to need it. By folding the event into the same write as the business data, you turn a two-system coordination problem into a one-system write followed by an eventually-consistent delivery — and the database has been doing reliable, ordered, atomic writes since long before any of us had a message broker to misuse. Let it do the part it's good at. Let CDC carry the mail.
Appendix A: Reference Implementations
The chapter keeps the code out of the narrative on purpose. Here it is, collected and annotated — read the prose for the idea, come here for the literal shape. All of it is illustrative rather than production-hardened: it's here to make the structure concrete, not to be pasted into a connector config and pointed at your production database before lunch.
A.1 — Outbox Table Schema and Transactional Insert
1-- Outbox table schema2CREATE TABLE outbox (3 id BIGSERIAL PRIMARY KEY,4 aggregate_type VARCHAR(255) NOT NULL, -- e.g., 'Order', 'Payment'5 aggregate_id BIGINT NOT NULL, -- FK to the business entity6 event_type VARCHAR(255) NOT NULL, -- e.g., 'OrderCreated'7 payload JSONB NOT NULL, -- full event payload8 created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),9 processed_at TIMESTAMPTZ NULL -- set by poller; NULL = unprocessed10);11 12-- Essential indexes13CREATE INDEX idx_outbox_unprocessed ON outbox (id ASC) WHERE processed_at IS NULL;14CREATE INDEX idx_outbox_created_at ON outbox (created_at ASC);15 16-- Transactional write: business data + event in one commit17BEGIN;18 INSERT INTO orders (user_id, total_cents, status)19 VALUES (42, 9900, 'placed')20 RETURNING id INTO _order_id;21 22 INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)23 VALUES (24 'Order',25 _order_id,26 'OrderCreated',27 jsonb_build_object(28 'order_id', _order_id,29 'user_id', 42,30 'total_cents', 9900,31 'status', 'placed',32 'created_at', NOW()33 )34 );35COMMIT;The processed_at IS NULL partial index is load-bearing. Without it, the poller's WHERE processed_at IS NULL query scans the whole table once enough rows pile up, and on a busy service that happens sooner than you'd guess. The created_at index supports the cleanup job's date-range deletes. Both belong in place before the table sees a single byte of production traffic — adding an index to a 200-million-row table that's actively being written is its own small adventure.
A.2 — Polling-Based Outbox Drainer (Python pseudocode)
1import time2import psycopg23import json4from kafka import KafkaProducer5 6POLL_INTERVAL_SECONDS = 27BATCH_SIZE = 5008 9producer = KafkaProducer(10 bootstrap_servers=["kafka:9092"],11 key_serializer=str.encode,12 value_serializer=lambda v: json.dumps(v).encode("utf-8"),13 acks="all", # wait for all in-sync replicas14 retries=5,15 enable_idempotence=True,16)17 18def drain_outbox(conn):19 with conn.cursor() as cur:20 # Fetch unprocessed rows, ordered by id (commit sequence)21 cur.execute("""22 SELECT id, aggregate_type, aggregate_id, event_type, payload23 FROM outbox24 WHERE processed_at IS NULL25 ORDER BY id ASC26 LIMIT %s27 FOR UPDATE SKIP LOCKED28 """, (BATCH_SIZE,))29 rows = cur.fetchall()30 31 if not rows:32 return 033 34 for row_id, agg_type, agg_id, event_type, payload in rows:35 topic = f"{agg_type.lower()}s.events" # e.g., "orders.events"36 key = str(agg_id)37 future = producer.send(topic, key=key, value=payload)38 future.get(timeout=10) # block; raises on failure39 40 # Mark published rows as processed (in bulk)41 ids = [r[0] for r in rows]42 cur.execute("""43 UPDATE outbox SET processed_at = NOW()44 WHERE id = ANY(%s)45 """, (ids,))46 conn.commit()47 return len(rows)48 49conn = psycopg2.connect(dsn="postgresql://...")50while True:51 try:52 count = drain_outbox(conn)53 if count < BATCH_SIZE:54 time.sleep(POLL_INTERVAL_SECONDS) # back off if nothing to drain55 except Exception as e:56 conn.rollback()57 print(f"drain failed: {e}")58 time.sleep(5)Two details carry the weight. FOR UPDATE SKIP LOCKED is how you run multiple poller instances without double-publishing: each one locks the rows it's working and skips the ones a sibling already holds. Leave it out and two pollers race the same rows and publish duplicates — which is exactly the trap waiting for the redundant second poller someone adds for safety. The future.get(timeout=10) call blocks until the broker acknowledges, making the poller synchronous per message: slower, but correct and easy to reason about. A production version pipelines the sends — batch, flush once, then update processed_at — for throughput. The synchronous version is the right starting point, and the version you should be able to explain before you optimize past it.
A.3 — Debezium Outbox Event Router Configuration
1{2 "name": "orders-outbox-connector",3 "config": {4 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",5 "database.hostname": "postgres.internal",6 "database.port": "5432",7 "database.user": "debezium",8 "database.password": "${file:/opt/kafka/secrets.properties:postgres.password}",9 "database.dbname": "orders_db",10 "database.server.name": "orders",11 "table.include.list": "public.outbox",12 "plugin.name": "pgoutput",13 "slot.name": "debezium_outbox_slot",14 "transforms": "outbox",15 "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",16 "transforms.outbox.table.field.event.id": "id",17 "transforms.outbox.table.field.event.key": "aggregate_id",18 "transforms.outbox.table.field.event.type": "event_type",19 "transforms.outbox.table.field.event.payload": "payload",20 "transforms.outbox.route.by.field": "aggregate_type",21 "transforms.outbox.route.topic.replacement": "${routedByValue}s.events",22 "transforms.outbox.table.tombstone.on.empty.payload": "false",23 "heartbeat.interval.ms": "10000"24 }25}route.topic.replacement uses ${routedByValue} — the value of aggregate_type — to derive the target topic dynamically, so a row with aggregate_type = 'Order' routes to orders.events and one with 'Payment' routes to payments.events, and a single connector drains the whole table without any application-level routing. Change the aggregate_type you insert; the routing follows. The heartbeat.interval.ms setting matters here for the same reason it did in the CDC chapter, only more insidiously: an idle outbox table won't advance the replication slot's confirmed LSN, so WAL piles up — and an idle outbox looks like everything is fine (no events to drain) right up until the slot has quietly eaten the disk. Set the heartbeat. Alert on slot lag. The silence is not the same as health.
Next: Chapter 12 — The Saga Pattern: distributed transactions across services, and why the Outbox pattern you just built is the primitive that makes them reliable.