Eventual Consistency

Strong consistency makes one promise, and it's a good one: write something, and everyone sees it. Immediately. You change your profile photo in New York, and a user in Los Angeles sees the new one before your finger leaves the return key. There is one shared truth, every observer agrees on it at all times, and you get to reason about the whole system as if it were a single machine with a single copy of the data. At the scale where most systems are born, that promise is real and effectively free. At the scale where most systems struggle, it's a fantasy you rent by the millisecond.

What's standing in the way isn't a bad decision someone made in a design review. It's physics. Data has to get from New York to Los Angeles — about 2,700 miles of fiber, carrying signals at roughly two-thirds the speed of light — and the round trip floors out around 40 milliseconds. Strong consistency across regions means every write blocks until every replica has acknowledged it, so you are speed-limited by the farthest node you own. For a system spanning US East, US West, and Europe, a strongly consistent write starts at 80–100ms of pure network latency before a single CPU does anything useful. Now serve a few hundred thousand writes a second through that floor and watch the queue form. You cannot optimize your way out of the speed of light, though every few years a team tries.

So systems make a trade. Accept that replicas drift apart for a moment, and in exchange get to answer fast and stay up when part of the network is having a bad day. The write returns the instant it's persisted locally; replication catches up afterward, asynchronously; readers might see slightly stale data for a window — milliseconds to seconds, depending on how far the data has to travel — and then everything converges. That's eventual consistency. It's a defensible trade, frequently the correct one, and at real scale the only one on the menu.

The uncomfortable part isn't the trade. It's how rarely anyone makes it on purpose. Eventual consistency is usually the default behavior of a distributed system, arrived at by accretion rather than decision. One team adds a read replica to take load off the primary. Another adds a cache. A third stands up a second region. Nobody draws a consistency model on a whiteboard, because at no point did adding one more box feel like a philosophical commitment. Then, eighteen months later, a user tweets a screenshot of their account balance showing the wrong number for five seconds after a transfer, and suddenly the whole org is fluent in replication lag.

This chapter is about making that reasoning deliberate before the tweet. What does eventual consistency actually mean in concrete terms? How long is "eventually," in numbers you can put on a dashboard? What breaks when you assume a consistency you don't have? Which patterns let you work with the replication window instead of pretending it's zero? And the question a principal engineer always circles back to: when is eventual consistency genuinely unacceptable, and what do you do on those specific paths?

There's a version of this section that opens with the CAP theorem — consistency, availability, partition tolerance, pick two — and derives everything from first principles. That version is rigorous and a great way to clear a room. Here's the version that survives contact with production.

DynamoDB offers two kinds of read off the same data. An eventually consistent read goes to any replica and comes back in about a millisecond. A strongly consistent read goes to the leader, waits for confirmation that the replica is caught up, and comes back in about ten. Same API, same cluster, same rows. Ten times the latency, and you bought exactly one thing with it: the guarantee that you're not looking at data a few milliseconds stale. Amazon files this under "single-region" behavior — and note that even inside one region, with replicas sitting a few milliseconds apart, strong consistency still costs you a full order of magnitude.

Stretch it across regions and the numbers stop being cute. Netflix runs in three AWS regions — US East, US West, EU West. A strongly consistent write to US East that has to be acknowledged by all three carries a floor of roughly 120ms, the round trip to EU West, before any processing happens at all. On a pipeline moving hundreds of millions of events a day, that 120ms isn't a latency line item — it's the ceiling on throughput, because every request touching multi-region-consistent data waits behind every other request touching it, all of them queued on the slowest acknowledgment. And your availability now degrades every time EU West sneezes, because a write can't complete until the replica on the far side of an ocean confirms it. You've coupled your uptime to the worst-behaved network link you own.

The alternative is to accept the window. US East writes immediately, US West and EU West catch up asynchronously, and inter-region lag runs somewhere around 100–500ms. A user in Los Angeles sees a recommendation built on data a few hundred milliseconds old and notices nothing, because a 200ms-stale movie suggestion is, for every purpose that matters, identical to a fresh one. It's a suggestion, not a court filing. Meanwhile the write finishes in single-digit milliseconds, throughput is bounded by how fast you can compute rather than how far light can travel, and EU West can fall off the map without dragging the pipeline down with it.

That's the whole trade in one breath. Strong consistency buys you an invariant — you always see the latest write — and pays for it in latency and availability. Eventual consistency buys you speed and resilience and pays for it with a window where different observers disagree. At small scale the invariant is nearly free and genuinely nice to have. At large scale it gets structural, and more often than not it's guarding against a problem your actual users don't have.

So the question is never "which is better." It's "for this piece of data, how much staleness can the user actually absorb, and what does it cost me to give them less?" Ask an engineer that directly and they'll almost always answer correctly. The trouble is that nobody asks. The replica gets added on a Tuesday to clear a CPU alarm, and the consistency model of the whole system gets decided by an infrastructure ticket nobody thought to label as one.

I've watched teams back into eventual-consistency bugs three different ways, and the thing they share is that none of them is exotic. Each is the natural result of building a distributed system without ever once saying the words "consistency model" out loud.

Assuming the read sees the write. A user changes their display name. The write lands on the primary in US East. The app immediately re-fetches the profile to show the confirmation — and the fetch goes to a read replica, which replicates asynchronously. Same-region lag might be 20ms; the app fires its read in 5. The replica answers with the old name. So the user reads "Profile updated successfully" sitting directly above their old name, concludes the save didn't take, and hits save again.

This isn't a corner case; it's one of the most common user-facing consistency bugs there is. And it happens for an organizational reason as much as a technical one: the team that added the read replica was solving a load problem, the team that wrote the profile flow was solving a correctness problem, and the two of them quietly disagreed about what the word "database" meant. The replica is not the database. It's a lagging copy of the database wearing the database's name tag. Nobody wrote that distinction down, so nobody defended it.

Forcing strong consistency everywhere. The obvious fix: route every read to the primary. Now reads that never needed a freshness guarantee — reads that would happily serve data from last Tuesday — all pay the primary's tax. The replica existed because the primary was already overloaded; sending everything back to the primary returns you precisely to the condition you added the replica to escape, except now you've also written code to do it. At scale this doesn't just fail to help. It turns one node into the bottleneck for the entire system, cheerfully doing work you'd spent real money spreading across a fleet.

Writing to cache and trusting replication to catch up. The most seductive of the three: after the write, optimistically update the local cache so the user sees their change instantly, and trust the database to catch up before anyone looks too hard. It works beautifully for the user who stays put on the same page and the same server. It falls apart for the user who navigates away and back and lands on a different app server whose cache never heard the news; or who closes the app, wiping local state, and reopens to a fresh read from the still-lagging replica; or — the real prize — who makes a second edit before the first has propagated, so now the cache and the database hold two different truths and "which one wins" is a question your data model never anticipated being asked. You shipped a race condition and called it a feature.

What ties all three together is a single shared assumption: somebody wrote "database" on the architecture diagram, and everyone downstream assumed the word meant the same object everywhere. In a replicated system it doesn't. There are several copies, they are not guaranteed to agree at any given instant, and "database" on the diagram is a polite cover over a question — which copy, and how fresh? — that somebody has to answer on purpose.

Do everything right — sensible architecture, good intentions, replicas where they belong — and eventual consistency still hands you a few failure modes that are genuinely startling the first time you meet them. They're worth naming, because a thing with a name is a thing you can put in a design doc before it puts itself in a postmortem.

Read-your-writes inconsistency. A user files a support ticket, then clicks "View My Tickets" the instant the form submits. Their ticket isn't there. So they submit again. Now there are two, the support team is annoyed at the duplicate, the user is confused about which one to track, and engineering gets a bug report titled "the submit button doesn't work" — which is false in the specific way that makes it nearly impossible to reproduce, because the button works fine every single time you try it.

The gap is the usual one: write to primary, read from replica, replica hasn't caught up. The write was perfect. The fix has a name — read-your-writes consistency — and a couple of shapes: after a write, route that user's subsequent reads to the primary for a beat, or hand the read a version token the replica uses to stall until it's caught up. It costs complexity. For data the user just touched with their own hands, it's almost always worth paying.

The phantom update. A user changes their shipping address, and a minute later gets an order confirmation showing the old one. It reads like a bug, and it is one — just not the bug they think. The confirmation service fired before the address finished replicating. From the service's point of view it read the address correctly; it simply read the version that was true a moment ago. From the user's point of view, they updated their address and the system ignored them.

What makes this one nasty is that it's intermittent in a way that maps to nothing the user can see. If confirmation fires 500ms after the update and replication takes 200, nobody ever notices. If confirmation fires at 150ms and replication takes 300, everybody does. Which world you live in depends on replication topology, queue latency, and whatever the network happened to be doing that afternoon — which means the bug is reproducible only in aggregate, and the ticket fills up with "cannot reproduce" comments. Teams sometimes "fix" it by slowing the confirmation service down, which is not a fix so much as a bribe, or by relabeling it a race condition and closing the ticket, which is at least honest.

Conflicting concurrent updates. The deep one. Two users edit the same record at the same time against two different replicas that haven't yet synchronized. User A in Chicago updates a shared document at 14:00:01. User B in London updates the same document at 14:00:03 — but neither replica has seen the other's write, so as far as each is concerned, it holds the only edit. Replication runs, and now there are two writes that both claim to be the truth. Which wins? Who decides? If your data model doesn't answer that, your infrastructure will answer it for you — almost certainly by taking the higher timestamp, which is "last write wins," which may be the last thing you actually wanted. Your storage engine has opinions about your business logic, and it did not ask for yours.

The lightest tool in the box is causality: knowing which writes happened before which. If you know update B happened after update A, then a replica holding A but not yet B is in a sane intermediate state — it's just behind. A replica holding B but not A is in an insane state, and you'd very much like to know which one you're looking at.

Amazon's original Dynamo paper (2007) used vector clocks for exactly this. Each object carries a version vector — a little map from replica to counter, like {server_a: 3, server_b: 2}. Whenever a replica modifies the object, it bumps its own counter. When replicas exchange updates, they compare vectors. If A's vector dominates B's — every counter equal or higher — then A is strictly newer and you just take A. If neither dominates, the two writes are genuinely concurrent: a real conflict, on two replicas, with no causal thread connecting them, and now you need a resolution policy.

The reason this earns its keep is that it tells "B is newer than A" apart from "A and B have no idea the other exists." The first case resolves itself. The second needs a human decision encoded somewhere — and a resolution strategy you only invoke when you're certain the writes are concurrent can be far simpler, and far less paranoid, than one that fires on every merge just in case.

The catch, and it's the whole catch: vector clocks detect conflicts. They do not resolve them. They will tell you, with mathematical confidence, that two writes are in conflict, and then fall silent on what to do about it. Resolution is a policy decision that lives above the storage layer, in code you have to write, about a business question only you can answer.

If vector clocks detect conflicts, Conflict-Free Replicated Data Types (CRDTs) refuse to have them. A CRDT is a data structure whose operations are commutative and associative: the order updates arrive in, and which replica saw them first, doesn't change where you end up. Every replica can merge every other replica's state mechanically, with no coordination, and they all land on the same value. No locks, no leader, no asking permission.

The canonical examples teach the whole idea. An increment-only counter: each replica counts its own increments locally, and merge takes the max seen per replica — no matter who counted what in what order, the global sum comes out right. A grow-only set: each replica adds members locally, merge takes the union, and you've just built "add item to cart" with no conflict logic at all. A last-writer-wins register is also technically a CRDT, with the caveats we're about to get into.

This isn't exotic; it's underfoot. Redis Streams lean on these ideas to absorb concurrent writes without global coordination. Google Docs' operational-transform engine — the thing that lets two people type into the same paragraph without shredding each other's sentences — is conceptually a CRDT over ordered text. When collaborative editing feels effortless, this is the machinery quietly doing the impossible underneath.

The constraint is the part the enthusiasts skip past: CRDTs only work for structures with the right algebra. A profile with one display_name field is not a CRDT, because two concurrent edits to a name have no natural merge. You cannot union "Alice" and "A. Chen" into a coherent name; you have to pick, and picking is a policy, not an algebra. CRDTs carry you cleanly across the cases that fit their shape — counters, sets, certain registers — and leave you exactly where you started on the cases that don't, which includes most of the rich domain objects you actually ship. Wonderful tool. Narrow doorway.

When a conflict can't be avoided and can't be merged, the bluntest resolution is to keep the newer write and throw the older one away. Last-write-wins uses the write timestamp as the tiebreaker: display_name set to "Alice Chen" at 14:00:01 and to "A. Chen" at 14:00:03, keep "A. Chen." Done.

Cassandra uses LWW as its default. DynamoDB's eventually consistent mode does too. It's trivial to implement, trivial to reason about, and — the part the docs mention quietly, if at all — it loses data without a sound.

Here's where it bites. Picture a collaborative document where two people are working different fields. User A sets the title at 14:00:01. User B sets the body at 14:00:02. In a row-level LWW implementation, B's write — being later — overwrites the entire row, title included, and A's title evaporates. Neither user saw a conflict. Neither got an error. The row is simply different from what both of them believe they left it, and the divergence goes unnoticed until someone opens the doc next week and asks who deleted the title. The answer is nobody. The answer is the timestamp comparator, doing precisely what you configured it to do.

LWW is fine when the field is a single self-contained value with unambiguous semantics — a status flag, a preference toggle, one attribute owned by one user. It's quietly destructive the moment independent fields share a row and updates to them are semantically unrelated, which describes nearly every interesting object you'll ever model. Reach for it as a deliberate last resort with your eyes open, not as the default because it was the one setting you didn't have to change.

Sometimes the right place to enforce consistency isn't the database at all — it's the application. This sounds like throwing in the towel and is actually a legitimate, load-bearing pattern, so it's worth slowing down for.

Take hotel reservations. The inventory system serves reads eventually consistently: the browsing UI shows room availability that might be 200ms stale, because a room free 200ms ago is almost certainly still free, and forcing strong consistency on every page of a browse experience would kneecap it for no real benefit. But when a user actually books, the application stops trusting the cheap read. It does an explicit check: read the current reservation record strongly, straight from the primary; confirm the room is genuinely still available; write the reservation behind a unique constraint that turns a double-booking into a database error instead of a customer-service incident. Eventually consistent browsing and strongly consistent booking live in the same product, on the same data, because the booking path was built differently on purpose.

This pattern — eventual consistency for reads, strong consistency plus a constraint for the writes that matter — is how most B2C transactional systems actually work, whether or not anyone on the team uses those words. The skill is telling the three kinds of operation apart: the ones that are truly reads, the ones that are truly writes, and the treacherous middle category that looks like a read but is secretly a write — "check availability and show it to the user in a way that implies they can act on it." That middle category is where the bodies are buried.

The failure mode is erosion. A team starts with a clean read/write split, everyone understands it, life is good. Then someone ships a feature that reads stale data and presents it as a guarantee the user can act on — a "you can claim this" badge computed off a replica — and now the browse path is load-bearing in a way nobody designed it to be, and the next double-booking is one unlucky pair of users away.

Consistency Window vs. Latency

The consistency window is the oldest data a reader might see. A 10ms window means reads can be up to 10ms stale; a 5-second window means up to 5 seconds. Shrinking the window costs money and latency — faster replication, more synchronous writes, or both — and the cost curve gets steep as you approach zero.

The useful question is not "what's the smallest window we can build" but "what's the largest window our users can't tell apart from zero." For a social feed, five seconds stale is invisible. For a stock ticker, five seconds stale is a different price and possibly a lawsuit. For an account balance right after a transfer, five seconds stale is the most uncomfortable five seconds in the product. Those are three different requirements with three different consistency budgets, which strongly implies they should not be served off the same replication topology — and absolutely should not be governed by the same config value someone set globally in 2019 and nobody has touched since.

One calibration worth memorizing, because you'll reach for it in design reviews: same-datacenter replication runs about 10–50ms of lag; different datacenters in one region, 100–200ms; cross-region, 500ms to 2 seconds. These aren't knobs you tune. They're physics and infrastructure handing you a menu. The only thing you actually choose is whether your consistency requirement fits inside one of those windows — and if it doesn't, you change the requirement or you change where the data lives, because you are not going to renegotiate the speed of light this sprint.

The CAP Theorem, Applied

You know the theorem. Here's the one thing most explanations fumble: partition tolerance was never optional. Networks partition. You don't get to choose a world where they don't; you only get to choose what your system does when it happens. So the real fork — for any distributed system that has to keep running during a partition — is between consistency and availability, and nothing else.

Choose availability — keep serving even when a partition means some answers are stale — and you have chosen eventual consistency, whether or not you said so out loud. Most B2C systems make exactly this choice, on purpose or by inertia, because "the service is down" is a worse afternoon for everyone than "the data might be 500ms old."

Choose consistency — reject or block any request you can't serve with provably fresh data — and you have chosen to go partly unavailable during a partition. That's the right call for a small, important set of operations where stale data is actively dangerous: payment capture, inventory reservation against a hard limit, identity and access checks. For everything else, the user looking at a 2-second-old recommendation is fine, and the user staring at a 503 because you're waiting on EU West to confirm a write is meaningfully worse off than they were a second ago.

The principal-engineer move is to make this concrete before you reach for strong consistency on any path: name the specific bad outcome you're preventing. If the answer is "the user might see stale data," ask the follow-up — "and then what happens to them?" If the honest answer is "they see an old number for a moment, then the right one appears," congratulations: you don't have a consistency requirement. You have impatience, and impatience is much cheaper to satisfy than a quorum.

Amazon DynamoDB

DynamoDB reads eventually consistent by default. This wasn't a cost-cut or an oversight; it's a position Amazon argued explicitly in the original Dynamo paper and has held for nearly two decades. The reasoning is plain: for the overwhelming majority of Amazon's reads — product pages, preferences, session data, recommendation caches — a 200ms-stale read carries no business consequence anyone can measure, and the latency and throughput you get back in exchange are enormous.

For the reads where staleness does carry a consequence — billing, account status, anything touching real money or access control — DynamoDB offers strongly consistent reads at roughly 10x the latency, guarantee included. And critically, Amazon made it a per-request choice, not a per-table setting. That's the right granularity, because two different operations on the very same data routinely have different consistency needs, and forcing them to share one model is how you end up overpaying on one path just to adequately serve the other.

The lesson from DynamoDB is not "use eventual consistency." It's "make the choice explicit, at per-request granularity, for a reason you can state, and measure what it costs." Amazon's engineers can tell you exactly which code paths issue strongly consistent reads and why. If you're running DynamoDB and you can't, you're not necessarily wrong — but you're paying the strong-consistency tax somewhere you've never checked, and the meter is running.

Netflix Multi-Region

Netflix runs three AWS regions live at once, taking real traffic in all of them. A user in Chicago watches an episode; the viewing event writes to US East; the recommendation engine in US West will fold that view into their profile — eventually. Replication takes one to two seconds.

Netflix's stated position: two seconds of lag on recommendation data is completely fine. Recommendations are probabilistic signals, not ledger entries. Whether your profile reflects that one episode at second zero or second two does not change what gets suggested in any way a human could perceive. And the upside of letting it lag — serving every region from its own local replicas instead of reaching across an ocean on the read path — is the difference between a system that scales and one that doesn't. Netflix moves on the order of 500 billion events a day. Bolting two-second cross-region consistency onto that pipeline wouldn't be a quality improvement; it'd be an outage you scheduled in advance.

Payments are the other story entirely. Subscription billing, plan changes, payment-method updates — those run through strongly consistent paths, frequently with explicit two-phase coordination, because the cost of stale data there is denominated in dollars and chargebacks. One company, one product, two consistency models, drawn cleanly along the line of "what actually happens when this data is two seconds old." That's the lesson DynamoDB's per-request knob hints at and Netflix states outright: the right consistency model is a property of the operation, not the system. Your recommendation service can be eventually consistent. Your billing service cannot. They coexist without contradiction, as long as somebody decided on purpose rather than by default.

The Dynamo paper (Amazon, 2007). The founding document for eventual consistency in production. It pushed vector clocks, quorum reads, and "always writable" semantics into the mainstream; DynamoDB is its direct descendant and Cassandra is its independent cousin. Read it once, end to end — not for the implementation details, which have aged, but for the reasoning about trade-offs, which hasn't.

Vector clocks. Causality tracking across replicas. Each object carries a version vector; replicas compare vectors on merge; dominance means "one is strictly newer," incomparable means "genuine conflict, your call." Detects conflicts without coordination. Does not resolve them, and won't pretend to.

Merkle trees. The trick that turns "are these two replicas actually in sync?" from an impossible question into a cheap one. Cassandra uses them in anti-entropy repair: hash the data into a tree, compare root hashes between replicas, and if the roots differ, walk down only the branches that disagree to find the divergent ranges. Without it, finding which rows drifted means comparing every row, which at scale is a polite way of saying "never." With it, you locate divergence in O(log n) and repair only what's actually broken.

Exercise 1: Sizing the consistency window.

You're replicating from NYC to LA. The physical round trip is about 40ms. Replication is asynchronous, fire-and-forget. You have five replicas — three in NYC, two in LA. Design a strategy that gives 99% of reads data no more than 100ms old.

One approach is quorum reads: require at least 3 of 5 replicas to have seen a write before you call it committed, and make sure at least one LA replica is in that quorum, so a reader in LA always hits a replica that has the write. The cost is that write latency is now bounded by the slowest replica in the quorum, and you've traded throughput for the guarantee. Now characterize the trade honestly: what happens to write throughput when the NYC-to-LA link gets congested? (The thing you just made a dependency is the thing that's now having a bad day.)

Exercise 2: Consistency budget for a real feature.

Pick one feature you actually shipped recently. Map every read and every write in it. For each read: what data does it touch, what's the source of truth, and what's the real consistency model in production — not the one you assumed? For each write: how long until a replica read reflects it?

Then ask the uncomfortable question: if every consistency window is 5x larger than you assumed, what breaks — and who notices first, your monitoring or your users?

← Chapter 7 (Stream Processing). Streams produce eventual consistency by construction. When "user updated email" fans out to downstream services over Kafka, each one updates its own read model on its own schedule. The stream is the replication mechanism; the topic's retention policy sets how far behind a consumer can fall. Everything here about replication lag and consistency windows applies, unchanged, to consumer lag.

→ Chapter 9 (Event Sourcing). Event sourcing pushes eventual consistency all the way up to the architecture itself. The event log is the source of truth; read models are projections that trail behind it. Designing one means deciding, explicitly, which projections may be stale and by how much — the same decisions from this chapter, applied to a more radical model.

→ Chapter 12 (Sagas). Sagas coordinate multi-step distributed transactions where each step may read data left by earlier steps that's replicated, not freshly consistent. A saga that checks inventory in step 1 and charges a card in step 3 has to assume the inventory it saw in step 1 might have changed before step 3 ran. Reasoning about that correctly is just the consistency-model fluency this chapter was trying to build, cashed out under load.

The intuition to carry out of this chapter: eventual consistency is not a property of your system — it's a property of each operation. Your recommendation feed should be eventually consistent. Your billing service must not be. The same codebase, the same cluster, maybe the same database, running different consistency models on different paths — chosen deliberately, every time, off the one concrete question: what happens to a user who reads this data two seconds after it was written? Most production systems already run several consistency models at once. They just never named them, so nobody owns them. Name them. Measure the windows. Alert when the windows grow. And retire, permanently, the habit of letting "database" mean whatever each team needs it to mean that afternoon.

Next: Chapter 9 — Event Sourcing: when the event log is the source of truth, and the read model is a projection derived from history.