Idempotency

In the physical world, paying someone twice is a story. You notice, you wince, you make an awkward phone call, somebody refunds you, and it's over by lunch. The whole episode is embarrassing precisely because it's rare — the machinery of cash and handshakes mostly does each thing once.

Distributed systems don't have that luxury, and they're not embarrassed about it. The same request arriving twice isn't a freak event there. It's the infrastructure doing exactly what it was built to do. Networks time out. Servers die mid-sentence. Load balancers retry. Clients retry. The retry isn't the bug — the retry is the feature, the thing standing between you and a system that gives up at the first dropped packet. The bug is what happens when the retry lands and you process it as if it were new.

It's the contract that turns a retry from a gamble into a non-event — the second attempt simply lands on the result the first one already produced. And "harmless redundant work" is a thing you will gladly pay for in a distributed system, because the alternative is harmful redundant work, and that one shows up on someone's credit card statement.

Here's the part that isn't obvious despite the idea being simple — the part seasoned engineers still walk past:

It arrives looking exactly like the original, because it is the original — same bytes, same intent, resubmitted by a client that timed out and feared the worst:

1   first attempt   POST /orders   { item: 42, qty: 1 }
2   the retry       POST /orders   { item: 42, qty: 1 }
3                   └──────── byte-for-byte identical ────────┘
4 
5   server's question →  "Have I already done this?"
6   the request's answer →  (nothing — no field in here says so)

From the server's chair, "the request never arrived" and "I handled it and the reply got lost" look exactly the same. So everything in this chapter is really one job: giving the server a way to answer that question.

Networks fail in a specific, nasty way that's worth naming exactly. The failure mode is not "the request didn't arrive." That one's easy. The failure mode is "I don't know if the request arrived." That gap — between "I sent it" and "I got an acknowledgment" — is the room where duplicates are born. Client sends. Server processes. Server crashes before it can say so. Client waits, gives up, retries. Now there are two orders, two charges, two of whatever was supposed to happen once, and nobody did anything wrong.

This is not an abstraction that lives in a whitepaper. It shows up as:

• A customer charged twice who is now on the phone and unhappy.
• A warehouse that decremented inventory four times for one sale and a stock count nobody can reconcile.
• Three identical order confirmations in an inbox, and a buyer who genuinely can't tell whether they bought one thing or three.

To the people fielding those, these aren't "distributed systems problems." They're just problems, and they land on a desk attached to a name.

And here's why this is a principal-level concern rather than something you toss to the infrastructure team with a shrug: idempotency needs to understand your business. You cannot solve it at the load balancer. You cannot solve it in the network layer. There is no generic box you can buy that answers "is this a duplicate?" because the question is really "is this the same payment / order / inventory move I already did?" — and that knowledge lives only in the application. Every time it gets ignored, a senior engineer eventually inherits the incident, reconstructs what happened from two rows that look unrelated, and quietly wishes whoever wrote the original handler had read this chapter.

Here's the ground we'll walk together:

• Why innocent-looking handlers manufacture duplicates, and why every step that produced one was individually correct
• How to design systems that either can't produce duplicates at all or can spot and ignore the ones they get — nearly as good, and often cheaper
• The three patterns that see real production use, what each one buys you, and the trade-off that decides which fits your business
• The failure modes that don't show up in testing and sit as "weird" tickets for months (the mid-request timeout gets everyone)
• How Stripe, Amazon, PayPal, and Uber each solved it under different constraints — and why copying someone's architecture without understanding why they chose it is the fastest way to inherit their right answer to solve your wrong problem

Precision matters here, because "handle duplicates" is the kind of requirement that sounds done the moment you say it and means almost nothing. Vague requirements don't fail vaguely — they fail in precise, traceable ways, usually in production, usually on a weekend.

The scenario, step by step:

1. Client sends "create order."
2. Server receives it, validates it, writes the order, decrements inventory.
3. Server gets ready to respond.
4. Server crashes. Or the network between server and client drops. Or the response shows up in 31 seconds and the client's timeout was 30.
5. Client has no acknowledgment. It cannot tell "the server did the work and the reply got lost" from "the server never did anything." Both look identical from where it sits: silence.
6. Client retries, as any reliable client should.
7. Server creates a second order and decrements inventory a second time.

Drawn out, the trap is the gap where the acknowledgment should be:

1   CLIENT                                SERVER
2     │                                     │
3     │ ──①── create order ───────────────► │
4     │                                     ├─② write order, −1 inventory   ✓
5     │                                     ├─③ prepare response
6     │            ✗ no ack                 ╳─④ crash │ net drop │ 31s > 30s timeout
7     │ ◄─────────(silence)─────────────────┤
8     │                                     │
9     │ "did it work? no way to know."      │
10     │ ──⑥── create order (retry) ───────► │
11     │                                     ├─⑦ write order, −1 inventory   ✓  ← again
12     │                                     │
13     ▼                                     ▼
14            Result: TWO orders. Inventory decremented twice.
15            Nobody did anything wrong.

Every single step in that sequence was correct. The client is supposed to retry on timeout — that's how reliable systems work. The server is supposed to process the requests it receives — also how reliable systems work. Two correct behaviors, plus one unreliable network, equals one wrong outcome. Congratulations: you've met distributed systems. They're like this all the time.

Everyone learns that HTTP POST is non-idempotent and PUT is specified as idempotent, and somewhere along the way that hardens into a belief that the verb protects you. It DOES NOT. The HTTP spec is a convention, not an enforcer. Nothing stops you from writing a PUT handler that inserts a fresh row on every call, and nothing stops you from writing a POST handler that's perfectly safe to hammer. The HTTP method is an opinion about intent. Your database is what actually decides what happens. Trusting the verb is how you get a handler that's idempotent in the documentation and duplicating in the logs.

There's a related trap worth calling out, because it's the first thing people reach for and it feels responsible: timestamp-based deduplication. "Only process if we haven't seen this user request this amount in the last N seconds." It reads like diligence. It collapses the moment real traffic touches it — when server clocks drift apart, when two genuine requests land in the same second, when the per-user timestamp log you waved off as "lightweight" quietly grows into one of your largest tables. The timestamp is a stand-in for the question you actually care about — "is this the same request, or a different request that happens to look the same?" — and it cannot answer that question. A proper identifier can. Hold that thought; it's where the real solutions start.

Stripe makes the stakes concrete in a way that's hard to argue with. They process billions of payment API calls a year. If 0.01% of those produce a duplicate charge, that's millions of customers seeing money leave their account twice. Sit with that ratio: a rate that rounds to zero as a percentage is a five-alarm fire in absolute numbers.

"Accidentally charging users is not acceptable" isn't engineering melodrama — it's a statement about whether the company gets to keep existing. And it's why idempotency becomes a first-class concern at the exact moment a company starts to scale: the expected number of incidents grows faster than the percentage rate falls.

Let's steelman the things engineers reach for before they've been burned, because each one is reasonable right up until the day it isn't. (All code in this section lives in Appendix A — read the idea here, type it out there.)

"Just assume no duplicates"

The appeal is real: zero overhead, no extra code, nothing to reason about. You process whatever arrives, you trust timeouts to be rare, and on the odd occasion a customer calls, you fix it by hand. At low scale this genuinely works. The duplicate rate is low, manual cleanup is annoying but survivable, and the code stays clean.

The failure mode isn't a cliff — it's a ramp, which is exactly why it's dangerous. As traffic grows, the absolute number of duplicates grows with it. The five support tickets a month become fifty, then five hundred. Someone finally does the math on duplicate payments and discovers the cost of cleanup has lapped the cost of just building deduplication — except now you're building it while triaging three hundred angry customers.

Timestamp-based deduplication

It looks defensible on the screen (full code: Appendix A.1). Three problems surface in production, reliably, in this order:

• Clock skew. Your servers disagree about what time it is — usually by tens to hundreds of milliseconds, and exactly when you'd least like them to. Two servers handed the same logical request resolve "now" to two different instants. The dedup key that looked razor-sharp in your single-box test is fuzzy across a fleet.
• Intent collisions. A user who buys the same coffee twice, five seconds apart, is not a duplicate — they're a returning customer. Timestamp dedup cannot tell "same request, sent twice" from "two real requests, same amount, same user." Set the window wide and you block legitimate repeat purchases; set it narrow and you miss real duplicates. There is no window width that gets both right, because the window was never the thing you needed to measure.
• Storage. A per-user log of every timestamp in the last 24 hours, across millions of users, is not a small structure — and worse, it grows with user activity, not request rate, so it's unbounded in the direction you can't plan for. Teams have gone looking for what's eating their database and found that the "lightweight dedup store" had quietly become one of the largest tables they owned.

Request-ID deduplication

Now we're closer to right (full code: Appendix A.2). The client generates a unique ID per request, the server checks it, and returns the cached response on a repeat. This is the shape of every good solution. But this naive version has three failure modes that will absolutely find you in production:

• The client generates duplicate IDs. This sounds absurd until you've seen it. Client libraries with bugs. Random number generators seeded from the same value (the classic: a fleet of containers that all boot at the same instant and seed from the clock). Languages with a UUID implementation someone wrote on a Friday. If your dedup key can collide, two unrelated user actions get collapsed into one — and now the second user's payment is silently returned as the first user's result. You didn't prevent a duplicate; you invented a far more interesting corruption, the kind that takes a forensic afternoon to even believe.
• The TTL problem. Expire the cache after 24 hours and any retry that limps in at hour 25 gets processed fresh. For payments this is especially mean: a client that's been retrying slowly, or that had an outage and came back the next day, manufactures a duplicate just by being late. Your dedup cache is only as good as its coverage window — and choosing that window is a business decision wearing a technical disguise.
• The distributed race. That cached check quietly assumes there's one place to look. Put it behind multiple servers and a shared Redis, and the window between "check" and "write" becomes a race:
• Request lands on Server A → checks Redis: not found → starts processing (200ms).
• The retry arrives 50ms in, before A has written anything → lands on Server B.
• Server B checks Redis: not found (A hasn't written yet) → also processes it.

This race is rare. It is also real, and it is not a thought experiment — it's the direct consequence of the gap between "check the cache" and "write the cache" in a system where things move at network speed. Shrink the gap, shrink the odds. But rare is not never, and "rare duplicate charges" is not a phrase you want to be defending in a postmortem with the word "SLA" in the room.

This section exists because the failure modes that hurt aren't the obvious ones. The obvious ones you catch in testing on the first afternoon. These are the ones that sit as open tickets for months, filed under "weird," until someone finally connects two dots that were never on the same page.

Expired deduplication state. You cache dedup info for 24 hours. A client has an outage, comes back on day two, and retries its last queued request. Cache miss. Processed again. You now have a duplicate charge timestamped 26 hours after the original — and your duplicate-detection monitoring, which was tuned to catch rapid-fire retries, sleeps right through it, because a 26-hour gap looks nothing like the retries you modeled. It surfaces in support as "I got charged twice but weeks apart," and the investigation means correlating two rows in your orders table that share nothing obvious.

The fix is not "keep state longer." Longer means more storage, and some requests genuinely should be reprocessable after a while — the monthly subscription renewal looks identical to last month's and is absolutely not a duplicate. The real fix is to model the idempotency window in business terms, not to pick a TTL and pray it's big enough. (If "the window is a business decision, not a technical one" sounds familiar, it's the same lesson the rate-limiter taught about reset intervals in Chapter 1. The windows in this book are almost never as technical as they look.)

Distributed dedup races under load. The race from the last section is rare at low RPS. At 10k RPS it's roughly a daily occurrence. The tell is subtle: duplicate records born 50–200ms apart — exactly the window between "check" and "write" plus network jitter. Under normal load you basically never see it. Under peak load you see it consistently, and here's the cruel twist: peak load is when cache writes take longer and when retries arrive faster, so the two factors that cause the race both get worse at the same moment. The failure is load-correlated, which is a polite way of saying it refuses to reproduce in staging and only performs for a live audience.

Idempotency key misuse. Stripe returns HTTP 400 if you reuse a key with different parameters, and they do it on purpose, because the alternative is worse: silently handing back a cached response for a different amount, a different recipient, a different currency. The bug behind this is almost always client-side — a key generated from a hash of partial request data, where the hash shifts the moment a field changes before retry. The error "idempotency key reuse" is reliably baffling to the engineer who hits it, because it looks like their retry logic is broken when the actual culprit is their key generation. (The error message is correct. The engineer's first three theories are not.)

The mid-request timeout. This is the one that has surprised more good engineers than any other on the list, because it slips through the gap between "check" and "record."

The handler checks the dedup cache — miss. It processes the request and calls the payment processor. The processor call times out after five seconds. The transaction went through on their end; the response just got dropped on the way back. So the handler doesn't know the result, and — reasonably! — declines to write a definitive entry to the dedup cache, because it has nothing definitive to write. The retry arrives. Checks the cache — miss, because the first attempt never recorded anything. Calls the payment processor again. Second charge goes through. You've now billed the customer twice, and your dedup cache has zero record of either charge, because both invocations exited without a clean result. The cache that was supposed to be your safety net was never given anything to hold.

Three patterns see real production use. They differ on complexity, latency, and whether idempotency is designed in or bolted on. Code for all three is in Appendix A; the bodies below stay in plain language on purpose, because the idea is the part worth carrying around in your head.

The client generates a unique identifier for each logical operation and sends it along. The server checks whether it's seen that identifier before. If yes: return the cached result, regardless of what's in the current request. If no: process, cache, return. (Full code: Appendix A.3.)

Two things about that handler aren't obvious until someone gets them wrong:

• You return the original result even if the params changed. If a retry shows up with a different amount, a different recipient, a different anything — you return the original result. You do not reprocess with the new values. The idempotency key is a claim that this is the same operation, and you take that claim at face value. If the client genuinely wants something different, they generate a new key. If they reuse a key with different params, that's a client error, and a well-built API says so out loud (HTTP 422 or 400, naming key reuse). Silently honoring the new params would let a client edit a transaction by retrying it with different numbers — a feature nobody asked for and a vulnerability everyone will regret.
• You cache before responding, not after. The order is: check → process → write cache → respond. Reverse the last two and a retry that arrives in the gap sees a miss and processes again. On a relational database the cache write belongs inside the same transaction as the work. With Redis in front of a relational store, write the durable store first and treat Redis as a read-through layer — so losing Redis costs you latency, not correctness.

Where it shines:

• API transactions where the client drives the retry — payments, order creation, anything where the same client retries with the same identifier it generated.
• It's why Stripe uses this shape for nearly everything in their API.

Where it struggles:

• Background jobs and queue-based processing, where the "client" is a message and the broker — not the original caller — may redeliver it.
• The broker knows message IDs, not your idempotency key. You either thread the original key through the message (deliberate design) or reach for a different pattern.

Instead of bolting dedup onto a handler, you design the handler so that running it twice produces the same outcome as running it once. The business logic is inherently idempotent: the database enforces uniqueness via a primary key or unique constraint, a duplicate hits the constraint, you catch the exception, and you return the existing record. (Full code: Appendix A.4.)

No dedup cache. No TTL. No race between "check" and "write," because the database lock is the serialization. This is the most elegant form of idempotency, because there's nothing extra to it: the business key that uniquely identifies the order is the dedup key, and the database's uniqueness guarantee is the dedup mechanism. You're not adding a layer — you're finally using the one you've been paying for all along.

Amazon's DynamoDB usage is the clean example: order creation does a put_item with ConditionExpression='attribute_not_exists(order_id)'. Condition fails, the item exists, return it. Condition succeeds, it's new, proceed. One call, atomic, inherently idempotent. (Full code: Appendix A.5.)

The failure mode worth planning for: what if the create half-succeeds before erroring? DynamoDB's atomic conditional write gives you clean semantics — the item exists or it doesn't, never a half-born state. In a relational database spanning multiple tables, "natural idempotency" usually means wrapping the write in a transaction that either fully commits or fully rolls back, so your "does it exist?" check always sees either a complete record or nothing — never a torso.

Where it shines:

• Operations with a stable, meaningful unique identifier — order IDs, payment IDs, booking IDs.
• When you'd rather lean on a database constraint you already trust than maintain a cache you'll have to babysit.

Where it struggles:

• Operations with no natural identity — "increment this counter," "append this log line," "process this event." There's nothing to collide on, so you assign an ID upstream or pick another pattern.
• Multi-table writes, where you need a transaction so the existence check never sees a half-built record.

For high-value transactions where "rare duplicate" is not an acceptable failure rate, both patterns above share one gap: they're check-then-act, and check-then-act always has a race window, however small. At high enough stakes, you don't want almost-exactly-once. You want exactly-once, and you're willing to pay for it.

The move is to acquire a distributed lease on the idempotency key before processing, so only one server can hold the key at any instant. The lock goes to ZooKeeper, etcd, or an equivalent consensus system; while it's held, no other server can touch the key. (Full code: Appendix A.6.) The race isn't shrunk — it's eliminated, by serializing access instead of narrowing the window.

The bill comes due as latency: a Paxos or Raft round-trip, 10–50ms per request, minimum. For a processor doing 50k RPS that's meaningful added latency and a brand-new availability dependency. If ZooKeeper goes down, your payment processing goes with it — unless you've built a graceful degradation path, which you should, but now that's more code to write and more code to be wrong.

PayPal's choice is instructive: distributed locks for transactions above a dollar threshold, lighter-weight cache-based dedup below it. The consensus cost scales with the cost of being wrong. For a $10 purchase, a 0.001% duplicate rate is an annoying support ticket. For a $100,000 wire, a 0.001% duplicate rate is a single event that costs more than a year of consensus infrastructure. The threshold where distributed consensus becomes worth it is a business calculation, not a computer science one.

Where it shines:

• High-value transactions where "rare duplicate" is unacceptable — large transfers, anything you cannot cleanly reverse.
• When you need exactly-once and can name, in dollars, why almost-exactly-once isn't good enough.

Where it struggles:

• Everyday high-throughput traffic — the 10–50ms round-trip is a tax levied on every request, including the overwhelming majority that never needed it.
• Availability: the consensus system becomes a dependency that can take payments down with it, so now you owe it a degradation plan too.

The honest default for most teams is the idempotency key with a durable backing store, reaching for natural idempotency wherever the domain hands you a unique ID for free, and escalating to consensus only on the specific high-stakes operations that can name their dollar figure. Running everything through consensus is wasted latency; running a $250,000 transfer through cache-only dedup is an unhedged bet. Match the pattern to the cost of being wrong.

Consistency vs. Latency

This is the core axis, and it does not resolve cleanly no matter how long you stare at it.

• Natural idempotency with unique constraints: no cache, sub-millisecond overhead, strong guarantees straight from the database. Only works when you have a stable unique key. Useless for operations without one.
• Cache-based dedup with Redis: fast (1–5ms), simple, works across a fleet. Eventually consistent — lose the Redis write before it persists and a retry can slip through. Needs a durable backing store if Redis is cache-only. Expiry becomes a correctness bug the instant the TTL is shorter than the retry window.
• Distributed consensus: strong consistency, exactly-once, 10–50ms overhead, real operational complexity, and a new way to fail (the consensus system itself). Worth it only when a duplicate costs meaningfully more than the latency and complexity.

Stripe's synthesis is the one most teams should copy: cache-based for the common path, with Postgres as the durable backing store so a Redis miss falls through to something persistent instead of opening a gap. The TTL is long enough that retries almost always hit cache. And their key handling catches what the cache can't — different params on the same key return a 4xx instead of silently doing the wrong thing.

What Needs to Be Idempotent?

The question with more leverage than it first appears: does idempotency apply only to the primary operation, or to every side effect?

The honest answer is that true end-to-end idempotency — payment processed once and confirmation email sent once and webhook fired once and analytics event recorded once — is extremely hard and demands coordination at every layer. In practice, nobody does this perfectly. Anyone who says they do has a side effect they haven't audited yet.

The pragmatic answer, which is what Stripe and Uber actually ship, is that the primary transaction is idempotent and side effects may run more than once. The confirmation email might fire twice. The webhook might deliver twice. That's acceptable when the side effect is naturally harmless (a second email is annoying, not corrupting) or when the downstream consumer dedups on its own (webhook receivers are expected to be idempotent).

Where it bites is the side effects that are neither harmless nor visible. An analytics pipeline that double-counts produces numbers that quietly steer product decisions wrong. An inventory decrement that runs twice oversells before anyone notices. These don't page anyone — they accumulate, which is worse.

Idempotency Key Lifecycle

A question that surfaces late in the design review and deserves to surface early: who generates the key, and when?

• Client-generated keys are the standard for synchronous APIs. The client mints a UUID before the request and reuses it for every retry of that operation. Clean — when the client is a human-controlled app with a clear notion of "this operation."
• Message-queue keys are the awkward case. There, the "client" is a consumer and the message was produced by someone else entirely, so the message itself must carry an idempotency key from the moment it was produced — which means producers have to be careful about key generation, and the key has to survive whatever transformations the message goes through in transit. Lose the key in a reshape and you're back to square one with extra steps.
• Server-assigned keys move correctness to the server: it issues a key after the first request and the client uses it for retries. Sounds appealing until you notice the client now needs a two-phase dance (get key, then use key) — and if the first request times out, the client never got a key and is right back where it started, only now confused about it.

Three diagrams worth drawing on a whiteboard when you explain this to your team:

Diagram 1 — Idempotency Key Request Flow. Client → Server with Idempotency-Key header. Server: check cache → miss → process → write cache before responding → respond. Second request: check cache → hit → return cached result, no processing. Draw the cache-before-respond ordering explicitly; the whole correctness argument lives in that order, and making it visible makes the constraint impossible to forget.

Diagram 2 — Natural Idempotency with Unique Constraint. Client → Server → Database (INSERT with unique constraint). First request: insert succeeds, return new record. Second: insert fails on the constraint, catch, return existing. Note there's no dedup cache in the picture — the constraint is the mechanism.

Diagram 3 — The Distributed Dedup Race. Two servers, one Redis. Request hits Server A: check Redis (miss) → processing begins (200ms). Retry hits Server B 50ms later: check Redis (still miss, A hasn't written) → B also processes. Both write to Redis. Two records in the database. This diagram is the entire motivation for the "pending" entry and for distributed locks — drawing the race is what makes the fix feel obvious instead of paranoid.

Stripe: Idempotency Keys with Postgres Backing

Stripe's implementation is public because they've written about it, and it's worth studying precisely because the edge cases they handle are the ones that bite everyone building something similar.

Every endpoint that creates or modifies state accepts an Idempotency-Key header. The key and the response body are stored in Postgres. Redis is a read-through cache in front: most requests hit Redis, miss falls through to Postgres, the result is written back. Lose Redis and you degrade to slightly higher latency, not to a correctness failure — which is the entire reason the durable store sits behind the cache.

The rules they enforce read like a list of scars:

• Same key, same params: return the cached response. The ordinary retry.
• Same key, different params: HTTP 400, error code idempotency_key_reuse. The client made a mistake and should mint a new key. Silently returning the old result would be a wrong answer dressed as success.
• Same key, request timed out mid-processing: the pending state. Stripe writes a "pending" marker before calling the payment processor; if the request times out without a result, retries find the marker and poll for completion rather than reprocessing. This is the mid-request-timeout fix from the failure-modes section, in production.
• Key expiry: 24-hour TTL. After that, same key = new request. Any retry older than 24 hours gets reprocessed, and Stripe accepts that as extremely rare and cheaper to resolve by hand than to store dedup state forever.

The detail that doesn't get enough attention: they monitor cache hit rates as a health signal. A sudden drop in idempotency cache hits means clients are generating new keys on retry instead of reusing old ones — which means someone's retry logic is broken. This catches a class of client bug before it shows up as duplicate charges weeks later. It's the rare piece of monitoring that tells you about a problem in someone else's code.

Amazon: Natural Idempotency at the Order Layer

Amazon's order creation is idempotent by design, not by bolted-on dedup. The order_id is generated client-side (or assigned at the very first step of checkout, before any external calls) and serves as both the business identifier and the dedup key. ConditionExpression='attribute_not_exists(order_id)' does the work — first write creates the order, later writes with the same ID fail the condition and return the existing one. No cache to manage, no TTL to reason about, no lock.

The design decision that actually matters is where the order ID is born. Generate it after some processing — after the fraud check, after the inventory reservation — and idempotency arrives too late, with multiple in-flight requests all racing toward the write. Amazon mints the ID before any external call, so the uniqueness check is the first meaningful thing that happens. Dedup at the start of the transaction, not the end.

What this demands structurally: every service handling order creation must be stateless with respect to order IDs. The ID rides in on the request; it is not generated inside the handler. Any change that mints a new ID inside the handler silently breaks the guarantee, because two requests for the same logical operation now produce two different IDs and both happily succeed. That's the kind of one-line refactor that looks harmless in review and reopens an incident six weeks later.

PayPal: Tiered Consistency Based on Transaction Value

PayPal runs two dedup strategies in parallel and switches on transaction amount.

Below a threshold (public knowledge puts it around $1,000, though the exact figure varies by product and jurisdiction), they use Redis-backed key caching with Postgres behind it — essentially Stripe's shape. Fast, slightly eventual, and it covers the overwhelming majority of transaction volume.

Above it, they take a distributed lock via ZooKeeper before processing. The lock guarantees one server per transaction, and the race vanishes. The cost is latency: a consensus round-trip at PayPal's scale adds 20–50ms. For a $250,000 wire, that latency is not up for discussion. For a $5 coffee, it would be a checkout regression people actually notice.

The insight is that the two failure modes have wildly different costs, so they get wildly different treatment. A duplicate $5 charge is recoverable and rare enough that support is cheaper than running consensus on every small transaction. A duplicate $250,000 transfer is not recoverable in any operational sense, and the consensus latency is priced against the risk of the event, not the raw cost of the infrastructure.

Uber: Partial Idempotency and Acceptable Side Effects

Uber makes explicit what most companies do by accident: not everything needs to be idempotent.

• Ride request creation is idempotent. The ride_request_id is generated before submission, stored with a unique constraint, and the handler returns the existing request on duplicate. Primary operation idempotent by design.
• Driver notifications are idempotent by contract. A driver who gets multiple notifications for one ride — because the notification system retried — checks the ride_id and ignores the dupe. Drivers are expected to dedup on their end.
• Rider SMS is allowed to duplicate. That service does not guarantee exactly-once delivery, so a rider might get two "Your Uber is arriving" texts. Uber's judgment: acceptable. Mildly confusing, not harmful. Making SMS globally idempotent would need coordination with the SMS provider that isn't worth the outcome.
• Analytics dedups downstream. Events may double-count on a retry, and the pipeline dedups on (ride_id, event_type). The producer doesn't try to prevent duplicates; the consumer is built to absorb them.

A quick tour of the tools you'll actually reach for. Full, annotated code for each is in Appendix A; here's what each one is for and where it'll bite you.

• Redis — the standard cache layer for idempotency keys. SET key value EX 86400 NX (set only if absent, with a 24-hour expiry) gives you atomic check-and-set, which closes the distributed race: two servers racing to write the same key get "succeeded" and "rejected" respectively, and the loser knows to return the cached value. (Full code: Appendix A.7.) The caveat everyone forgets: Redis is usually deployed for speed, not durability. Run with appendonly no, restart the node, and your cache evaporates — and a miss is a potential duplicate. Fine for short-window retries; for Stripe-grade guarantees, put a durable store behind it.
• PostgreSQL — durable dedup storage for when cache loss is unacceptable. The key is a PRIMARY KEY, which gives you both the uniqueness constraint and an O(log n) lookup; an index on expires_at powers a cleanup job that prunes old rows. (Full code: Appendix A.8.) Skip the cleanup and the table grows unbounded with historical request data — the timestamp-log trap from earlier, wearing a nicer schema. The cost: every cache-miss request writes before responding, adding write latency to your p99. For payment APIs, worth paying.
• DynamoDB — natural idempotency via conditional writes. The conditional write is atomic at the DynamoDB level: the item exists and you get ConditionalCheckFailedException, or it doesn't and your write lands. No separate check, no window. One call on the happy path, two on the retry path. (Full code: Appendix A.5.)
• ZooKeeper / etcd — distributed locks when consensus is required. Use ephemeral nodes (ZooKeeper) or lease-based keys (etcd) so the lock disappears automatically if the holder dies — no indefinite lock-hold when a server falls over mid-transaction. (Full code: Appendix A.9.) The critical failure mode: if the lock service is unavailable, acquisition fails and the request fails. That's the right behavior — better to fail cleanly than to process without the lock — but your retry logic has to handle it and your SLA has to have room for it. Processing without the lock to dodge the wait is not an option, no matter how reasonable it sounds at 3am.

These don't have clean answers, which is the point. Paste any of them into an AI and you'll get a confident, well-organized response in four seconds that reads beautifully and skips the only thing that matters: your system, your constraints, your blast radius. The value is in the arguing, not the answer. Sit with each one long enough to disagree with your first instinct before you go find something to agree with you.

Exercise 1: The Timing Hole

Walk through this:

1. Client sends a request with Idempotency-Key: abc-123.
2. Server processes it (takes 5 seconds).
3. Server writes to the dedup cache.
4. Server sends the response.
5. The response takes 31 seconds in transit due to congestion. Client timeout is 30.
6. Client times out, retries with Idempotency-Key: abc-123.
7. Retry arrives.

Does dedup work? Yes — assuming step 3 happens before step 4. The retry finds the entry written in step 3 and returns the cached response.

Now reverse 3 and 4: the server responds, then writes the cache. The client times out before the response lands (same 31-second delay), retries, and the retry arrives before the cache write completes. Miss. Duplicate processing.

The fix: the cache write must precede the response, in the same transaction or atomically with respect to it. In practice — write to your durable store, then return. Redis as your only cache layer is insufficient unless its writes complete synchronously before the HTTP response goes out, which most implementations don't guarantee and most engineers assume they do.

Exercise 2: The Migration

You're migrating your idempotency store from Redis-only to Postgres-backed, running both during the transition (dual writes: new requests write Redis and Postgres; reads check Redis first, fall through to Postgres).

A client sends a request on day 1. It hits Server A, which writes Redis only (the Postgres write hasn't rolled out there yet). On day 2, the client retries. The retry hits Server B, which checks Redis — evicted, 24-hour TTL, original was 25 hours ago. Falls through to Postgres — no entry, because the original write never reached Postgres. Server B processes it. Duplicate.

Design the migration sequence that prevents this.

The answer is migrate reads before writes. First, roll out Postgres reads (as fallback) with no Postgres writes — now every server checks Redis then Postgres, and Postgres is harmlessly empty but wired correctly. Second, roll out Postgres writes — both stores now populated. Then wait one full TTL (24 hours) before decommissioning Redis reads, so every live key has had a chance to land in Postgres. Only then drop Redis from the read path. The ordering guarantees no miss ever falls through to an empty Postgres before Postgres is fully populated.

Exercise 3: The Scope Problem

You have an idempotent payment endpoint, backed by a dedup cache, working correctly. The payment triggers a notification (SMS: "Payment received!") via an async message on a queue.

A consumer crash-and-restart causes the message to be delivered twice. The customer gets two texts.

Fix it without making the notification system idempotent end-to-end.

• Option A: thread the idempotency key through the message payload; the notification consumer dedups on it — if it's already sent for this key, skip. Adds coordination, keeps the fix inside the notification system.
• Option B: accept the duplicate delivery and rate-limit notifications per customer per event — "one payment confirmation per payment ID, regardless of deliveries." Idempotency by another name, scoped to the observable outcome (one SMS) rather than the mechanism (one message processed).
• Option C: make queue delivery idempotent at the broker (Kafka, SQS dedup IDs). Tradeoff: broker-level exactly-once is expensive and has latency implications.

The right answer depends on your org. Option A needs cross-team coordination. Option B is self-contained. Option C is infrastructure cost. Name the binding constraint, then choose — and notice that "which is technically best" was never the deciding question.

This chapter and the retry chapter are the same problem from opposite ends. Retries are why idempotency matters; idempotency is what makes retries safe. A system with one and not the other doesn't really have either — you've got retry logic that will eventually manufacture duplicates, or idempotency that never gets exercised because nothing retries. They only make sense as a pair.

The message queue chapter revisits idempotency in a different shape: at-least-once delivery in Kafka, SQS, and friends means consumers receive duplicates by design, not by failure. The patterns here apply directly — natural idempotency in the consumer, a dedup cache keyed on message offset or ID — but the failure modes shift, because now the "client" is the broker, and the broker is correct to deliver more than once. You're not defending against a bug; you're absorbing a guarantee.

The saga pattern (Volume II) requires idempotency at every step, because a saga that fails halfway re-executes from its checkpoint — so every step must be safe to run again. The patterns here are the building blocks. Read this chapter, then the saga chapter, and the second one stops feeling like magic.

Dead letter queues exist partly because idempotent processing sometimes fails repeatedly. An idempotent consumer that can't process a message ships it to the DLQ for inspection — and the DLQ retry has to be idempotent too, because you'll replay that message several times while debugging it. This is the recursive property of the pattern: once you commit to idempotent processing in one place, everything feeding it needs to be idempotent too, or you've just moved the duplicate problem one hop downstream and given it a new name.

The key intuition from this chapter: in a distributed system you don't get to choose between "requests process once" and "requests process sometimes more than once." That choice was made for you by physics and packet loss. The only choice you actually get is between "sometimes more than once, with visible corruption" and "sometimes more than once, with the correct outcome anyway." The second one is idempotency. It isn't optional at scale — it's just a question of whether you've formalized it, or whether you're going to find out the hard way how much you hadn't.

The chapter keeps the code out of the narrative on purpose. Here it is, collected, runnable, and annotated. Each entry is the full version of something described in plain language in the body — read the prose for the idea, come here when you want to type it out. All of it is illustrative rather than production-hardened: the naive versions are here to be broken, and even the good patterns skip edges (auth, observability, error taxonomy) you'll meet in the real thing.

A.1 — Timestamp-Based Deduplication (the naive trap)

1request_time = headers["X-Request-Time"]
2if not db.has_recent_request(user_id, request_time):
3    db.create_order(user_id, amount)
4    db.log_request(user_id, request_time, now())
5    return new_order
6else:
7    return cached_response

Fails three ways in production: clock skew across servers, intent collisions (a real second purchase looks identical to a duplicate), and unbounded per-user storage. The timestamp is a proxy for "is this the same request?" and it can't answer that question. Don't ship this.

A.2 — Request-ID Deduplication (naive)

1request_id = headers["X-Request-ID"]
2 
3cached = db.get_cached_response(request_id)
4if cached:
5    return cached
6 
7result = create_order(user_id, amount)
8db.cache_response(request_id, result)
9return result

The right shape, but this version has three holes: clients can generate colliding IDs, the cache TTL can expire before a slow retry arrives, and the gap between get_cached_response and cache_response is a race in a distributed deployment. A.3 is the hardened version of this idea.

A.3 — Idempotency Key Deduplication (Pattern 1)

1idempotency_key = headers["Idempotency-Key"]
2 
3cached = db.get_idempotency_response(idempotency_key)
4if cached:
5    return cached.response  # Return original result, even if params changed
6 
7result = process_request(request)
8db.cache_idempotency_response(idempotency_key, result, ttl=24*3600)
9return result

Two non-obvious rules: return the original result even if the retry's params differ (reused key + different params is a client error → 4xx), and write the cache before responding (on a relational DB, inside the same transaction).

A.4 — Natural Idempotency via Unique Constraint (Pattern 2)

1def create_order(user_id: str, amount: int, order_id: str) -> Order:
2    existing = db.query(
3        "SELECT * FROM orders WHERE id = %s",
4        order_id
5    )
6    if existing:
7        return existing  # Already created; return it
8 
9    order = db.create(
10        "INSERT INTO orders (id, user_id, amount) VALUES (%s, %s, %s)",
11        order_id, user_id, amount
12    )
13    return order

The unique constraint on id is the dedup mechanism — no cache, no TTL. For multi-table writes, wrap this in a transaction so the existence check never sees a half-built record. (The select-then-insert shown here is readable but still races under concurrency; in production, lean on the constraint and catch the duplicate-key exception, as in A.5.)

A.5 — Natural Idempotency in DynamoDB (Pattern 2)

1try:
2    table.put_item(
3        Item={'order_id': order_id, 'user_id': user_id, 'amount': amount},
4        ConditionExpression='attribute_not_exists(order_id)'
5    )
6    return {"status": "created", "order_id": order_id}
7except ConditionalCheckFailedException:
8    existing = table.get_item(Key={'order_id': order_id})
9    return {"status": "exists", "order": existing['Item']}

The conditional write is atomic — no check-then-act window. Either the item is new and the write lands, or it exists and you catch the exception, read it, and return it. One call on the happy path, two on the retry path.

A.6 — Distributed Lock with Consensus (Pattern 3)

1idempotency_key = headers["Idempotency-Key"]
2 
3with distributed_lock(idempotency_key, ttl=60):
4    cached = db.get_idempotency_response(idempotency_key)
5    if cached:
6        return cached.response
7 
8    result = process_request(request)
9    db.cache_idempotency_response(idempotency_key, result, ttl=24*3600)
10    return result

The lock serializes access to the key, eliminating the race instead of shrinking it. The cost is a consensus round-trip (10–50ms) on every request and a new availability dependency. Reserve it for high-value operations that can justify the tax.

A.7 — Idempotency Key in Redis (atomic SET NX)

1SET idempotency:key-abc-123 '{"status": "created", "order_id": 999}' EX 86400 NX

NX makes the set conditional on the key not already existing; EX 86400 sets a 24-hour expiry. Two servers racing get "succeeded" and "rejected" — the loser returns the cached value. Atomic, but only as durable as your Redis config: with appendonly no, a restart drops the cache and a miss becomes a potential duplicate.

A.8 — PostgreSQL Durable Dedup Store

1CREATE TABLE idempotency_responses (
2    idempotency_key VARCHAR(255) PRIMARY KEY,
3    response_body   JSONB NOT NULL,
4    created_at      TIMESTAMP NOT NULL DEFAULT now(),
5    expires_at      TIMESTAMP NOT NULL
6);
7 
8CREATE INDEX ON idempotency_responses (expires_at);

PRIMARY KEY gives uniqueness plus an O(log n) lookup; the expires_at index powers a cleanup job. Skip the cleanup and the table grows unbounded — the same trap as the timestamp log, nicer schema. Every cache-miss request writes before responding, so this adds to your p99; for payments, worth it.

A.9 — ZooKeeper Distributed Lock

1with zk.Lock(f"/idempotency/{idempotency_key}", identifier="server-1"):
2    if not db.get_cached(idempotency_key):
3        result = process_request(request)
4        db.cache(idempotency_key, result)
5    return db.get_cached(idempotency_key)

Ephemeral-node locking: if the holding process dies, the lock disappears automatically, so a crashed server doesn't hold the key forever. If ZooKeeper is unreachable, acquisition fails and so does the request — the correct outcome, but one your retry logic and SLA both need to expect.

Next: Chapter 3 — Multi-Level Caching: making things fast without making them wrong.