Multi-Level Caching

If you have one cache, you have a race condition. If you have two caches, you have a consistency problem. Large-scale systems solve this with multi-level caching — and then spend the next several years living with what they solved it with.

That first line gets quoted in design reviews like it's a koan. It isn't. It's a warning label that somebody peeled off the box before you started.

Caching is the first thing you reach for when performance goes sideways, and it works — gloriously, the first time. You drop Redis in front of the database, latency falls from 80ms to 2ms, the hit rate settles at 90%, and for about a quarter everyone agrees you're a genius. Then traffic doubles. Hit rate slips to 85%. You add memory, claw it back to 87%, and feel fine about it right up until you do the arithmetic: 13% of 500k requests per second is 65k misses per second, your database tops out around 10k, and you have just discovered that you didn't solve the problem so much as relocate it somewhere more expensive.

The naive move is to add another cache. The sophisticated move is also to add another cache — the difference is that the second time you know exactly what you're buying and what you're paying for it.

Because every layer you add buys you throughput and bills you for it later. The invoice arrives in three parts: a consistency problem, an invalidation problem, and a warming problem. These are not items for the "tech debt" backlog you'll get to never. They are live failure modes, and they have the manners to show up only at peak load, which is the one moment you have no spare attention for them. The cache stampede — we'll get to it — is named with a specificity that tells you somebody stood there and watched it happen. Somebody did. I have. It sounds exactly like the name implies, except louder, and at 3am.

Here's the part that catches engineers who genuinely know caching cold: the failure modes are load-dependent, which means they are also staging-proof. Your system looks immaculate under normal traffic. It comes apart precisely when traffic spikes, which is precisely when the on-call phone rings and someone wants a root cause in thirty seconds. Cache bugs don't misbehave in the test environment where you'd catch them. They're patient. They wait for an audience.

The scale problems are real, and the companies that have them are specific. Netflix computes a personalized feed for 200 million-plus users with an ML pipeline that takes minutes per run — so you can't recompute on demand (too slow) and you can't cache everything forever (too big, too stale), and the entire interesting part is the word "and." Amazon's purchase patterns follow a Zipf distribution so lopsided that the top 0.1% of products eats a wildly disproportionate share of traffic, which means the strategy that's correct for the long tail actively hurts the head if you're naive enough to treat them the same. Spotify has 100 million tracks and streams maybe a thousand of them for something like 70% of plays on a given day, which tells you most of that catalog is, on any given Tuesday, furniture.

All three run multi-level caching. None of them run it the same way, because the problems aren't the same shape. That's the whole chapter: learning the shapes well enough to recognize the one you actually have, instead of the one the blog post you read had.

Here's the ground we'll cover:

• Cache-aside, write-through, and refresh-ahead — and when each one earns the complexity it costs
• Negative caching — the pattern nobody implements until the bots arrive and explain why they should have
• Cache warming — how to come back from a cold start without taking the database down a second time
• Consistency models across layers — how stale is too stale, and the part everyone forgets: too stale for whom
• How Netflix, Amazon, and Spotify built their hierarchies, and why the differences aren't accidents
• What a principal engineer actually has to decide the moment someone says "we should add another cache layer"

Do the arithmetic precisely, because intuition lies to you here and does it with a straight face. Take a system where:

• Your database handles 10k QPS
• Your cache handles 100k QPS
• Your application layer is seeing 500k QPS

At a 90% hit rate, 50k requests per second miss and land on the database. That's five times what the database can do. Your system is already in trouble — you just can't see it yet, because an overloaded database doesn't refuse requests, it queues them, and queue latency reads as "a bit slow" on the dashboard right up until it reads as "everything is on fire." Slow is just fast's way of not telling you yet.

So you do the obvious thing and chase the hit rate. More memory, tuned TTLs, you get to 95%. Now it's 25k misses per second — still 2.5x capacity. You bought time, not a solution, and you'll keep buying time at a worse and worse exchange rate, because eventually you're caching everything worth caching and the only misses left are genuinely cold data with no warm version to serve. There is no setting on the single-cache dial that gets you out of this.

This is the moment a second cache layer earns its keep. It also, in the same gesture, hands you a problem.

The Consistency Problem

You write to the database. Now you have to invalidate the cache. Fine — but which cache? L1 (local to each app server), L2 (regional Redis), both? Invalidate L1 and forget L2, and users on different app servers see different versions of reality. Invalidate both, and you've opened a window between "L2 cleared" and "L1 cleared" where some servers serve new data and some serve old. This is consistency skew, and the maddening thing about it is that nothing is broken. Both copies are behaving exactly as designed. They just disagree, the way two correct watches set to different time zones disagree.

At small scale this is a curiosity. At large scale it's a support ticket reading "my profile photo updated on my phone but not my laptop," and your honest first instinct — admit it — is to assume the user is wrong. They aren't. You are. The bug is in the seam between two systems that are each individually fine, which is the most expensive place a bug can live.

The Cache Miss Cascade

A miss is expensive, and not because the database is slow in absolute terms. 100ms is not slow. 100ms becomes slow when a thousand people are standing behind it. A popular item expires; 1,000 concurrent requests all want it; all 1,000 queue behind one 100ms database call. Some of them give up and time out. Their clients, being clients, immediately retry. Now the database is serving the original thousand plus their retries, the queue is longer than it was, and you've taught your own traffic to attack you. This is the thundering herd, and yes, it was named by someone watching it from the inside.

The Warming Problem

Every cache is born empty. Restart Redis after an incident and your hit rate goes from 90% to 0% in the time it takes the process to come up — and the first few thousand requests now fall straight through to a database that is already exhausted from the incident that made you restart Redis in the first place. So it cascades. And here's the cruel symmetry of it: the cascade is a second outage, caused entirely by fixing the first one.

I've watched a team restart a cache after an incident — visibly relieved, the worst behind them — and then watched database latency climb to 500ms inside sixty seconds as the cold cache let everything through. The first outage was bad luck. The second one was self-inflicted, and everyone in the room knew it, which is a specific and memorable flavor of silence.

"Just buy a bigger Redis."

The reasoning is clean: more memory, higher hit rate. And it's true, for a while, and then it quietly stops being true and nobody updates the reasoning. Past a certain size you're paying to cache data that gets touched once a week. Eviction pressure isn't even the problem — the cold data isn't fighting anything, it's just sitting there, never evicted, never read, holding memory hostage. Hit rate flattens out; cost keeps climbing in a straight line; somebody six months from now asks why the Redis bill looks like that.

The confusion worth naming: engineers quietly equate "cache is full" with "cache is working." They are unrelated facts. A full cache at 85% hit rate doesn't have a size problem, it has a strategy problem, and more RAM cannot fix a strategy problem any more than a bigger parking lot fixes the fact that you keep parking the wrong cars in it.

"Add Redis everywhere, let each layer manage itself."

Redis at the app tier. Caching at the API gateway. Query caching at the database. Each layer cheerfully caches whatever it happens to see, with no idea the others exist. You have just built a consistency nightmare and distributed the blame so thoroughly that no single component is at fault.

Watch it play out: you update the database. The app-tier Redis gets invalidated, good. But the gateway cache never heard about the write — it caches HTTP responses, not database mutations, and as far as it knows nothing happened — so it keeps serving the stale response for another sixty seconds. Your user sees the change in the mobile app (which hits the app tier directly) and not on the web (which goes through the gateway), and now you're debugging a bug that only exists in the gap between two caches that have never been introduced to each other.

"Just use TTL for everything."

Set a 60-second TTL. Data's stale for a minute, tops. What's the worst that happens?

For critical data — account balances, order status — sixty seconds of staleness is wrong in the way that ends up in a deposition. For trivial data — trending rankings — sixty seconds is also wrong, just in the opposite direction: you're recomputing a ranking every minute that hasn't meaningfully moved in hours, burning compute to confirm nothing changed.

But here's the part that gets left out of the design doc every single time: all keys with the same TTL expire at the same instant. Populate 10,000 keys at startup, all with TTL=3600, and in exactly one hour you have 10,000 simultaneous expirations and the thundering herd you were worried about — except you didn't need a popular item to trigger it, you needed coordinated expiry, and you poured the coordination into the foundation yourself.

The fix is TTL jitter: a small random offset on every TTL so expirations smear across time instead of detonating together. It is one line of code. It is left out constantly, and it is always re-learned the same way, which is to say expensively.

Cache Stampede

The platonic failure mode — the one every other one is a variation of. A popular key expires. In the gap before anything repopulates it, 1,000 concurrent requests all miss and all hit the database in the same breath. The database, which was handling 8k QPS without complaint, suddenly sees 9k. It slows. Requests time out. Clients retry. Next second: 18k QPS, and the trend is not your friend.

The signature is unmistakable once you've seen it and bewildering before then: regular, periodic load spikes on a perfect interval that happens to match your TTL. Pull up the database CPU graph and it's a sawtooth — a tooth every 60 seconds, each one a little taller than the last, marching toward the moment the system tips over. Engineers seeing it for the first time burn a couple of hours ruling out cron jobs, deploys, and the moon before someone says "wait, what's our TTL?"

Consistency Skew

User A updates their profile. The write handler invalidates L1 (app-server local). It does not invalidate L2 (regional Redis), because the invalidation code was written back when there was only an L1, and adding L2 was a different ticket, owned by a different person, six months later. User A's next request lands on a server with a warm L1 and sees the new profile. User B lands on a different server, misses L1, falls through to L2, and sees the old one.

This arrives as "I changed my name but my coworker still sees the old one," and the first investigation turns up absolutely nothing — because every cache is behaving correctly according to its own logic. The bug isn't in any component. It's in the coordination between them, which is the one place your debugger can't set a breakpoint.

Cache Pollution

You cache everything. Memory fills. LRU starts evicting whatever it deems unpopular — and "unpopular," to LRU, means "not touched recently," full stop. The trouble is that your most important data might be batch-computed and consumed in waves, not sipped continuously. LRU evicts the batch result during a quiet stretch, the wave arrives, and you take a full miss on the single most expensive thing in the system to recompute. LRU did its job perfectly. Its job was just the wrong job for that key.

Bigger cache doesn't save you — it postpones the eviction, but the wave is almost always bigger than the headroom you bought, so you've spent money to be wrong slightly later.

Write Amplification

You go write-through: every write hits the cache, which then propagates to the database. Cache writes are sub-millisecond; database writes are 5–20ms; so your write throughput is now governed by the database and the cache is just idle, writing at database speed, contributing nothing on the write path. Every bit of read throughput you gained, you quietly handed back on writes.

This is the one that passes design review with flying colors and fails the load test. Every sentence in the design is true. The write does go through the cache. The cache does propagate. The contract is honored to the letter. And the performance is terrible, because correctness and performance are different promises and you only wrote one of them down.

Negative Cache Misses

A user looks up a username that doesn't exist. You don't cache the result, because — reasonably enough — why would you cache nothing? The next lookup for that same username hits the database. Still nothing. You are now spending database capacity, over and over, to re-confirm a fact you already established: that this thing isn't there.

At normal volume that's just waste. Point a bot at it — enumeration, checking which accounts exist — and it's a denial-of-service vector wearing a login form. The database gets pummeled with lookups for accounts that were never going to exist, at a rate it can't sustain, while your cache sits a few feet away faithfully caching every successful lookup, doing its job beautifully, helping with precisely the wrong half of the traffic.

The default, and the one you've almost certainly already written. On a miss, the application reads the database, populates the cache, returns the result. The cache sits "aside" from the main flow — it's nowhere near the write path. In code it's exactly that sentence: check the cache, and on a miss read through to the database and backfill the entry before returning. (Full code: Appendix A.1.)

What makes it appealing: it's simple enough that any engineer can read it cold and know what it does. The cache is optional — if it falls over, the app just talks to the database and nobody downstream notices. Any data, any TTL, no ceremony.

What makes it fail: the thundering herd when a popular item expires. The consistency window between writing the database and invalidating the cache. The cold-start cliff after a restart. The same three problems, every time.

The non-obvious confusion point: engineers dutifully add write-side invalidation — cache.delete(f"user:{user_id}") after an update — and feel finished. They are not finished. They've invalidated one of the keys that holds this user's data. If you also cache user_by_email:{email}, that copy is now stale, indexed under a name the invalidation never thought to look up. "Cache invalidation" earned its spot on the short list of hard problems in computer science not because the delete is hard, but because remembering everywhere you put the thing is hard — and you will be sure you got them all right up until a stale one surfaces in production.

When to use it: read-heavy data that changes rarely and forgives a few seconds of staleness. User profiles, article bodies, product descriptions — the stuff where "slightly old" costs you nothing.

Instagram's timeline caching is cache-aside at its core: a timeline is computed and cached, a miss just regenerates it, and for 99% of reads the cache answers in under a millisecond. The miss is survivable, so they let it be survivable instead of engineering it away.

The name is the mechanism: a write goes through the cache on its way to the database. Where cache-aside kept the cache off to the side of the write path — writes hit the database directly and the cache only found out later — write-through drops the cache right into the middle of it. Every write updates the cache and the database as one logical operation, so the cached copy can never drift behind the database. Picture the write path as a pipe from your application down to the database; write-through welds the cache into that pipe, and nothing reaches the database without updating the cache on the way past.

The payoff lands on the read side: a read never misses on data you just wrote, because the instant the write returns, the new value is already sitting warm in the cache. Compare that to cache-aside, where a write touches only the database and the cache stays stale until the next miss repopulates it (or your invalidation logic gets around to it). Write-through buys away that staleness — and pays for it up front, on every single write, which is the catch the rest of this section is about.

In code it's a write to the cache, then the database, with the cache rolled back if the database write throws — so a failed write can't leave the cache holding a value the database never accepted. (Full code: Appendix A.2.)

What makes it appealing: reads are always fast and always fresh, the cache is warm for every key anyone ever wrote, and the thundering herd can't touch recently-modified data because it never expired out from under you.

What makes it fail: your write throughput is now capped by the database, not the cache, and the cache-then-database ordering means a cache hiccup blocks the entire write. For a write-heavy system, write-through turns your fast cache into an expensive relay that adds a hop and removes nothing.

The failure mode that deserves a sentence of its own: if the database write fails and your rollback isn't careful, the cache is now holding data the database has never heard of. Congratulations — your cache has become the source of truth by accident, which is a thing that can happen to you but should never be a thing you chose.

When to use it: write-heavy systems where staleness is a correctness bug, not an annoyance. Account balances, order status, anything where reading the old value has a direct business consequence. Stripe runs write-through for payment state for exactly this reason — a stale read of a payment record isn't a UX wrinkle, it's money being wrong, and money being wrong is the one thing a payments company is not allowed to do.

Instead of waiting for the miss, you refresh the data before it expires. Background jobs watch TTLs and repopulate proactively, so a popular item is never actually absent when someone asks for it. In code it's a normal read with one addition: on a hit, check the remaining TTL, and if the entry is close to expiring, enqueue a non-blocking background refresh while still returning the current value — a true miss (cold start or eviction) pays the full recompute latency. (Full code: Appendix A.3.)

What makes it appealing: miss latency on popular items simply goes away. Hit rate stays high under load, and the thundering herd never forms, because the entry doesn't expire while it's being actively used — you keep topping it up.

What makes it fail: wasted compute, and a new dependency you might not notice you've taken on. You're refreshing data for users who may not come back before the next refresh, and at scale your background job system has quietly become load-bearing. The day it falls behind — and it will, on the day you least want it to — entries start expiring before they're refreshed, and you are now standing in the exact thundering herd this whole pattern existed to prevent, except surprised about it.

The non-obvious part: refresh-ahead only works if you know which items are worth refreshing. Refresh everything and you're running your most expensive computation continuously, demand or no demand. Refresh only recently-accessed items and now you need to track recent access — which is its own data structure, with its own memory cost and its own bugs. The pattern sounds like a one-liner and turns out to be a small system living inside your cache, with all the maintenance that implies. Nobody puts that in the estimate.

When to use it: expensive-to-compute data with predictable access. Netflix uses refresh-ahead for the top slice of user recommendations: the ML pipeline takes 2–5 minutes, so recs are cached for an hour and a background job re-runs the pipeline at the 50-minute mark — meaning an active user is, in effect, never the one who pays the recompute cost. Someone always pays it; Netflix just makes sure it's a background job and not a person staring at a spinner.

Cache the fact that the lookup found nothing. In code it's ordinary cache-aside with a third case: a hit on the sentinel returns None without touching the database, a real miss checks the database and stores either the result or the sentinel, and only a genuine value earns the long TTL. (Full code: Appendix A.4.)

The implementation detail that bites everyone: you can't use None for "this doesn't exist," because your cache also returns None for "I've never heard of this key." Those are two completely different facts — "I looked, it's not there" versus "I haven't looked yet" — and if you collapse them into the same value, you'll re-query the database on every supposedly-cached miss and wonder why negative caching did nothing. Hence the sentinel: a value that explicitly means I checked.

The invalidation requirement: when a user actually does get created, you have to clear the negative entry. Skip that, and your newest, most excited users can't log in — the cache reports, with total confidence, that they don't exist. This has happened. In the real world. To real people. And it takes an embarrassingly long time to diagnose, because every part of the signup flow looks correct and the cache is lying politely and consistently.

When to use it: anywhere "not found" is a common query. Login by email (negative caching keeps failed attempts off the database). Username availability checks. Fraud systems probing known-bad identifiers. Email providers negative-cache nonexistent addresses aggressively, specifically to take the profit out of enumeration attacks — if checking 10,000 addresses costs the attacker the same as checking one cached "nope," the attack stops being worth running.

Consistency vs. Availability

Write-through gives you consistency and takes your write throughput as payment. Cache-aside gives you availability — a cache failure is survivable — and charges you in stale reads. There is no free option in the middle; there's only the question of which bill you'd rather pay.

Stripe picked write-through for payment state because a stale read there is a correctness bug that touches money, and they can absorb the write-throughput cap because payment writes were never the bottleneck. For the product catalog — descriptions, pricing tiers — they use cache-aside, because a 30-second staleness window on a product blurb hurts no one. Same company, two patterns, chosen per how much the data costs you when it's wrong. That's the actual skill: not picking a pattern, but pricing the wrongness.

Latency vs. Memory

Each layer adds a hop, but a hop that hits saves you an expensive trip downstream — so you're trading memory and complexity for latency. Three or four layers is typical at scale: L1 (app-server local), L2 (edge CDN or regional), L3 (central Redis). Past four, the coordination cost of keeping them consistent starts to outrun the latency you're saving, and you're adding layers that cost more to keep honest than they save in speed.

Hit Rate vs. Complexity

Cache-aside, a junior engineer can debug on their second day. Refresh-ahead with event-driven invalidation needs someone who holds the whole picture in their head: the background job system, the key namespace, the invalidation fan-out, and the fallback path for when the background system lags. That's not free. You pay it in on-call load, in the length of incident investigations, and in the look on a new hire's face when they ask "wait, where does this data actually come from?"

Netflix's recommendation cache is genuinely, defensibly complex — because the alternative, recomputing recommendations per request, would put a multi-second stall on every page load, and that's a worse product. The complexity is buying something a user can feel. The trap is copying the complexity without copying the scale that justified it. At a tenth of Netflix's traffic, the simple version usually wins, and the elaborate version is just a monument to a problem you don't have yet.

Invalidation Cost

Write-through: cheap — you invalidate inline, in the same breath as the write. Cache-aside with TTL: cheapest — you don't invalidate at all, you just wait it out. Event-driven invalidation: expensive — now you need an event bus, an invalidation service, fan-out logic, and monitoring for all of it.

Amazon uses TTL for most product data and event-driven invalidation only for price changes, and the logic is exactly as blunt as it should be: a wrong price needs to vanish fast, because a product on sale at the wrong number generates refunds, angry tweets, and customer-service hours; a wrong product description can drift for a few minutes and nobody on Earth notices. They spent the expensive mechanism on the data that punishes you for being slow, and the cheap one everywhere else. That's not a clever architecture. It's just refusing to pay for consistency where staleness is harmless.

Netflix: Refresh-Ahead for the Top Percentile

Netflix's recommendation system is expensive in the most literal sense: the ML models that build a per-user recommendation set take minutes to run, which means computing them at request time is off the table before the conversation even starts. So they're cached. The interesting questions are the ones that start after that: for how long, and what happens at the moment they expire?

Netflix tiers the strategy by how active you are:

• Highly active users (top percentile by session frequency) get refresh-ahead. A background service watches TTLs and re-triggers the pipeline before entries expire, so the people who use Netflix most never see a miss — which is also, not coincidentally, the population most likely to notice one.
• Occasional users are cached without refresh-ahead. If the entry expires before they come back, the system falls back to less-personalized recommendations — popular titles in your genre — while the personalized set recomputes. One slightly-worse session. Netflix looked at that trade and took it, because warming the cache for someone who shows up monthly is compute spent on a maybe.
• Dormant users get evicted entirely past a threshold. Haven't logged in for six months? The system isn't going to keep your recommendations toasty on the off chance. That's not neglect; it's a correct read of the odds.

The operational cost is the catch: that background refresh system is itself load-bearing infrastructure. When it falls behind — pipeline fleet pegged, queue backing up — TTLs start expiring before refresh catches them, and your most active users start seeing generic content. So Netflix monitors refresh lag as a first-class SLA signal, because the cache hit rate is downstream of the refresh system's health, and pretending otherwise just means finding out during an incident.

The insight worth stealing: the cache and the system that fills it are both production infrastructure. Build the cache and treat the population pipeline as a background nicety, and you've built something that looks correct in the diagram and comes apart at peak — when the diagram isn't the thing answering pages.

Amazon: TTL Plus Event-Driven Invalidation

Amazon's catalog has a property that makes the caching decision genuinely interesting rather than rote: most of a product's attributes change slowly (description, images, category), but a few change fast and consequentially (price, inventory, review count). Same record, two completely different temperaments.

Treat everything the same and you're stuck choosing between two bad options:

• Long TTL: prices go stale, customers see the wrong number, chargebacks and apologies follow.
• Short TTL: everything refreshes constantly, and the cache is nearly useless for the 90% of attributes that didn't change.

So Amazon splits it. Most product data: 1-hour TTL, plain cache-aside. Price changes: published to an internal event stream, where an invalidation service is subscribed and deletes the affected cache entry the moment a price moves. The next request misses, fetches the live price, and moves on.

That's three pieces of infrastructure — the event stream, the invalidation service, and the monitoring that confirms the invalidation service isn't quietly lagging. The invalidation service is the quiet dependency, the one nobody thinks about until it's behind: when it lags, prices stay stale longer than the design allows, so Amazon treats it as part of the price-change pipeline's SLA, not as a footnote in the caching code.

The confusion point for engineers: the invalidation service looks optional. The TTL expires the entry eventually anyway, so why build the whole apparatus? True — but "eventually" on a 1-hour TTL means a product can sit at the wrong price for up to an hour, and at Amazon's volume that hour has a dollar figure attached that you could write on a whiteboard. The invalidation service is justified by that number shrinking. The lesson isn't "build an invalidation service." It's "do the multiplication for your own system before you decide you don't need one."

Spotify: Multi-Level with Geographic Distribution

Spotify's traffic is skewed so hard it's almost a gift to the cache designer: 100 million tracks in the catalog, but on any given day a few thousand of them account for most of the streams. That skew isn't a problem to manage — it's leverage to exploit, if you build for it.

Their architecture, per their public engineering writing, runs roughly:

• L1: Client cache. Your phone caches the last several hours of audio and metadata. Most replays never touch the network — the cheapest request is the one that stays on the device.
• L2: CDN edge. Audio for the hot tracks lives on edge nodes near users' ISPs. A stream of "Blinding Lights" gets served from a box a few miles away, not from Spotify's origin halfway across a continent.
• L3: Regional Redis cluster. Per-user personalization — recently played, "made for you," liked songs — cached in your home region. A Swedish listener's data is served from Stockholm, not from wherever the central database happens to live.
• L4: Central database. The long tail nobody upstream has seen. The top thousand tracks essentially never reach this far down — by the time you're at L4, you're answering a question that's genuinely rare.

The whole design is built around a single fact about the data — access is wildly non-uniform — and the response is to let different tiers serve different parts of the distribution instead of pretending it's flat.

What makes it non-trivial is the geography. Replicating L3 regionally means writes have to propagate, and propagation takes time. You update a playlist on your phone in Stockholm; the write goes to the Stockholm cluster; meanwhile you're physically in Singapore, reading from a Singapore cluster that's a beat behind. For most playlist operations, a beat behind is invisible and fine. For the operations where it isn't — synced playlists, collaborative playlists where two people are editing the same thing — Spotify reaches for more aggressive invalidation. The decision is made per feature, not stamped across the whole system, because "how stale is too stale" has a different answer for "your liked songs" than for "the playlist you and your friend are both editing right now."

Redis is the workhorse for L2/L3. SET key value EX 3600 for TTL-based cache-aside, DEL key for explicit invalidation. The caveat that catches people: Redis ships tuned for speed, which means appendonly no by default, which means your cache evaporates on restart. For a cache you can cheaply repopulate, fine. For one whose cold start is genuinely painful — recommendations, expensive aggregations — you either turn on persistence or you have a warming strategy ready before the restart, not improvised during it.

Memcached beats Redis on raw simple key-value throughput precisely because it does less — no rich data types, no persistence, no scripting, just cache. At very high QPS over simple values it can edge Redis out. Most teams still run Redis anyway, because the throughput gap is smaller than the cost of operating two different cache systems, and "we run two caching technologies" is a sentence that sounds fine in a meeting and miserable on call.

CDNs (Cloudflare, Fastly, CloudFront) are L2 caches living at the network edge, serving content from somewhere physically near the user and shaving 50–100ms off long-distance requests. Their invalidation is slow — propagation can take minutes — so they're for content that changes on the order of hours, not seconds. Try to use a CDN for fast-moving data and you've just bought yourself a globally distributed staleness problem.

In-process caches are your L1 — Python's functools.lru_cache, Java's Guava Cache, or a plain dict with a timestamp. Fast (no network hop), small (process memory), and per-process (every app server keeps its own, with all the consistency-skew fun that implies). Right for reference data that barely changes and is cheap to re-read when a process restarts. Wrong for anything that has to agree across servers.

Exercise 1: The Hit Rate Problem

Your system has: - 500k application QPS - Cache hit rate: 85% - Cache miss rate: 75k QPS hitting the database - Database capacity: 10k QPS - You're at 7.5x database capacity on misses alone

Without adding database capacity, design a strategy to get database-bound misses under 10k QPS (a 98% hit rate).

Options: 1. Add an L2 layer (regional Redis) to catch misses before the database 2. Add refresh-ahead on the top 1% of most-accessed keys 3. Add negative caching (if a meaningful share of misses are "not found")

Which is right? It depends entirely on what's missing: - Misses bursty and clustered on popular-item expiry → refresh-ahead on the hot keys - Misses scattered across keys that don't exist → negative caching - Misses on cold data a local L2 would have caught → second tier

The exercise has no answer until you know the distribution of your misses — which is the actual lesson. "Add a cache layer" without first asking what is missing is how you add complexity and watch performance stay exactly where it was.

Exercise 2: The Consistency Hole

You've got write-through caching. A user updates their email:

1. Cache updated with the new email
2. Database write attempted
3. Network failure — the database write fails
4. Cache rollback attempted (cache.delete)
5. Network failure — the cache delete also fails

Now the cache holds the new email, the database holds the old one, and every read returns the new email — until the entry expires, the app reads from the database, and the data silently rolls back to the old value. The user changed their email, saw it stick for an hour, and then watched it un-change itself.

How do you prevent this?

Hint: the database write and the cache rollback need to be atomic — or the cache should be written after a confirmed database write, not before. The ordering cache.set → db.write → (rollback if needed) has the failure mode above. The ordering db.write → (if success) cache.set doesn't — but it means reads between the write and the cache update see stale data. Both orderings trade something. Name the trade for each, and decide which failure your system can actually live with. There's no ordering that has neither problem; there's only the one whose problem you prefer.

Exercise 3: Cache Warming After Disaster

Your Redis primary fails. The cluster fails over to a replica that's been 200ms behind on writes. The replica becomes primary carrying 200ms of stale data. The old primary recovers and rejoins as the new replica. You now have two nodes that diverged.

Design the warm-up and reconciliation:

1. Which node's data is authoritative?
2. How do you handle the staleness in the new primary during that 200ms gap?
3. How do you prevent the thundering herd when clients reconnect to a cold cluster?
4. At what point is it actually safer to flush everything and warm from the database?

There's no clean answer — it turns on your TTLs, your database headroom, and whether 200ms of staleness is better or worse for you than 0ms of cache (a full cold start). The entire point of the exercise is to have this argument on a whiteboard, while calm, instead of in an incident channel, while not.

← Chapter 1 (Rate Limiting): A high miss rate under load is exactly the scenario rate limiting exists to survive. When refresh-ahead lags and misses spike, rate-limiting the database-bound path is the difference between graceful degradation and a cascade. The two are complements, not alternatives — one caps the throughput, the other reduces the demand.

← Chapter 2 (Idempotency): Cache invalidation needs to be idempotent. If your invalidation handler fires twice — and with at-least-once delivery off an event bus, it will — the second firing has to be harmless. cache.delete is idempotent by nature. cache.decrement is emphatically not, and the difference is a bug you'll only find under load.

→ Chapter 7 (Feed Architectures): Personal feeds lean on multi-level caching hard — pre-computed feeds in Redis, popular content in the CDN. The consistency questions from this chapter — how stale can a feed item be? — graduate from implementation detail to first-class design decision.

→ Chapter 8 (Search Systems): A search index is a cache — it stores a derived representation to make queries fast. The refresh-ahead and invalidation patterns here map straight onto index freshness. When a product changes, how fast should search reflect it? Same question, different vocabulary.

→ Chapter 10 (CQRS): The read side of CQRS is, essentially, a cache that grew up and got a schema. Populating and invalidating that read model is the same set of problems from this chapter, one level of abstraction higher and wearing a nicer suit.

The intuition to walk away with: caching isn't a solution to a performance problem — it's a trade of one set of problems for another, hopefully cheaper, set. The database problems (latency, throughput) get smaller. The cache problems (consistency, invalidation, warming) appear in their place. At sufficient scale you need both, which means you have to manage both, which means the comforting mental model — "the cache is a thing that sits in front of the database" — is the model that gets teams hurt. The teams that do this well treat the cache as a system in its own right: its own failure modes, its own observability, its own SLA. The teams that don't are the ones squinting at a database CPU graph that sawtooths every 60 seconds, certain something is wrong, and not yet realizing they built it.

The chapter keeps the code out of the narrative on purpose. Here it is, collected, runnable, and annotated. Each entry is the full version of something described in plain language in the body — read the prose for the idea, come here when you want to type it out. All of it is illustrative rather than production-hardened: the patterns skip the edges (auth, observability, error taxonomy, metrics) you'll meet in the real thing.

A.1 — Cache-Aside (Lazy Loading)

1def get_user(user_id: str) -> User:
2    result = cache.get(f"user:{user_id}")
3    if result is None:
4        result = db.query("SELECT * FROM users WHERE id = %s", user_id)
5        cache.set(f"user:{user_id}", result, ttl=3600)
6    return result

The cache is optional by construction — delete the first three lines and the function still returns the right answer, just slower. That property is the whole appeal: a cache outage becomes a latency event, not a correctness one. The traps live off-screen — invalidating every key that holds this user's data, and the stampede when a hot key expires.

A.2 — Write-Through with Rollback

1def update_user(user_id: str, data: dict) -> User:
2    cache.set(f"user:{user_id}", data, ttl=3600)
3    try:
4        db.write("UPDATE users SET ... WHERE id = %s", user_id, data)
5    except Exception:
6        cache.delete(f"user:{user_id}")  # Rollback cache if write fails
7        raise
8    return data

Watch the ordering and the rollback: cache first, then database, and if the database write throws, undo the cache so it can't end up holding a value the database never accepted. Drop the rollback and the cache silently becomes a source of truth you never designed. Exercise 2 walks through what happens when the rollback itself fails.

A.3 — Refresh-Ahead

1def get_recommendations(user_id: str) -> list:
2    value = cache.get(f"recs:{user_id}")
3    if value is None:
4        # Cold start or evicted — pay the miss latency
5        value = recommendation_pipeline.compute(user_id)
6        cache.set(f"recs:{user_id}", value, ttl=3600)
7        return value
8    
9    time_until_expiry = cache.ttl(f"recs:{user_id}")
10    if time_until_expiry < 600:  # Expiring within 10 minutes
11        # Schedule non-blocking background refresh
12        background_queue.enqueue(refresh_recommendations, user_id)
13    
14    return value

A normal read plus one branch: when the remaining TTL drops under the threshold, enqueue a background refresh and return the current value anyway, so the entry never expires out from under an active user. The cost is hidden in background_queue — that queue is now load-bearing infrastructure, and the day it lags, the thundering herd you avoided comes right back.

A.4 — Negative Caching with a Sentinel

1CACHE_MISS_SENTINEL = "__DOES_NOT_EXIST__"
2 
3def get_user_by_email(email: str) -> User | None:
4    result = cache.get(f"user_email:{email}")
5    
6    if result == CACHE_MISS_SENTINEL:
7        return None  # Cached miss — don't touch the database
8    
9    if result is None:
10        # True cache miss — check the database
11        db_result = db.query("SELECT * FROM users WHERE email = %s", email)
12        if db_result is None:
13            # Cache the non-existence
14            cache.set(f"user_email:{email}", CACHE_MISS_SENTINEL, ttl=300)
15        else:
16            cache.set(f"user_email:{email}", db_result, ttl=3600)
17        return db_result
18    
19    return result

The sentinel does the real work: it separates "I looked, it's not there" from the cache's own "I've never heard of this key" — both of which are otherwise None, and collapsing them means re-querying the database on every supposedly-cached miss. Don't forget to delete the sentinel when the record is actually created, or your newest users get told they don't exist.

Next: Chapter 4 — Distributed Locks: How to coordinate work across processes without creating a new single point of failure.