Request Coalescing

Request coalescing is one of the most underrated patterns in distributed systems — not because it's obscure, but because the engineers who need it most are usually too busy fighting the symptom to notice the cause. A popular cache key expires at peak traffic, latency spikes, the dashboards go red, and the reflex is to add more cache, add read replicas, or tune TTLs with the quiet optimism of someone rearranging deck chairs. None of it helps, because none of it is the problem. The problem is that a hundred concurrent requests for the same row all went to the database in the same instant, and nothing in the system was built to stop them.

The pattern is conceptually simple, which is part of why it's so easy to skip: when multiple requests arrive for the same in-flight operation, only one of them does the work. The rest wait. When the work finishes, all of them get the same result. From the database's point of view, there was one client. From the users' point of view, there were a hundred, and every one of them got served. That gap — one query, a hundred happy callers — is the whole trick.

Everyone who hits this ends up naming it themselves, which is why it answers to so many names. Go ships it as singleflight, and Rust borrowed the name wholesale. Java buries the same behavior inside Guava's LoadingCache. Python, true to form, makes you build it yourself out of a dictionary and a Future. Varnish calls it request collapsing; Nginx spells it proxy_cache_lock. One pattern, a different label in every ecosystem — and anything reinvented this many times isn't a fad, it's fundamental. The renaming is just each community paying the same tuition.

The core intuition. Here's what sits under all of it, stated plainly: a cache miss at peak traffic is not one expensive operation. It's one expensive operation times every request that arrives in the window between "cache expired" and "first fetch returned." Put real numbers on it and the whole chapter fits on a single line:

5,000 req/s × 80 ms (0.08 s) fetch window = 400 identical queries, one row → coalesced into 1.

Those 400 aren't 400 different problems. They're the same problem, 400 times over, each request racing the others to fetch a row that was already being fetched. Coalescing collapses those 400 into one.

The reason this catches experienced people out — and it does, regularly; I've watched it happen to engineers who'd been caching things correctly for a decade — is that nothing about the naive version looks wrong. It isn't a bug. There's no exception, no corrupt data, no line you can circle in review. The code that fetches on a miss and fills the cache is the code in every tutorial, and it's correct. It just carries a property nobody mentions: it's perfectly safe at 50 requests a second and a self-inflicted denial-of-service at 5,000, and the only thing standing between those two worlds is a number that lives in production, not in the repo. So you ship the correct code, it behaves for a year, and then one launch makes a single key hot and the correct code calmly tries to take down your database. You can't catch it in code review because there's nothing to catch. You catch it in the postmortem.

Here's the ground we'll cover:

• Why hot keys turn caching from a defense into a liability — and why the danger scales with success, not failure
• The naive fixes — more cache, locks, polling — that look right and fail under load in their own instructive ways
• The shared-future pattern that actually holds, and how it collapses a whole miss window into a single fetch
• The new failure modes that come free in the box — the completion race, the timeout cascade, the coalescing chain
• How Go's singleflight, Guava's LoadingCache, and Varnish and Nginx ship the same idea under different names
• What a principal engineer weighs before reaching for coalescing — including when the cure costs more than the stampede it prevents

The problem coalescing exists to kill has a name — the cache stampede — and the herd in it isn't your users, it's your misses.

Hot keys are a specific configuration of load that turns ordinary, well-behaved caching into a liability.

Take Netflix's recommendation engine. Generating a set of recommendations is genuinely expensive — seconds of CPU, a fan of downstream service calls, real memory pressure — so the obvious and correct move is to precompute it and cache the result. This works beautifully right up until a popular profile's cache entry expires at a moment when 100,000 of that profile's sessions are live. The recommendation service does not receive one request to recompute. It receives the same recomputation 100,000 times in parallel, each instance dutifully racing the other 99,999 to write the identical answer to the identical key. The cache — the thing you added to protect the expensive computation — just scheduled it a hundred thousand times at once.

Most services meet a less cinematic version of the same thing. A product page for a just-launched item takes 10,000 requests a second. The product data is cached — that's table stakes, you did it on day one. But when the entry expires, every request that lands in the gap between expiry and refill goes straight to the database, and at 10,000 RPS with an 80-millisecond round-trip, that gap is wide enough for 800 simultaneous queries for a single row to march through it shoulder to shoulder.

This is the cache stampede: a thundering herd with a specific trigger — not a surge of new users, but a single cache miss on something popular. The counterintuitive part, the part that bites people with years of caching scars, is that it scales with success. A cold key that expires gets hit by the one unlucky request that happened to want it; that request quietly fetches and refills, and nobody notices. A hot key that expires gets hit by every session that wanted it, all at once. Which means the keys most worth caching are precisely the keys whose expiry hurts most. The better your cache does its job, the bigger the crater when an entry blinks out — because caching never removed the load, it postponed it and let it pool, and a stampede is just what pooled load looks like the instant the dam goes.

The reflex fix is to reach for the TTL knob. Shorter TTL means fresher data and fewer stale reads, the reasoning goes — which is true, and beside the point. Shorter TTL means more frequent expiries, which means more stampedes per hour, not fewer. Longer TTL cuts the frequency but makes each stampede worse, because the longer a popular key lives, the more concurrent demand has stacked up behind it by the time it finally lets go. You're not turning the problem off with that dial; you're just choosing between trampled-often-and-gently and rarely-and-hard. The actual fix isn't on the dial at all: it's making a miss on any given key produce exactly one fetch, no matter how many requests arrive while that fetch is in flight.

Three approaches get built, in roughly this order, before a team arrives at shared futures. Each is locally reasonable. Each fails in a way that teaches the next one.

No coalescing, just more cache. Every request checks the cache, misses, queries the database, fills the cache. At low concurrency the first fill quietly overwrites the second and nobody's the wiser. At high concurrency you get your 800 simultaneous queries to one row, and the instinct is to buy your way out — more cache nodes, more read replicas. It doesn't touch the problem. More cache servers just spread the misses across more boxes; the miss window is exactly as wide as it was. More read replicas hand the database more capacity to absorb the stampede, which means you've raised the ceiling on how bad it can get before something snaps. That's not a fix. That's a bigger boat, so you can take on more water before you sink.

Lock-based coalescing. Grab a per-key lock, check the cache, fetch if it's empty, release. (Full code: Appendix A.1.) This is the version everyone writes second, and it shows up with three problems on a fixed schedule. First, every waiting request holds a thread while it blocks on the lock — invisible at low concurrency, and at high concurrency the exact mechanism by which thread-pool exhaustion becomes your next incident, the one that somehow ends up worse than the one you were fixing. Second, when the holder finishes and releases, the waiters wake in a pack, all check the cache, all find it freshly warm — redundant reads that add no real load but make the whole thing look cheaper in a benchmark than it is in a storm. Third, if the lock holder dies mid-fetch, you now need lock expiry and reclaim logic, or every request for that key wedges until the TTL clears. You set out to remove one coordination problem and came home with two.

Poll-based waiting. Drop the lock. On a miss, write a sentinel — "fetch in progress" — and start fetching. Everyone else who sees the sentinel spin-waits, re-reading the key until real data shows up. (Full code: Appendix A.2.) Simple to write. Grim at scale. Ten thousand waiters re-polling Redis every 10 milliseconds is a million Redis reads a second — you've picked the stampede up off the database and set it down on the coordination layer, larger than you found it. And if the fetcher crashes before it writes the result, the sentinel just sits there until its TTL expires while every waiter loops in the dark — no error, no signal, just a key that says "almost ready" and means nothing. Depending on how you've set the sentinel TTL against the request timeout, you've converted a cache miss into a synchronized multi-second hang for every concurrent user in the window. Poll-based coalescing is how a lot of people first learn what a future is: by building something that behaves like one in all the ways that hurt and none of the ways that help.

The thread tying all three together is the same: the failure only shows up under the concurrency that made coalescing necessary in the first place. At 50 concurrent requests for a hot key, lock-based coalescing is fine — you'll demo it, it'll pass review, you'll move on with a clear conscience. At 50,000 it falls over, and by then it's load-bearing in production and the teacher is the incident. This is the recurring cruelty of scaling problems: the one environment where you'd want to reproduce the bug — the test environment — is by definition the one place it can't form.

Implement shared futures correctly and the naive failure modes go away. In their place you inherit a new set — subtler, and specific to the fix. This is the tax. It's worth paying, but you should read the line items before you sign.

The race at completion. A request for key K arrives at the precise instant the in-flight future for K resolves and its result is mid-write to the cache. Depending on timing, the newcomer sees "no future in flight, nothing in cache yet" and starts a fresh fetch — duplicating work that finished microseconds ago. This is a deduplication miss, not a correctness bug: the data's fine, you just paid for the same fetch twice. The window is tiny and the cost is one extra query, so name it precisely and then mostly leave it alone — shared futures dedupe during the in-flight window, not across the seam between "future resolved" and "cache written." You can close that seam by extending the in-flight window to cover the cache write. Almost nobody does, and almost nobody should; it's a lot of machinery to save the occasional duplicate query.

The timeout cascade. A request coalesces onto an in-flight future, waits, times out, and hands an error back to its caller. The caller retries. Here's the fork that decides whether your coalescing survives contact with reality: does that retry re-attach to the still-running future, or kick off a brand-new fetch? If it starts a new one, you've broken coalescing at the exact moment you needed it — under the load that produced the timeouts to begin with. (Full code: Appendix A.3 tracks future state so retries re-attach instead of spawning.) The behavior you want is for a retry to find the existing future, if it's still alive, and wait on it. The behavior most simple implementations ship is to yank the future out of the map the moment any caller times out — so the next retry finds nothing, starts fresh, and your one fetch quietly becomes one-fetch-per-impatient-caller. Build the re-attach on purpose, or meet this one during an incident, which is where it prefers to introduce itself.

Cascading coalescing chains. This one is structurally unsettling. Service A coalesces its requests for X onto Service B. B coalesces its calls for X onto Service C. C's fetch fails. B's in-flight futures all reject at once; A's in-flight futures, waiting on B, all reject at once a beat later. Every request anywhere in the chain fails in the same moment — not from load, but because coalescing broadcast the failure with exactly the same brutal efficiency it would have broadcast a success. The blast radius of one failure in C is every concurrent request stacked across the combined coalescing windows of A, B, and C, delivered as a single synchronized wall of errors. The property that makes coalescing beautiful on the happy path — one result, fanned out to everyone waiting — is the same property that makes it vicious on the sad one. The circuit breaker from Chapter 4 is what keeps that synchronized wall of errors from instantly becoming a synchronized wall of retries and closing the loop.

The mechanism is a dictionary mapping each in-flight key to a future that stands in for the pending result. A request arrives and checks the dictionary. If a future is already there for this key, it waits on it. If not, it creates one, stores it, runs the fetch, and resolves the future with the answer. Everyone who waited unblocks the instant it resolves. The dictionary entry gets cleaned up on completion — success or failure, no exceptions. (Full code: Appendix A.4.)

Two details matter more than their length in any description would suggest.

First, error propagation. When the fetch fails, the future rejects, and every waiter gets that same exception at the same instant. It's all-or-nothing: one backend failure becomes N application errors, simultaneously. Whether that's fine or a catastrophe depends entirely on what your callers do next. If every one of those N errored callers independently retries, you've just scheduled a synchronized retry storm — the thundering herd from Chapter 5, summoned this time not by a load spike but by a single failed fetch. If your callers retry with backoff and jitter, you're fine. This is not an edge case to handle later; it is the primary failure behavior of the pattern, and it deserves to be designed on purpose rather than discovered at 3 a.m.

Second — and this is the one I've watched bite twice, in two different services, both written by people who knew exactly what they were doing — the cleanup has to run in a finally block, not at the tail of the success path. If the in-flight entry isn't removed when the fetch fails, the failed future stays in the map, and every subsequent request for that key finds it, attaches, and receives the original error immediately. Forever. You've built a cache, but for failures: one transient blip gets memoized and replayed to every caller who comes after, until somebody restarts the process. Both times, the happy path had been tested into the ground and the exception path had been assumed to tidy up after itself. It does not. The test that catches it takes thirty seconds to write: force the fetch to raise, immediately request the same key again, and assert that the second call attempts a fresh fetch instead of handing you the cached ghost of the first failure.

Coalescing and caching aren't competing options; they're complementary layers, each solving a different half of the problem. The cache — the multi-level one from Chapter 3 — handles the cost of fetching popular data over and over. Coalescing handles the coordination problem the cache creates at its own seams: what to do when a pile of requests all miss at the same moment. They stack.

The usual arrangement, top to bottom: cache, then coalescing, then database. A cache hit never touches the coalescing layer — fast path, zero coordination overhead. A miss falls into the coalescing layer, which guarantees at most one fetch per key reaches the database, writes the result back to cache on its way out, and resolves every waiting future. The requests right behind them hit the now-warm cache and never knew anything happened. (Full code: Appendix A.5.)

One practical thing every toy implementation skips: negative results. If a user account doesn't exist, "not found" is itself a fact worth caching — because if you don't, every request for that missing user triggers a fresh fetch, and coalescing gives you nothing, since each fetch returns quickly and the next request misses just as fast. Coalescing only helps while a fetch is in flight; an instant "nope" is never in flight long enough to collapse anything onto it. Deciding which negatives to cache, and for how long, is a separate decision from building coalescing, and you have to make it deliberately. A "not found" you forgot to cache is just a hot key in a disguise, waiting for a few thousand requests a second to come looking for a user who was never there.

Go's singleflight

Go ships golang.org/x/sync/singleflight, which is the pattern stripped to its studs. The whole API is essentially one function: Do takes a key and a function, dedupes concurrent calls by key, and hands the single shared result to everyone who asked for that key while the function was running. (Full code: Appendix A.6.)

What's telling is what the package refuses to do: no timeout handling, no retry logic, no cache integration, no observability hooks. It is the coordination primitive and nothing else, and it expects you to compose the rest. For a standard-library package that's an unusual amount of restraint, and almost certainly the right call — the composition decisions depend on context the package can't see. A DNS resolver and a recommendation engine want completely different things from timeout behavior and error handling, and any default the package picked would be wrong for one of them. Shipping no opinion turns out to be the only honest opinion on offer.

The package also ships DoChan, which returns a channel instead of blocking, so callers can build their own timeout with a select and singleflight never has to learn what a deadline is. That clean separation is exactly what the lock-based and poll-based approaches structurally can't offer. And the package runs throughout Go's own standard library — the DNS resolver included — which is a reasonably strong signal that Google's infrastructure people consider this primitive basic enough to live in the toolbox rather than in some internal framework.

The third return value of Do is shared bool — true when this call got a result computed by another goroutine rather than running the function itself. Almost nobody uses it, which is a quiet little tragedy, because logging or counting when shared is true hands you the deduplication ratio for free: the fraction of calls that did real work versus the fraction that rode along on someone else's. It's the single number that tells you whether your coalescing is doing anything at all, and it's sitting right there in the return signature, ignored, like a gauge no one ever looks at.

Java: Guava's LoadingCache

In Java, Guava's LoadingCache with a CacheLoader gives you the same behavior. When several threads ask for a key that isn't loaded, Guava lets exactly one of them run load() while the rest block, then distributes the result to all of them. (Full code: Appendix A.7.)

Guava's Striped class solves a neighboring but different problem, and the two get confused often enough to be worth pulling apart. Striped hands out a fixed pool of locks, assigned by key hash, so you get roughly key-level lock granularity without allocating a lock object per key. That's not coalescing — waiters still have to check the cache after they take the lock — but it makes lock-based coordination cheap when the key space is huge. The catch is hiding in the word "fixed": Striped with 64 stripes has 64 buckets no matter how many keys exist, so for a cache with three blistering-hot keys it may give you nothing at all, because all three can hash into the same bucket and serialize against each other for no reason any human chose. LoadingCache doesn't have that failure, because it coalesces on the actual key, not a hash of it. For hot-key problems, reach for LoadingCache. Keep Striped for contended writes, where per-key mutex overhead is the thing you're fighting and bucket-level granularity is good enough.

CDN-Level Coalescing: Varnish and Nginx

None of this has to live in your application code. Varnish's request coalescing and Nginx's proxy_cache_lock directive do the same job at the proxy: on a miss, hold the concurrent requests for the same URL, fire a single request upstream, then serve all of them from the one result.

That reframes the design question from "how do I add coalescing to my app" to "which layer is the stampede actually happening at?" If your hot keys are public endpoints behind a CDN or reverse proxy, and two requests are identical at the HTTP level — same URL, same cache-key headers — you can kill the stampede without touching application code at all, which is the kind of win that should make you briefly suspicious. The limit is right there in "identical at the HTTP level." The moment the response depends on something the proxy can't see in the request — authorization scope, per-tenant routing, anything application-level — the proxy can't safely collapse those requests, and coalescing has to move back into the app. Knowing which of those two worlds you're in is the whole decision, and it's worth making before you wire anything up rather than after.

All-or-nothing failure versus per-request retry. The shared-future model broadcasts the result — triumph or failure — to every waiter at once. One backend error becomes N application errors in the same millisecond. The alternative is partial success: let each waiter notice the failure and retry on its own. The trouble is that "retry on its own," multiplied by N waiters, is N simultaneous retries aimed at a service that just demonstrated it's struggling — the thundering herd again, in a different coat. There's no cozy middle. Choose the model deliberately: for most read paths over idempotent, cacheable data, all-or-nothing is correct, and you lean on Chapter 5's backoff-and-jitter in the callers to keep the retry wave from re-forming. If you can't trust the callers to retry sanely, put a single retry inside the coalescing layer — one attempt, on behalf of all the waiters together, never N attempts triggered by N callers each looking out for itself.

Memory cost of in-flight tracking. The in-flight dictionary is bounded by how many distinct keys are in flight at once, which for most services is a small, comfortable number. The cost climbs when your load is spread across many distinct keys — lots of different users each pulling their own data at moderate concurrency — because each entry holds a future plus references to every waiter on it, and that bookkeeping grows with waiters. It's rarely a true crisis, but it belongs on your radar, because it points at something deeper: coalescing pays best when a few keys soak up most of the traffic. If your actual problem is diverse-key load rather than hot-key load, coalescing gives you less per key and charges you more in overhead — you're running a stampede-control system for a stampede that never forms. Measure your key concentration before you reach for this tool, or you'll spend real effort optimizing a bottleneck you don't have.

Request identity and authorization scope. Coalescing is only safe when two requests for the same key genuinely deserve the same answer. The benign way this breaks is inconsistent key generation — a trailing slash here, a header cased differently there, query parameters serialized in a different order — which splits requests that should have coalesced into separate fetches. That's a missed optimization: mildly wasteful, completely harmless.

The malignant way it breaks is when authorization scope isn't in the key. If user A and user B both request resource X, and X returns different data depending on who's asking, and your coalescing key doesn't encode who's asking — then A and B coalesce, and one of them receives the other's data. That's not a performance bug. That's a textbook authorization bypass, served at high throughput, and it has the worst signature a bug can have: it passes every single-user test you will ever write, surfaces only under concurrent load with two differently-privileged users hitting the same key in the same window, and gets discovered either by a security audit or by the kind of incident that arrives with a disclosure email attached. Put authorization scope in the key everywhere it could conceivably matter, and do it before you turn coalescing on — not in the retro afterward, when the question is no longer hypothetical.

Coalescing on writes. The pattern is built for reads. Point it at non-idempotent operations and you've written a data-integrity bug wearing a performance optimization's clothes. Two concurrent "refresh user X's recommendations" calls collapsing into one computation: fine, both callers wanted the same fresh result. Two concurrent "increment user X's view count" calls collapsing into one write: congratulations, you've lost a view, and you won't know which one or when. The dividing line is idempotency, which is Chapter 2's entire subject. If you're not certain an operation is safe to dedupe, it isn't. The price of being too cautious is one redundant query. The price of being too clever is data loss you'll meet for the first time weeks later, in an audit, with no idea how long it's been quietly happening.

Scenario. Your product-catalog service is handling a heavily-hyped launch. At peak, the product page takes 8,000 requests a second. Product data is cached with a 5-minute TTL. The database query is 80 milliseconds round-trip. The service runs as 8 pods with no cross-pod coordination.

Work the numbers. In the 80-millisecond miss window, at 8,000 RPS, roughly 640 requests arrive. Spread across 8 pods, that's about 80 concurrent misses per pod — and with no coalescing, 80 queries per pod, 640 in total, every one of them for the same row.

Your task. Design two coalescing strategies: one that takes 640 down to 8, and one that takes 8 down to 1. For each, name the new failure mode you just signed up for.

Hint. Pod-local coalescing — Go's singleflight inside each pod — gets you from 640 to 8. Cross-pod coalescing — a shared in-flight registry in Redis, distributed lock, pub/sub to wake the waiters — gets you from 8 to 1, and hands you a new dependency and a new failure mode in the same motion: when Redis is unreachable, the coalescing layer fails, every request falls through to the database at once, and you've rebuilt the exact stampede you were preventing, now with extra steps. Is 8 acceptable? For a well-indexed row behind a connection pool, almost certainly — stop there. For a 3-second computation with expensive downstream calls, almost certainly not. The real question isn't whether you'd prefer 1 to 8; of course you would. It's whether the operational weight of cross-pod coordination is worth the gap between 8 and 1 — and you can't answer that until you know what one of those fetches actually costs.

← Chapter 5 (Retries). Coalescing is the in-application answer to the thundering herd, approached from the cache side. Chapter 5 meets the same herd as a retry phenomenon — many clients failing together and retrying in correlated waves. This chapter meets it as a cache-expiry phenomenon — many requests missing together and fetching at once. Different trigger, identical shape underneath: many callers, one piece of work, one shared result. Learn to see that shape and you'll start catching it in places that have nothing to do with caches.

← Chapter 4 (Circuit Breakers). When the in-flight future fails and every waiter gets the error in the same instant, the breaker is what stops those N simultaneous errors from becoming N simultaneous retries. Without it, coalescing's all-at-once error delivery is a perfect ignition source for a retry storm. The two compose cleanly: coalescing guarantees one fetch, the breaker guarantees that one failure doesn't turn into ten thousand retries aimed at a service that's already on the floor.

→ Chapter 7 (Feed Architectures). Feeds are where you meet coalescing's mirror image. Fanout — one write that has to land in thousands of followers' feeds — is the structural inverse of what this chapter did: coalescing collapses N identical requests into one piece of work; fanout explodes one event into N pieces of downstream work. The design questions reflect across the same axis. How do you bound the amplification? What happens when one of the N fails? How do you assemble a coherent answer when the pieces arrive at different times? Hold both patterns in your head at once and the whole many-to-one and one-to-many design space stops being two puzzles and becomes one.

The intuition to carry out of this chapter: a cache miss is not one missed lookup. Under concurrent load it's a missed lookup multiplied by every request that arrives during the miss window — and that window is set by your backend latency, not by anything you can reach on the cache-config dial. The miss is the ignition. The query time is the window. The concurrent requests are the fuel. Coalescing doesn't make the miss cheaper; it makes sure the window gets paid for exactly once, no matter how many requests pour into it. The database got asked one question. Everyone who needed the answer got it. The other 999 requests never had to be the database's problem at all.

The chapter keeps the code out of the narrative on purpose. Here it is, collected, runnable, and annotated. Each entry is the full version of something described in plain language in the body — read the prose for the idea, come here when you want to type it out. All of it is illustrative rather than production-hardened: the snippets omit the thread safety, metrics, connection management, and distributed-systems ceremony the real thing demands — and the first two are broken in deliberately instructive ways.

A.1 — Lock-Based Coalescing (the mutex approach)

1import threading
2 
3_locks = {}
4_locks_lock = threading.Lock()
5 
6def get_with_lock(key, fetch_fn, cache):
7    # Get or create a per-key lock
8    with _locks_lock:
9        if key not in _locks:
10            _locks[key] = threading.Lock()
11        key_lock = _locks[key]
12 
13    with key_lock:
14        result = cache.get(key)
15        if result is None:
16            result = fetch_fn(key)
17            cache.set(key, result)
18    return result

Every waiting thread blocks on key_lock while holding a system thread. Fine at low concurrency, catastrophic at high concurrency when threads are exhausted. If fetch_fn raises, the exception propagates to the thread holding the lock, but other waiters don't receive it — they acquire the lock next, find an empty cache, and attempt their own fetch. Under failure, this degrades to one-fetch-per-waiter rather than one-fetch-period.

A.2 — Poll-Based Waiting (the spinning sentinel)

1import time
2 
3SENTINEL = "__IN_PROGRESS__"
4 
5def get_with_poll(key, fetch_fn, cache):
6    existing = cache.get(key)
7 
8    if existing == SENTINEL:
9        # Someone else is fetching — spin until ready
10        while cache.get(key) == SENTINEL:
11            time.sleep(0.01)
12        return cache.get(key)
13 
14    if existing is not None:
15        return existing
16 
17    # First request — claim the key and fetch
18    cache.set(key, SENTINEL, ttl=5)
19    try:
20        result = fetch_fn(key)
21        cache.set(key, result)
22        return result
23    except Exception:
24        cache.delete(key)  # Remove sentinel on failure
25        raise

10,000 waiters polling every 10ms → 1,000,000 Redis queries per second. If the fetch raises and sentinel deletion fails for any reason, all waiters loop until they time out individually with no error surfaced. The cache.delete(key) in the except block is load-bearing and still leaves a window between cache.set(key, SENTINEL) and cache.delete(key) where a crash drops the sentinel permanently. The 5-second TTL is the fallback for that window, which means a 5-second silent hang for every concurrent waiter.

A.3 — Timeout-Aware Singleflight (retries re-attach to in-flight futures)

1import threading
2from concurrent.futures import Future
3 
4class SingleFlightWithTimeout:
5    def __init__(self):
6        self._in_flight = {}
7        self._lock = threading.Lock()
8 
9    def call(self, key, func, timeout=None):
10        with self._lock:
11            if key in self._in_flight:
12                future = self._in_flight[key]
13                is_initiator = False
14            else:
15                future = Future()
16                self._in_flight[key] = future
17                is_initiator = True
18 
19        if is_initiator:
20            threading.Thread(target=self._run, args=(key, func, future), daemon=True).start()
21 
22        try:
23            return future.result(timeout=timeout)
24        except TimeoutError:
25            # This caller timed out, but the future is still running.
26            # A retry will find it in _in_flight and attach rather than spawning a new fetch.
27            raise
28 
29    def _run(self, key, func, future):
30        try:
31            result = func()
32            future.set_result(result)
33        except Exception as e:
34            future.set_exception(e)
35        finally:
36            with self._lock:
37                self._in_flight.pop(key, None)

The key property: a caller that times out raises TimeoutError locally, but does not remove the future from _in_flight. The fetch continues. A retry from that caller finds the future still in-flight and attaches to it rather than starting a competing fetch. Simple implementations that remove the future from the map on any caller timeout cause retries to become new fetches, defeating coalescing under exactly the load conditions that produce timeouts.

A.4 — Core Shared Future Pattern (canonical form)

1import threading
2from concurrent.futures import Future
3 
4class SingleFlight:
5    def __init__(self):
6        self._in_flight = {}
7        self._lock = threading.Lock()
8 
9    def call(self, key, func):
10        with self._lock:
11            if key in self._in_flight:
12                return self._in_flight[key]
13            future = Future()
14            self._in_flight[key] = future
15 
16        # Run fetch outside the lock so waiters can check _in_flight concurrently
17        try:
18            result = func()
19            future.set_result(result)
20        except Exception as e:
21            future.set_exception(e)
22        finally:
23            with self._lock:
24                self._in_flight.pop(key, None)  # MUST run even on failure
25 
26        return future

The finally block is load-bearing. If func() raises and the key is not removed, the next request for this key will find the failed future and receive the error immediately — indefinitely. The coalescing layer becomes a cache for failures. Test this explicitly: force func to raise, make a second request for the same key, and verify the second request attempts a fresh fetch rather than receiving the cached error.

A.5 — Cache-With-Coalescing Integration

1class CacheWithCoalescing:
2    def __init__(self, cache, sf: SingleFlight):
3        self._cache = cache
4        self._sf = sf
5 
6    def get(self, key, fetch_fn, ttl=300):
7        # Fast path: cache hit, skip coalescing entirely
8        cached = self._cache.get(key)
9        if cached is not None:
10            return cached
11 
12        # Slow path: coalesce, fetch once, populate cache
13        future = self._sf.call(key, lambda: fetch_fn(key))
14        result = future.result()
15 
16        # All coalesced callers execute this set, but with the same value.
17        # If the cache write is expensive, track the initiator and write only once.
18        self._cache.set(key, result, ttl=ttl)
19        return result

Every caller in a coalesced group executes cache.set() with the same result — redundant writes, not incorrect ones. For most caches and value sizes, the overhead is negligible. If the value is large (multi-KB serialized objects) or the cache write latency is non-trivial, track which caller initiated the fetch and have only that caller write. The added complexity is rarely justified; measure first.

A.6 — Go singleflight (standard library)

1package main
2 
3import (
4    "context"
5    "golang.org/x/sync/singleflight"
6)
7 
8var sf singleflight.Group
9 
10func getProduct(ctx context.Context, productID string) (Product, error) {
11    result, err, shared := sf.Do(productID, func() (interface{}, error) {
12        return fetchFromDatabase(ctx, productID)
13    })
14    if err != nil {
15        return Product{}, err
16    }
17 
18    // shared=true means this result was computed by another goroutine.
19    // Increment a counter here to measure your deduplication ratio.
20    if shared {
21        metrics.Increment("singleflight.coalesced", "key", productID)
22    }
23 
24    return result.(Product), nil
25}

sf.Do blocks the caller until the function completes, then returns (value, err, shared). The shared bool is true for every caller that received a result computed by another goroutine — i.e., every call that coalesced rather than fetched. Tracking this gives you the deduplication ratio without any additional instrumentation on the fetch itself. Use sf.DoChan instead of sf.Do when you need to implement timeout logic via a select statement — it returns a channel rather than blocking, so callers can time out without singleflight needing to know about deadlines.

A.7 — Guava LoadingCache (Java)

1import com.google.common.cache.CacheBuilder;
2import com.google.common.cache.CacheLoader;
3import com.google.common.cache.LoadingCache;
4import java.util.concurrent.TimeUnit;
5import java.util.concurrent.ExecutionException;
6 
7LoadingCache<String, Product> cache = CacheBuilder.newBuilder()
8    .maximumSize(10_000)
9    .expireAfterWrite(5, TimeUnit.MINUTES)
10    .recordStats()  // enables hit rate, load count, load time metrics
11    .build(new CacheLoader<String, Product>() {
12        @Override
13        public Product load(String productId) throws Exception {
14            // Only one thread calls this per key while others block and wait.
15            return fetchFromDatabase(productId);
16        }
17    });
18 
19// Callers:
20try {
21    Product product = cache.get(productId);
22} catch (ExecutionException e) {
23    // If load() threw, all blocked callers receive ExecutionException wrapping the original.
24    throw new ServiceException("fetch failed", e.getCause());
25}

When multiple threads call cache.get(key) for an absent key, Guava invokes load() once and blocks all other threads until it completes, then distributes the result. recordStats() enables cache.stats(), which exposes hit rate, miss rate, load count, and average load time — the metrics you need to verify coalescing is functioning and to detect degradation. If load() throws, every blocked thread receives an ExecutionException; handle this path explicitly, since Guava will not retry automatically.

Next: Chapter 7 — Feed Architectures: when one write has to fan out to a thousand feeds, and how to keep the many-on-write from becoming the outage.