Retries

A retry is the most reasonable thing you can do when a request fails, and the most dangerous, and it's both for exactly the same reason: it doubles down on a request that just failed. Get the timing right and you paper over the small constant rudeness of the network — the dropped packet, the GC pause, the half-second a node spends rebooting — and nobody downstream is any the wiser. Get it wrong and you take a service that was having a bad minute and hold it under until it has a bad hour.

Here's the sentence that took me a few outages to actually believe: a request failing is not a fact about the request. It's a fact about this attempt, on this connection, at this instant. The server may have done the work and died before it could answer. It may never have heard you. The packet may have arrived and then sat in a full connection queue until the OS quietly dropped it on the floor. From where you're standing, every one of these looks identical — silence, then a timeout — and the only honest reply to silence is to ask again. Carefully.

"Carefully" is carrying the entire chapter on its back.

Without retries, every transient blip becomes a user-visible error, and in a system where one user request fans out to a dozen downstream calls, a hiccup in any one of them fails the whole thing. A payments company that treated every timeout as a permanent failure would decline a card every time a GC pause hit a downstream service. Nobody can run a business that way. Retries are not optional.

And yet: retries amplify load, and they do it at precisely the worst moment. A service fails because it's overloaded; you respond by sending it more of the same traffic that overloaded it, multiplied by every client that just timed out. It's the infrastructure version of telling a drowning person to swim harder. The service that was struggling under 1,000 requests is now under 2,000 — the original load plus all the retries — and you have not helped it recover. You've finished it off.

The pattern that threads this needle — exponential backoff with jitter, fenced by a retry budget, gated by a circuit breaker (Chapter 4), and made safe to repeat by idempotency (Chapter 2) — is most of what this chapter is about. None of those properties come free, and the road from "retry on timeout" to "retry without starting a stampede" runs directly through at least one production incident. Most teams pay that toll in person. This chapter is an attempt to let you skip the line.

Here's the ground we'll cover:

• Why each obviously-reasonable retry approach fails in its own instructive way
• Exponential backoff, and why the randomness isn't decorative
• Retry budgets — the mechanism that keeps retries from becoming the outage
• Deadline-aware retries, and why most teams skip them right up until the incident that makes the case
• How Google, Stripe, and Netflix arrived at the same discipline through three different doors
• The questions to ask your team before an incident asks them for you

The retry storm in three acts: amplification, synchronization, and no off switch.

"Retries cause load" is true and useless. Here's the specific shape of the disaster.

Service A calls Service B. B slows down — not crashes, slows, which is worse — because a slow query crept onto the hot path, or a replica fell behind, or someone shipped a build that's 20% heavier and nobody's noticed yet. A's calls to B start timing out. A retries. Immediately. Because that's what the code does, and the person who wrote the code was not thinking about today.

B was handling 1,000 requests a second before the slowdown. Now it's handling those 1,000 plus 1,000 retries from the clients that just timed out — 2,000 requests a second aimed at a service that was already buckling at 1,000. B slows further. More clients time out. More retries. Inside a minute you've gone from "B is a little slow" to "B is gone, and the retry storm is the outage now."

The detail that makes this miserable to debug is that cause and effect come apart in the dashboards. The slow query that kicked it off may have cleared in thirty seconds. The retry storm it lit runs for minutes, because the retries are now generating the load that generates the timeouts that generate the retries. The original problem is long dead. The incident is running on its own exhaust — a perpetual-motion machine whose only output is pages. I once spent the first forty minutes of an outage hunting a downstream that had been healthy for thirty-nine of them: the slow query that lit the fuse had cleared almost immediately, and what we were actually staring at was our own retries, looping, manufacturing the very timeouts we were chasing.

The first fix anyone reaches for is to wait between retries. Wait a second, try again. It feels responsible. It gives B a second to breathe.

It doesn't work, and the reason is the one nobody sees coming: all 1,000 clients that timed out are running the same code, so they all wait the same one second, and then they all retry at the same instant. You didn't spread the load. You gathered it up, held it for a second, and delivered it to a struggling service as a single synchronized punch. This is the thundering herd, and fixed delays summon it with eerie reliability.

Exponential backoff looks like the cure: wait 1 second, then 2, then 4, then 8. The retries fan out over time; the service gets more room with each round. Except the 1,000 clients all failed at the same moment, so their backoff schedules are identical — they retry together at t+1, together at t+3, together at t+7. You've traded one continuous storm for a series of synchronized spikes. Genuinely better. Still a stampede, just one that pauses politely between charges.

Jitter is the part that actually works, and it works by a mechanism that feels too cheap to be the answer: add randomness. Each client waits a random interval inside the backoff window rather than the whole window, so 1,000 clients that failed together scatter their retries across a span of time instead of stacking them on a single instant. The service sees a drizzle instead of a flash flood. We'll come back to why this is more than a hack — it's one of the prettier results in distributed systems — but the one-line version is that the randomness is load-bearing, not garnish.

Then the problem none of the above touches: how many times? A client willing to retry 100 times across a schedule that grows to a minute between attempts can hammer a recovering service for the better part of two hours — holding threads, re-submitting operations that may or may not be safe to repeat, feeding the next cascade the whole while. You need a ceiling, and it has to be tied to something real — the circuit state, the retry budget, the time the caller has left — not a number somebody typed because five felt round.

Every engineer who has implemented retries has written approximately this sequence, in this order, usually one production incident apart.

Attempt one: just call it again. Catch the timeout, call the service a second time, and hope. (Full code: Appendix A.1.) This works exactly when the problem was a single dropped packet that already healed itself — which is often enough to be seductive and rare enough to be a trap. The moment the downstream is genuinely overloaded, the retry lands before the server has finished choking on the original, and you've doubled your contribution to the pileup for the price of one caught exception.

Attempt two, after someone points at the dashboard: add a delay. Loop up to five times, sleep a second between tries. (Full code: Appendix A.2.) The engineer has learned "wait a bit," which is real progress. What they haven't learned yet is that a thousand callers running this exact loop — or one service fielding a thousand concurrent callers — produces a thousand retries one second after the timeout, to the millisecond. The delay didn't disperse the herd. It scheduled it.

Attempt three, after reading a blog post: make the delay exponential. Sleep 1, 2, 4, 8, 16 seconds. (Full code: Appendix A.3.) Now retries actually spread across time, and this is the first version that could plausibly give a sick service room to stand up. The flaw is correlation: clients that failed together back off together, on the same schedule, and arrive in waves spaced just far enough apart to show up on your latency graph as a tidy heartbeat. (When your incident's retry pattern is regular enough to set a watch by, that's the tell.)

Attempt four, after attempt three causes the incident: add jitter. Randomize the wait inside the backoff window. (Full code: Appendix A.4.) This one works. The randomness breaks the correlation, the clients that timed out together now retry at different moments, and the load smooths into something a recovering service can survive. Its only remaining sin is the hardcoded range(5) — an attempt count connected to nothing. Maybe the service needs six attempts and you quit one short. Maybe it needs fifty and you've signed up to spend the next hour as a card-carrying participant in the storm. The count has to connect to something the program can observe — the budget, the breaker, the deadline — and that's the rest of the chapter.

Here's the part worth sitting with, because it explains why experienced engineers keep rediscovering this from the inside rather than reading about it once and being done. Every step in that sequence is locally reasonable. Catching a timeout and trying again is reasonable. Adding a delay is reasonable. Each client, considered alone, is behaving impeccably. The catastrophe is entirely emergent — it lives in the correlation between a thousand well-mannered clients, none of which can see the other 999. That's the blind spot: retries are written as a per-call, per-client decision, and the failure is a system-level property that no single call site can see. You can read every retry block in the codebase, find nothing wrong with any of them, and still have a storm waiting for the next slow Tuesday. The bug isn't in the code. It's in the arithmetic of everyone running the same correct code at the same moment.

The engineers who skip straight to attempt four exist. They've been on the receiving end of an attempt-three incident, which is an extremely effective teacher. Everyone else walks the path.

Retry amplification. The arithmetic is humbling precisely because each step is so reasonable. A hundred clients, each willing to retry ten times, can turn 100 requests into 1,000 — ten times the load, aimed at a service whose entire problem was that it already had too much. A healthy-looking 20% of headroom does not survive contact with a 10× multiplier. The math is indifferent to the fact that every individual client was being sensible; sensible, multiplied by a hundred and correlated in time, is a DDoS you built yourself and now pay to host.

The deadline mismatch. Full exponential backoff over five attempts is 1 + 2 + 4 + 8 + 16 = 31 seconds of waiting, which is a peculiar amount of time to spend when the user's browser gave up at 30. You've built retry logic that structurally cannot succeed before the only person who cared has left the building. The retries fire anyway, because the service has no idea the caller is gone — burning threads and downstream capacity to compute an answer with nowhere to go. It's the office worker still polishing slides for a meeting that ended an hour ago.

Retrying non-idempotent operations. This is the one that shows up in the retrospective under a heading like "we didn't realize this could be retried." A POST /payments returns a 504; the original charge actually succeeded but the response evaporated on the way back; the retry charges the card again. Now there's a double charge, an over-decremented inventory count, and a refund nobody's proud of. Chapter 2 is this failure mode's home turf. Retries are only safe on operations that are safe to repeat, and "safe to repeat" is not a property you get for free — it's idempotency keys, dedup logic, and a handler that treats at-least-once delivery as exactly-once on purpose. Bolt retries onto a system that never thought about idempotency and you haven't added resilience; you've added a data-corruption bug on a timer.

The cascading retry storm. This is amplification with a blast radius. A retries B; while B is straining to recover, B's own calls to C and D are timing out, so B is retrying those; meanwhile C and D have other clients who are also retrying. Every hop multiplies the load on the hop beneath it, and a hiccup in one service propagates outward through the whole dependency graph as an expanding wavefront of retries. The thing that takes down the company was not the service with the original hiccup. It was every service standing behind it, each one faithfully amplifying the panic of the one in front.

The mechanism is the one AWS wrote up in 2015, in a post that remains the clearest thing anyone has published on the subject: wrap the call in a loop, and on each failure sleep for a random interval drawn from zero up to an exponentially growing ceiling, capped so it can't run away. (Full code: Appendix A.5.)

Two parameters do all the work, and both are easy to set without thinking. base is the floor on the first backoff — usually a second, lower for fast internal calls where a full second is an eternity. cap is the ceiling, usually around 60 seconds, and it exists to stop the exponential from doing what exponentials do: without it, ten doublings carry the backoff window past seventeen minutes, which is a retry schedule nobody designed, nobody documented, and nobody is watching the graph for.

The word doing the real work is full. "Full jitter" means the wait is uniformly random across the whole window — random(0, window) — not the exponential value with a little noise sprinkled on top. People conflate the two and assume the difference is cosmetic. It isn't: full jitter spreads retries far more evenly at the tail, which is exactly where a recovering service is most fragile. The gap between "exponential with some jitter" and "full jitter" is a few characters of code and a meaningfully gentler load curve on the service you are trying not to kill.

Why does randomness work at all? Because a thousand clients each making an independent random draw from the same interval will, with no coordination whatsoever, spread themselves smoothly across it. No central scheduler, no service-mesh magic, no clients talking to each other. Each one acts in pure isolated self-interest, and the aggregate is cooperative — a smooth, survivable load. It's one of the few places in distributed systems where selfishness composes into something polite, and it costs exactly one call to a random number generator.

The trouble with max_attempts=5 is that it's a number with no referent. Nobody computed it; somebody picked it. Retry budgets throw out the per-client count and ask a system-level question instead: what fraction of my total traffic am I willing to let be retries?

Google's SRE book documents the pattern, and the recommended ceiling is 10% — allow retries only while retries are under a tenth of total requests — with the pointed footnote that even 10% is high for a healthy system. The implementation keeps a running count of successes and retries and refuses to retry once the ratio crosses the line. (Full code: Appendix A.6.) A production one needs sliding windows, thread safety, and per-dependency tracking, but the idea fits in a sentence: retries are a budget, not a right, and when the budget's spent you fail fast instead of piling on.

What this buys you is a hard ceiling on amplification, set at the service level where it belongs rather than the client level where it can't be controlled. A service taking 100 requests a second with a 10% budget sees on the order of 110 — not 400, not 1,000 — no matter how many individual clients are panicking, no matter how aggressive each one's local retry loop is. And the real ceiling is tighter than that clean number suggests: as the dependency actually sickens and its successes dry up, the retry ratio crosses the line faster, so the budget clamps down harder at exactly the moment you need it to. The budget doesn't care how many clients there are. It caps the sum.

The math is the smaller half of the value. The larger half is that the budget forces a conversation a team would otherwise never have: what fraction of our requests are we willing to spend on retries? A team that has answered that has started treating retry behavior as a property of the system. A team that hasn't will be introduced to the question in production, by the system, at a time of the system's choosing — and the system has a cruel sense of timing.

The enabling mechanism is context propagation: the original caller's deadline rides along through the entire call chain, so every service in the path knows not just that it should hurry but exactly how much time is left on the clock. gRPC does this natively. HTTP services do it by convention — a header (X-Request-Deadline, X-Timeout-MS, pick one and be consistent) that every hop is responsible for reading, honoring, and passing forward, and that exactly one team always forgets to forward.

With a deadline in hand, the retry decision grows a precondition: before sleeping, check whether there's enough time left to bother. (Full code: Appendix A.7.) If you've got two seconds left and the next backoff is four, you will certainly miss the deadline — so fail now, immediately, rather than sleeping four seconds to produce an answer the caller abandoned two seconds ago.

That check is a small piece of code and a genuine shift in posture: don't retry for your own benefit when there's no longer anyone on the other end to benefit. It kills a whole category of waste where services deep in the graph retry heroically — burning CPU, holding connections — for a request the API gateway gave up on and returned an error for ages ago. The work has nowhere to land. It's a kitchen still cooking an order for a table that paid and left. Deadline propagation is what lets the kitchen see the empty table; deadline-aware retries are what make it put down the pan.

The tradeoff that actually bites is retry count against downstream amplification, and the numbers stay counterintuitive until you've run them once with your own outage attached.

Take a downstream at 70% utilization — 30% headroom, which on paper reads as healthy. A client that retries once on failure sends, at the bad moment, up to 2× its normal volume. 2× into 70% is 140%, which is over the line, which means the service is now overloaded by retries even though nothing was wrong with it before the retries started. Read that twice, because it's the whole trap: the retries are the cause, not the response. The service didn't fail and then get retried. It got retried and then failed.

Google's published guidance is blunt and worth stealing: three retries for most off-critical-path services, one retry for anything on the critical path. The reasoning is latency, not load — on a critical path, the time spent grinding through backoff cycles can blow past the user's patience, so you're better off failing fast with a clean error than spending ten seconds retrying your way to the same error with worse manners.

The backoff-strategy question sounds like it should be a long debate and usually isn't. Exponential with full jitter is the right default for anything under load, because it spreads better than anything else this simple. Linear backoff (1s, 2s, 3s, 4s) earns its place in the narrow case where you care about fairness over spreading — rate-limited clients with a known reset window, say. Pure exponential without jitter is the one to be suspicious of: it looks like the sophisticated choice and quietly keeps the clients correlated, so it helps less than its reputation promises. The gap between "exponential" and "exponential with jitter" is trivial to write and enormous in production, which is a sentence you could carve over the door of this whole chapter.

The thing nearly everyone oversimplifies: not every error is worth retrying, and the status code is a hint, not a contract. Retrying a 400 is almost always wrong — the request was malformed and will be exactly as malformed the second time. A 404 is a judgment call that turns on your consistency model: reasonable in an eventually-consistent system where the record might not have propagated yet, pointless in a strongly-consistent one where "not found" means not found. A 503 is usually worth retrying; it's practically an engraved invitation. A 500 depends entirely on whether it's a transient handler failure or a deterministic bug that will greet every identical request with identical enthusiasm. The policy has to be fitted to the actual error space of the actual service. Retrying everything uniformly is how you turn one bad request into the same bad request, sent five times.

Google SRE: Retry Budgets at Scale

Google's retry doctrine, spread across the SRE book and a decade of conference talks, hangs on one move: cap amplification at the service, not the client. Let every client retry N times on its own judgment and your worst-case amplification is N×, which for a service already underwater is frequently the difference between recovering and not. The budget pulls that ceiling down to a number the service itself controls and the clients cannot override by panicking harder.

The documented figure is 10% — retries should be no more than a tenth of traffic to a service — and in practice a healthy service runs well below it. A sustained retry rate above 1–2% is already worth a look: it means either a genuinely flaky dependency or retry logic that's too eager, and the budget alone won't tell you which. The retry success rate will. If your retries are mostly succeeding, the dependency is flaky and the retries are doing their job. If retries fail at the same rate as first attempts, you're not retrying — you're generating load with extra steps, and you should be failing fast instead.

The part that's organizational rather than technical, and therefore the part that doesn't actually get done: budgets need client teams and server teams to talk. The client owns the retry behavior; the server owns the capacity and has to expose the retry rate as a metric. When those two operate in silos — client team tuning retries it can't measure the impact of, server team absorbing load it doesn't know are retries — they meet for the first time during the incident, which is the most expensive venue ever devised for an introduction.

Stripe: Idempotency-First, Then Retry

Stripe's approach is worth studying because it starts in the right place, which is one place earlier than most teams start: idempotency is a prerequisite, not a follow-up ticket. Every charge request carries a client-generated idempotency key, and the server deduplicates on it for a configured window, so retrying a payment is provably safe — a duplicate gets recognized and the original result replayed, rather than charging the customer a second time for the privilege of a flaky network.

On that foundation they layer deadline propagation: a payment call with a 10-second timeout won't be retried at the 8-second mark if the next backoff is 4 seconds, because the service can see there's no path to finishing in time and would rather return a clean failure than a baffling one. The customer gets "that didn't go through, try again" instead of a spinner that resolves, twelve seconds later, into an error for a request they'd already given up on.

The transferable lesson is the ordering: build idempotency first, then retries — never the reverse. Bolt idempotency onto a system that already retries and you're auditing every handler in the codebase for hidden side effects, one grim spreadsheet row at a time. Stripe inverts it: idempotency is the contract the server publishes, and retries are just clients exercising their right to resubmit within it. Get the contract right and retries stop being dangerous and start being boring, which is the highest compliment you can pay a distributed-systems primitive.

Netflix: Adaptive Backoff

Netflix's retry behavior is less a fixed schedule and more a feedback loop. Their Hystrix library — since largely handed off to Resilience4j — tracked per-dependency health and latency in real time and used those signals to decide how hard to keep leaning on a struggling dependency: as a service's error rate and latency climbed, the breaker tightened and clients eased off; as it recovered, they leaned back in. A dependency answering comfortably got the benefit of the doubt; one drifting toward trouble got progressively more room, without a human in the loop.

The intuition is that a fixed backoff treats a healthy service and a dying one identically, which is faintly absurd when you say it out loud — distinguishing the two is the entire job. Adaptive backoff reads degradation as a request for more room and grants it without a human in the loop, then tightens back up as latency falls toward baseline, so clients return to normal behavior on their own. The retry policy tracks the actual state of the dependency instead of the state someone guessed at, months ago, while writing the config on a day when everything was fine.

The cost is complexity, and it's not a rounding error: latency telemetry per dependency, per-service tuning, and enough monitoring to notice when the adaptive machinery itself misbehaves. Netflix can sign that check because their dependency graph is enormous and manual tuning at that scale would cost more than the machinery does. At most companies' scale, fixed exponential backoff with jitter is the right answer, and adaptive backoff is a beautifully engineered solution to a problem you're allowed to not have yet. Knowing which of those you are is the actual skill.

Scenario. Service A sends 100 RPS to Service B. B can sustainably handle 110 — call it 10% headroom. B has a brief degradation. A's current retry policy: up to 3 retries on any 5xx, fixed 1-second delay between attempts.

Work the worst case. If every request fails and triggers all 3 retries, that's 100 original + 300 retries = 400 RPS into a service that tops out at 110. The storm pushes B to nearly 4× its capacity, which means B does not recover — it cascades, and now the retry policy is the outage.

Your task. Redesign A's retry policy so that A never sends more than 110 RPS to B, under any failure scenario you can construct.

Hint: a 10% retry budget holds retries to roughly a tenth of traffic — on the order of 110 RPS total — and clamps harder as B's successes dry up. Layer full jitter on top and even those retries spread across time instead of landing together, so the instantaneous peak sits below the average. Then add the circuit breaker from Chapter 4: once B is clearly failing rather than flickering, stop retrying altogether and fail fast, because past that point every retry is just load aimed at a service that has already told you, plainly, that it has nothing left to give.

← Chapter 2 (Idempotency). Retries are safe only when the operation is safe to repeat. Before you add a retry to any call, confirm the handler on the other end was built for at-least-once delivery. If it wasn't, the retry you're adding isn't resilience — it's a data-corruption bug on a delay timer, and it goes off the first time a response gets lost in flight.

← Chapter 4 (Circuit Breakers). The breaker is what stops the retries when a service is past saving. Retries without a breaker are how you get the infinite storm; the breaker is the off switch. They divide the labor cleanly: backoff governs how aggressively you retry while there's still hope, and the breaker decides when hope has resolved into "just stop."

→ Chapter 12 (Dead Letter Queues). When the retries are exhausted — budget spent, circuit open, operation still not done — the work has to go somewhere. Dropping it on the floor is rarely the right call. A dead letter queue is the somewhere: park the failed operation, let the dependency recover on its own schedule, and process it later instead of blocking a caller on it now.

The thing to carry out of this chapter is that a retry is never a private decision. It feels like one — a few lines at a single call site, handling a single failure, on a single request — and that's exactly the illusion that keeps retry storms perennial. The blast radius of those few lines is every service downstream of you, multiplied by every other client running their own reasonable version of the same few lines, correlated by the shared instant you all failed together. Backoff, jitter, budgets, and deadlines aren't four tricks; they're four ways of forcing a local decision to account for a global consequence. Get them right and a thousand clients failing at once produce a survivable drizzle. Get them wrong and they produce a service that was fine until its friends tried to help.

The chapter keeps the code out of the narrative on purpose. Here it is, collected, runnable, and annotated. Each entry is the full version of something described in plain language in the body — read the prose for the idea, come here when you want to type it out. All of it is illustrative rather than production-hardened: the snippets skip the timeouts, error taxonomy, thread safety, and metrics you'll need in the real thing, and the first four are deliberately broken in instructive ways.

A.1 — Immediate Retry (the optimist's reflex)

1try:
2    return call_service()
3except Timeout:
4    return call_service()  # surely it'll work this time

Works only when the failure was a single self-healing packet drop; the instant the downstream is actually overloaded, the retry arrives before the server finished the original and you've doubled your load for the price of one caught exception.

A.2 — Fixed Delay (the herd, rescheduled)

1for attempt in range(5):
2    try:
3        return call_service()
4    except Timeout:
5        time.sleep(1)

The pause feels responsible, but a thousand callers running this loop all wake one second after the timeout and retry in unison. The delay preserves the thundering herd intact — it just slides it one second to the right.

A.3 — Exponential Backoff, No Jitter (correlated waves)

1for attempt in range(5):
2    try:
3        return call_service()
4    except Timeout:
5        time.sleep(2 ** attempt)  # 1s, 2s, 4s, 8s, 16s

Finally spreads retries across time, but clients that failed together share one schedule and arrive in synchronized spikes at t+1, t+3, t+7. The giveaway is a retry pattern regular enough to set a clock by.

A.4 — Exponential Backoff with Jitter (the first one that works)

1for attempt in range(5):
2    try:
3        return call_service()
4    except Timeout:
5        backoff = 2 ** attempt
6        jitter = random.uniform(0, backoff)
7        time.sleep(backoff + jitter)

The random offset breaks the correlation, so clients that timed out together retry at different moments and the load smooths out. This is the version to actually ship — its only remaining sin is the hardcoded range(5), an attempt count tied to nothing real.

A.5 — Full Jitter (the canonical form)

1def retry_with_backoff(func, max_attempts=5, base=1, cap=60):
2    for attempt in range(max_attempts):
3        try:
4            return func()
5        except TransientError:
6            if attempt == max_attempts - 1:
7                raise
8            # Full jitter: uniform random within the exponential window
9            sleep_time = random.uniform(0, min(cap, base * (2 ** attempt)))
10            time.sleep(sleep_time)

The AWS-2015 formulation: wait a uniform-random interval across the whole exponential window (random(0, window)), not the window plus noise. base floors the first backoff; cap stops ten doublings from quietly scheduling a seventeen-minute wait.

A.6 — Retry Budget (amplification, capped at the source)

1class RetryBudget:
2    def __init__(self, budget_ratio=0.1, window_seconds=60):
3        self.budget_ratio = budget_ratio
4        self.successes = 0
5        self.retries = 0
6 
7    def can_retry(self):
8        total = self.successes + self.retries
9        if total == 0:
10            return True
11        retry_ratio = self.retries / total
12        return retry_ratio < self.budget_ratio
13 
14    def record_success(self):
15        self.successes += 1
16 
17    def record_retry(self):
18        self.retries += 1

Reframes retries from a per-client count into a system-level rate: spend retries only while they're under some fraction of total traffic (Google uses 10%), and fail fast once the budget's gone. This toy version skips the sliding window, thread safety, and per-dependency tracking the real thing needs — but it bounds amplification to a number you chose rather than a number your client count chose for you.

A.7 — Deadline-Aware Retry (check the clock before you wait)

1def retry_with_deadline(func, deadline, max_attempts=5):
2    for attempt in range(max_attempts):
3        remaining = deadline - time.time()
4        next_backoff = min(60, 2 ** attempt)
5 
6        if remaining < next_backoff:
7            raise DeadlineExceeded("not enough time left to retry")
8 
9        try:
10            return func()
11        except TransientError:
12            time.sleep(random.uniform(0, next_backoff))

Before sleeping, check whether enough of the caller's deadline remains to make the attempt worth it; if not, fail immediately instead of computing an answer no one is waiting for. Requires the deadline to be propagated down the call chain — gRPC does it natively, HTTP does it by a header someone always forgets to forward.

Next: Chapter 6 — Request Coalescing: turning a thousand identical in-flight requests into one piece of work, shared.