Rate Limiting and Traffic Shaping

Every system has a breaking point, and it will not send a calendar invite before it arrives. One minute the dashboards are green and somebody is talking about a hackathon. The next, latency is climbing in a straight line, responses are timing out, and the on-call engineer is scrolling a runbook whose most relevant entry is titled "Misc." The system didn't warn anyone. Systems never do. They just start saying no, badly, to everyone at once.

When requests arrive faster than your infrastructure can serve them, you only ever have two options. You can let everything degrade together, or you can decide who gets served and who gets turned away. That's the whole job. Rate limiting is just making that decision on purpose, with a policy you wrote while calm, instead of letting an overloaded thread pool make it for you at 2am.

Here's the part that takes people a few outages to internalize: rate limiting is not mainly about protecting your servers. Your servers are perfectly capable of falling over without any assistance from your traffic. What you're actually protecting is everything downstream of the moment of overload — your users' experience, the SLAs you signed, and your ability to say with a straight face what your system will do when load doubles. An API with no limits doesn't have a performance problem under load. It has a "we are about to find out how our dependencies fail in production, all of them, simultaneously" problem. A rate-limited API has defined behavior. That's the difference between an incident and a graph.

And then there's the part nobody puts in the design doc: overloaded systems don't fail evenly. They fail unfairly. With no limits in place, capacity goes to whoever is most aggressive — which is almost never whoever is most valuable. The enterprise customer with a retry loop stuck in a while True will happily take 10,000 RPS and starve out a thousand well-behaved clients making ten requests a minute. The viral post pointing at your endpoint will bury the paying customers under people who will never sign up. Rate limiting is how you decide that the loudest client doesn't automatically win. Which means it's a product decision wearing an engineering costume — somebody is choosing who matters when you can't serve everyone, and "whoever retries hardest" is a choice too, just a bad one.

Here's the ground we'll walk together:

• Why the obvious solution (count requests, reset every minute) breaks in production in two distinct and humbling ways
• The four algorithms that see real production use, what each one optimizes for, and when each is the right choice
• How Netflix, Stripe, Instagram, and Uber made different design choices—and why each was correct given their specific constraints
• The one failure mode that surprises engineers every single time (hint: the problem is that your clients are too well-behaved)
• The organizational questions that need answers before you write a single line of rate limiting code

Let's be precise about what we're solving, because "requests are coming in too fast" makes it sound like a traffic ticket when it's closer to a flood.

Every service has a maximum sustainable throughput — the request rate past which latency degrades unacceptably, queues grow without bound, or the system starts shedding work whether you told it to or not. Call it T. Under normal load you sit comfortably below T and everyone is happy.

The trap is the word "normal."

Normal is a story you tell yourself between incidents. Real traffic is spiky, occasionally adversarial, and has no respect for your capacity planning spreadsheet.

Three distinct things drive the spikes:

Malfunctioning clients. A single integration bug — a retry loop with no backoff, a cron job set to every second instead of every hour, a webhook that fires on every state change including the ones it caused — can produce thousands of requests per second from one source. You've seen this. Everyone who has run a service has seen this. The honest question was never whether one of your clients would point a firehose at you by accident. It's whether you'd already built the umbrella when they did.

Viral events. A launch, a well-timed post, a press mention on a slow news day. Traffic goes up 10x to 50x in seconds. Autoscaling cannot react that fast — and even on the day it can, your database still can't, because databases don't scale sideways on a moment's notice no matter how much the architecture diagram implies they do.

Actual attacks. DDoS, scraping at industrial scale, credential stuffing. Some fraction of your traffic does not wish you well, and rate limiting is part of how you stay standing.

When one of these hits, the reflex is to add servers. This works right up until it doesn't, and the place it stops working is the database. Your application tier is usually happy to scale horizontally. Your database is not, or at least not on incident timescales. Once you're database-bound, every additional app server just buys you one more pool of threads sitting in a queue, waiting politely on the same overloaded box. More waiters do not make the kitchen faster. The only move left is to send fewer requests to the kitchen — which means turning some away before they ever reach it.

This is the sentence worth tattooing somewhere visible: rate limiting isn't a performance optimization, it's a load-shedding strategy. You are not making requests faster. You are deciding which requests get served when you can't serve all of them, and the algorithm you pick is that decision, encoded.

If you've worked an incident, you've felt the moment this matters. It's 9pm, a marketing email just went out to the whole list, and the graphs are bending the wrong way. Someone says "can we just spin up more pods?" and you do, and for ninety seconds it looks like it helped, and then the database connection pool saturates and the new pods are simply more processes waiting in the same line. That's the moment the room gets quiet. Adding capacity didn't shed load; it just spread the same starvation across more machines. The only lever that actually moves anything is the one that turns some requests away before they reach the part that's drowning — and if you didn't build that lever in advance, you're now building it at 9pm with everyone watching.

There's also the cascade, which is how a local problem becomes a company-wide one. Accept everything, process it slower than it arrives, watch the queues grow, watch memory fill, and eventually something dies — and it takes its neighbors with it, because everything that depended on the dead service is now also having a bad day. Rate limiting is the thing that keeps the load entering your system bounded even when the demand outside it isn't. The demand is allowed to be insane. The intake is not.

A few numbers for scale. Netflix serves 200+ million streaming users, and on the order of 15% of their traffic is retries — clients that hit a transient blip and tried again. Sit with that for a second: roughly one in seven requests exists only because an earlier request didn't land. Without something holding the line, a small bump in errors becomes a retry storm that feeds on itself, and the retries are now the outage. Stripe runs payment APIs where one customer's broken integration absolutely cannot be allowed to degrade everyone else's ability to take money. Uber handles surge events where the flood of "is there a car near me" queries threatens the exact systems needed to actually dispatch the cars. Same fundamental problem in all three. Wildly different right answers, because they were optimizing for different constraints — and that's the whole point of this chapter.

Everyone who has implemented rate limiting for the first time has written the same thing, give or take a variable name. You keep a counter and the timestamp of when the current minute started. A request comes in; if the minute's up, you reset the counter and restart the window; if the count is under the limit you allow it and increment; otherwise you reject. A counter, a clock, and a reset. (Full code: Appendix A.1.)

This is fixed window rate limiting, and its charm is obvious: you can understand it in thirty seconds, explain it to your manager in one sentence, and ship it before lunch. That charm is exactly the trap. It looks so reasonable that nobody stress-tests the assumption baked into it, and it has two failure modes that only show up once real traffic is leaning on it.

The boundary spike problem. Say your limit is 100 requests per minute. A client sends 100 requests in the last second of minute N. The counter resets at the boundary. They send 100 more in the first second of minute N+1. Congratulations: you just served 200 requests inside a two-second span, and your rate limiter, which was specifically built to prevent this, signed off on every one of them and felt good about itself. The limit said "100 per minute" and the system technically obeyed — across two different minutes that happened to share a fence post. Anything downstream that genuinely cannot survive 200 requests in two seconds now has your full attention.

The reason smart people miss this is that the per-window math is correct. Each window really did stay under 100. The bug isn't in the counting; it's in the belief that "100 per fixed minute" means "100 per any minute." It doesn't. It means "100 per the specific minutes I happen to draw boundaries around," and your clients did not agree to respect your boundaries. The ones being throttled will naturally pile up against the reset and release the instant it flips. Scrapers will figure out your window timing and aim for the seam on purpose. You didn't build a limit; you built a metronome with a vulnerability on the downbeat.

The distributed systems problem. Your service runs on N instances. Each one keeps its own counter, because that was the easy thing and it worked fine in staging where N was 1. In production your real ceiling is N × LIMIT, your SLA still says LIMIT, and the gap between those two numbers is exactly as large as your fleet. The obvious fix is a shared counter in Redis — which trades the problem for a network round-trip on every request, a brand-new single point of failure, and latency welded directly into your hot path. This is the recurring tax of distributed systems: correctness across machines is never free, and anyone who tells you otherwise is selling a benchmark run on one machine.

There's a third problem that gets less airtime: the signal you send when you reject. Return a vague error with no hint about when to come back, and you've solved this second's load problem by manufacturing next second's. A client that gets an uninformative rejection does not go make a cup of tea and reflect. It retries. Immediately. Which generates more load, which generates more rejections, which it retries again. Your rate limiter has quietly built a positive feedback loop that amplifies the exact thing it was deployed to suppress. We'll come back to this, because it's important enough to deserve its own argument later.

Before the algorithms, the field guide — the things that break in production that the textbook chapter on rate limiting tends to leave out, presumably because they're embarrassing.

Clock skew. Rate limit windows are built on time, and time is more negotiable than you'd like. If two servers disagree about what second it is, their windows don't line up, and a limit you reasoned about as one thing is quietly several slightly-different things. This sounds like a thought experiment until you've lived through a leap second. In 2012, a Linux kernel bug turned a one-second leap-second adjustment into correlated CPU spikes across a long list of major companies at the same instant — systems that were individually fine fell over together because the one assumption they all shared, that time moves forward smoothly, briefly stopped being true. Your rate limiter assumes synchronized clocks. They are synchronized well enough, most of the time, which is a sentence that has preceded a lot of incidents.

Thundering herd at recovery. This is the one that gets everybody, including people who've been doing this for fifteen years.

You ship rate limiting. It works. Load spikes, the limiter rejects the excess, the spike passes, the window resets. Clean. Except here's what actually happened while you were congratulating yourself: every client you rejected was waiting for that reset. And when it comes, they don't trickle back politely — they all return at the same microsecond, because you trained them to. You didn't remove the overload. You compressed it into a periodic spike and scheduled it for the worst possible moment, which is right after a recovery, when your system is at its most fragile and your caches are cold.

The tell is unmistakable once you've seen it. Pull up the request-rate graph after deploying naive fixed-window limiting and you'll find a spike every 60 seconds, regular as a pulse, as the windows reset and the dammed-up traffic comes through all at once. The limiter solved the overload problem and replaced it with an oscillation problem, and oscillation problems are how engineers end up staring at a heartbeat-shaped graph at 2am asking why traffic peaks precisely on the minute mark. The cruel part is that the better-behaved your clients are — the more faithfully they honor your reset — the sharper the spike, because well-behaved clients synchronize beautifully. You got punished for your users' good manners.

Rate limiting the wrong thing. This is a design decision whose consequences you will discover in production if you skip thinking about it now.

Your options, and what each one costs you:

• IP address: No authentication needed, which is convenient right up until you remember VPNs, corporate proxies, and NAT. Thousands of humans routinely share one IP. Limit by IP and you will, eventually, throttle an entire 5,000-person company because one employee left a weekend script running in a loop. They will not find this funny.
• User ID: Genuinely fair per user, at the cost of requiring authentication on every limited request and per-user state that grows right alongside your user count.
• API key: Usually the right primary answer — it isolates integrations, doesn't punish a shared IP, and gives you granular control. The catch is that keys get shared, leaked, and committed to public repos with the enthusiasm of a golden retriever. (This is not hypothetical. Go look at what's in your dependency graphs. Take a breath first.)

The grown-up answer is layered: IP-based limiting as a blunt first line against unauthenticated abuse, API-key limits as the real fairness mechanism, and per-user limits reserved for specific sensitive operations. One layer is a policy. Three layers is a defense.

Memory explosion. Track rate limit state per (user × endpoint × source_ip × time_window) and your storage grows with the product of those dimensions, which is a multiplication that gets away from you fast. Millions of users, dozens of endpoints, and suddenly your rate limiter — a thing meant to protect you — is the component paging someone about memory. There are real tools for this: sorted sets in Redis, count-min sketches, approximate counters. They work. But you want to have chosen one before deployment, not while watching the Redis memory graph climb toward the ceiling with the unhurried confidence of something that has done this before.

The backpressure signal problem. When you reject a request, the response isn't just an outcome — it's an instruction. The status code tells the client what to do next, and clients take it literally:

• HTTP 500 → "the server broke." Client retries, aggressively, because surely it'll work this time.
• HTTP 400 → "I broke." Client may stop retrying forever, including the requests that would've succeeded.
• HTTP 503 → "temporarily down." Many clients read this as "try again basically now."
• HTTP 429 with Retry-After → "you're over the limit; come back in N seconds." Client waits, then retries on schedule.

The right answer is 429 with Retry-After. This isn't etiquette — it's flow control. You are handing the client the one piece of information that lets it space its retries instead of guessing, and a client that guesses will guess in a way that synchronizes with every other guessing client and rebuilds the thundering herd you just spent a section learning to fear.

The token bucket is the workhorse. If you only ever learn one of these cold, learn this one.

Picture a bucket that holds N tokens. Tokens drip in at a fixed rate R per second, up to the bucket's capacity, and then they stop — the bucket doesn't overflow into credit you can hoard forever. Each request spends one token. Empty bucket, no token, request rejected. The implementation is just that sentence in code: on each request, add however many tokens have accrued since you last looked (capped at capacity), then spend one if you can. (Full code: Appendix A.2.)

That separation matters because real traffic is bursty in ways that are completely legitimate. A user opening your mobile app fires off several requests at once — auth, preferences, initial data — and they all land in the same heartbeat. A strict per-second limit would reject some of those and the user would experience your perfectly healthy product as flaky. Token bucket handles this without you having to special-case it: a client that's been idle has saved up tokens, so the natural startup burst sails through, and only a client trying to sustain that rate runs dry.

Stripe states this in their public docs almost verbatim in token-bucket terms: "100 requests per second, burst to 500." That's refill_rate=100, capacity=500. A well-behaved client cruising under 100 RPS never even notices the limit exists. A client that suddenly needs to backfill a pile of data gets 500 requests of grace before the ceiling lowers onto it. Everyone gets what they actually need and nobody gets to abuse it for long.

Where token bucket falls short: it's stubbornly per-entity. It does nothing to protect a shared downstream resource from the aggregate of many clients. Give 10,000 clients a tidy 100-RPS bucket each and they will, collectively, present 1,000,000 RPS to your database while every single one of them is technically a model citizen within their own limit. Your limiter is satisfied. Your database is not. For aggregate protection you need a global limit, and a pile of per-entity buckets does not add up to one.

Use when — per-entity limiting where accommodating bursts makes the product feel better:

• API-key limits
• Per-user limits
• Single-client limits

The leaky bucket looks like token bucket's twin and behaves like its opposite. Token bucket asks "do you have a token to spend?" Leaky bucket says "get in line; you'll be served at a fixed pace and not one bit faster." It enforces a strictly constant output rate no matter how spiky the input.

Requests enter a queue — the bucket. A process drains that queue at exactly rate per second. Queue's full when a request arrives, request gets rejected. In code it's a queue plus a drain step: before each new request you remove however many requests should have leaked out since you last checked, then accept the newcomer only if there's room left. (Full code: Appendix A.3.)

The behavioral gap between the two is the whole story. Send 100 requests at the same instant:

• Token bucket (capacity=100): takes all 100 immediately and lets your service chew through them as fast as it can.
• Leaky bucket (rate=10/sec, capacity=100): takes all 100 into the queue, then releases them at exactly 10 per second. Request number 100 leaves ten seconds after it arrived, having waited its turn whether it wanted to or not.

Leaky bucket turns a jagged input into a flat output. What you trade for that flatness is responsiveness: clients eat queuing latency even when they're nowhere near abusing anything. For user-facing traffic that's usually the wrong deal — nobody wants their dashboard to load slower so that the rate limiter can feel tidy. But for traffic flowing out to systems with their own hard limits — an SMS provider, a webhook endpoint, a partner API that will ban you if you exceed their quota — it's exactly right. The leaky bucket guarantees you never breach the downstream ceiling regardless of how much your own users are throwing at you, which is a guarantee you'll be glad to have in writing when the partner's account manager emails.

Use when — you need a strict, predictable output rate into a downstream system:

• SMS notification pipelines
• Webhook delivery
• Any integration where the other side has a hard limit you're contractually or practically forbidden to cross

The sliding window log is the "obviously correct" design — the one you'd write if accuracy were the only thing you cared about and memory were free, which it is right up until it isn't.

For each entity, keep a sorted log of every request timestamp. To decide on a new request, count how many entries fall within the last window duration. Under the limit, accept and append the timestamp. Over, reject. No fence posts, no boundaries, no seam to exploit. The code is exactly that: drop the timestamps older than the window, then check the length of what's left. (Full code: Appendix A.4.)

This kills the fixed-window boundary spike completely. There's no reset to game because there's no window in the fence-post sense — the limit is computed over a true rolling period. A client who fired 100 requests at t=59s carries every one of them in the log until t=119s, and the minute boundary it happened to straddle is irrelevant. The math is finally exactly what the limit promised.

The cost: memory proportional to (requests per entity × window duration). At 10,000 requests per second per entity over a 60-second window, that's 600,000 timestamps. Per entity. Multiply by millions of entities and you've designed a rate limiter whose storage footprint rivals the data it's protecting, which is the kind of thing that looks fine in the design review and ominous in the capacity dashboard.

Use when — accuracy is non-negotiable and volume is bounded:

• Payment gateway limits
• Compliance-sensitive APIs
• Anything where a single violation has a direct business consequence — a billing error, a regulatory audit trail, a number that has to be exactly right because someone will eventually check it

There, the memory bill is a rounding error next to the cost of being wrong. Everywhere else, your finance team would like a word about why the rate limiter has a bigger RAM budget than the thing it's protecting — so keep reading.

The sliding window counter is the hybrid that gets you most of the log's accuracy for a rounding error of its memory. It's what a large share of high-traffic systems actually run, for the unglamorous reason that it works and it's cheap.

The trick: don't log every timestamp. Approximate the count in the sliding window by blending two adjacent fixed windows, weighted by how much of the previous window still overlaps the rolling one. You keep two counters — this window's and last window's — and when the window rolls over, last becomes this and this resets to zero. The estimate is last window's count scaled by its overlap, plus this window's count so far. (Full code: Appendix A.5.)

Run the arithmetic once by hand and it clicks. Window is 60 seconds. You're 45 seconds into the current one. The previous window saw 60 requests; this one has seen 30 so far.

overlap_ratio = 1 - (45/60) = 0.25, so a quarter of the previous window is still inside your rolling view. Estimated count = 60 × 0.25 + 30 = 15 + 30 = 45.

It's an approximation, and it's worth being honest about how rough. The true number could be anywhere from 30 (if all of last window's requests landed in the non-overlapping 75%) to 90 (if they all bunched into the overlapping 25%). But in practice the error stays under about 10%.

Memory cost: two integers per entity. O(1). That number, more than any elegance argument, is why it wins.

Where it breaks down: high-value operations where "approximately" isn't good enough. Billing, compliance limits, anything where a 10% overage translates into real money or a real audit finding. Those go to the sliding window log and pay the memory bill on purpose.

Use when — most production API rate limiting, full stop. This is your default starting point:

• High-RPS APIs where two integers per entity is the kind of memory budget you want
• Feed and timeline endpoints (Instagram uses this shape for feed requests)
• Anything where you can't name a specific need for exact accuracy out loud

Unless you have that specific accuracy requirement, this is where you start.

Accuracy vs. Latency of the Check

Every rate limit check adds latency to the request path, and "every" is doing a lot of work in that sentence. An in-memory check on local state costs nanoseconds. A Redis check costs 1–5ms. That sounds like nothing until you do the multiplication: at 100k RPS, every extra millisecond of check latency is roughly 100 more requests sitting in your connection pool at any instant, doing nothing but waiting. Your rate limiter, the component whose job is to relieve pressure, is now itself a source of it.

Stripe's call: distributed sliding window counter with eventual consistency. They explicitly tolerate transient overages of a few seconds while counter state propagates between nodes. That's a deliberate trade — slightly fuzzier limiting in exchange for measurably lower latency on every request. For API-key limits where a handful of extra requests inside a two-second window costs essentially nothing, it's plainly the right call. The thing to take away isn't "be eventually consistent." It's that Stripe named the cost they were willing to eat and chose accordingly, and your trade-off math might land somewhere else entirely.

Burstiness Tolerance

The intuitive answer is "strict limits are safer." The intuitive answer is wrong, or at least too simple to be useful.

Real users hit your API in bursts, because that's what using software looks like — opening an app, running a checkout, loading a dashboard all fire several requests in a tight cluster, frequently in parallel. Clamp down with a strict per-second limit and you'll reject a chunk of completely ordinary behavior and teach your users that the product is unreliable. You will have successfully protected the system from its own customers.

Token bucket's capacity is the dial here. Set capacity to 5× refill_rate and a client can spend five seconds' worth of requests instantly — absorbing the startup burst — while still being unable to sustain that pace. Size it against the bursts you actually observe from legitimate clients, then throw synthetic attack traffic at it to confirm it still says no when it should. Tuning to real data and skipping the adversarial test is how you end up with a limit that's generous to users and equally generous to bots.

The failure mode lives at the other extreme: a burst capacity so large it protects nothing. capacity=10000, refill_rate=100 lets a client fire 10,000 requests in one second before the limit so much as clears its throat. If 10,000 requests in a second is what melts your database, then your rate limiter is a very expensive return True. It will pass every audit, satisfy every config review, and protect you from precisely nothing on the night it matters.

Per-User vs. Global Limits

These two goals fight each other, and pretending they don't is how systems get half-protected. Global limits cap total RPS and protect the infrastructure, but they let one user hog the whole allowance — the loudest client wins again. Per-user limits enforce fairness but demand per-user state that doesn't love scaling across every endpoint you own.

Netflix's answer: per-user soft limits for fairness, global hard limits for protection, both live at once. The per-user limits stop one customer's bug from ruining everyone's evening; the global limits stop the whole fleet from saturating no matter how the per-user math shakes out.

Stripe's answer: per-API-key limits, strict and isolated. One customer's integration meltdown is fenced inside their own key, and nobody else's SLA so much as flinches. The price is more complex state management, and for a company that exists to move other people's money, that price isn't even a discussion.

Retry Signaling

We touched this earlier; it earns a second pass, because your retry response design quietly determines how much load your rate limiter actually generates. You're not just rejecting a request — you're programming the client's next move.

HTTP 429 + Retry-After: correct. "You're limited; come back in N seconds." Well-behaved clients back off and return on time, and the burst gets smeared out instead of stacked up.

HTTP 429 without Retry-After: better than the alternatives, but now you've asked the client to guess, and you'll get a spread of retry times that's accidentally decent load distribution. Some wait, some come right back. It works by luck rather than design, which is fine until your luck changes.

HTTP 503: a number of client libraries read this as "server's having a moment, try again shortly" and retry almost immediately. You wanted to shed load and instead you've issued an invitation. Now you have a retry storm wearing a 503.

Silent drop (no response): the client assumes the connection died, retries immediately, and often does so with more aggressive connection behavior than before. This is the worst outcome on the board, and it's the default you get by doing nothing thoughtful.

Stripe's 429 carries the limit that was exceeded, the current count, and the reset time, and their SDK turns that into automatic backoff. The result is that most customers who hit Stripe's rate limits have no idea it ever happened — the SDK absorbs it silently. That's the bar: rate limiting so well-signaled that it's invisible to everyone behaving reasonably, and only the abusers ever see the wall.

Diagram 1: Token Bucket Refill Over Time

Plot two lines on the same chart. X-axis: time (0–60 seconds). Y-axis: tokens available (0 to capacity). The first line shows the refill curve—starting from some depleted state and climbing linearly toward capacity, capped at the top. The second line shows request consumption—tokens dropping with each accepted request, then refilling. The rejection zone sits below the x-axis intercept: where the token count is zero and new requests are turned away.

What the diagram should make viscerally clear: a client at full capacity can send a burst immediately. A client that has been running at refill rate has accumulated tokens equal to elapsed time × refill rate—they have a small buffer, but not a large one.

Diagram 2: Fixed Window Boundary Attack

Two adjacent windows (0–60s and 60–120s). Show 100 requests arriving at t=58–60s (end of first window) and another 100 arriving at t=60–62s (start of second window). Both windows show "under limit." Between them, mark the two-second span: 200 requests. Annotate downstream: "Database receives 200 requests in 2 seconds. Database expected 100/minute."

Diagram 3: Sliding Window Counter Across Two Windows

Two 60-second windows. The current time sits at the 45-second mark of the second window. Draw arrows showing the overlap: 25% of the first window (15 seconds) overlaps with the sliding window. The calculation is visible: previous_count × 0.25 + current_count = estimated_count.

Diagram 4: Distributed Rate Limiting

Multiple application servers → Redis (centralized counter) → response. Annotate with the round-trip latency (1–5ms) on the Redis path. Mark the Redis node as a potential single point of failure. Draw a dotted fallback arrow: if Redis is unreachable, degrade to in-process counters with a "degraded accuracy" label. This diagram should communicate that distributed rate limiting is not free—you're trading accuracy for consistency, and you need a failure plan.

Netflix: Token Bucket for Burst Accommodation

Netflix traffic has a shape you could set your watch to: evenings and weekends run 3–5× the baseline, and inside those peaks, individual users fire sharp little bursts every time someone opens the app to argue about what to watch. Per-user token bucket with a generous burst capacity handles both layers without breaking a sweat — the macro daily wave and the micro per-user spikes.

Their early version used 60-second fixed windows, and it failed in exactly the way the failure-modes section warned about: throttled clients all released at the minute boundary and produced a load spike like clockwork, every 60 seconds, forever. Moving to token bucket dissolved the synchronized reset — clients deplete tokens at their own pace and replenish at their own pace, so there's no shared boundary for them to pile up against. No metronome, no heartbeat-shaped graph, no 2am.

But the decision that mattered more than the algorithm was operational: rate limit configuration lives in a centralized config service, not baked into application code. The reasoning is the kind you only fully appreciate mid-incident. When a runaway integration is hammering the API right now, you need to tighten that client's limit in seconds. If turning the screw requires a deploy, your time-to-remediation is measured in minutes on a good day and "after the build queue clears" on a bad one. Externalized config turns a deploy into a config push, and that's the difference between handling the incident and narrating it.

Stripe: Distributed Sliding Window with Intentional Overage Tolerance

Stripe's constraints are unusual in a specific way: per-API-key limits that hold across multiple geographic regions, accurate enough to stop real abuse, but never so strict that an ordinary payment flow gets interrupted. A falsely rejected payment isn't a retry — it's a lost sale and a dent in trust.

Their design: a sliding window counter per API key, backed by a distributed store, with explicit tolerance for transient overages while state propagates between regions. A client might run slightly over its limit for two or three seconds while the counter updates replicate. Stripe's judgment is that a tiny, brief overage is a fine price to avoid the latency and availability risk of synchronous global consensus on every request. They decided the roughly 0.1% accuracy loss was worth roughly 10ms per request, wrote that trade down, and moved on. The lesson is the explicitness, not the specific numbers.

Where they spent beyond the algorithm is the communication. Their 429 looks like this:

1HTTP/1.1 429 Too Many Requests
2X-RateLimit-Limit: 100
3X-RateLimit-Remaining: 0
4X-RateLimit-Reset: 1609459200
5Retry-After: 30

That gives a client everything it needs to back off intelligently without guessing a single value, and the official client library acts on it automatically. The payoff is that most Stripe customers who hit a rate limit never find out — the SDK quietly waits and retries, and the developer's only evidence is that everything kept working.

Instagram: Soft Limits and Gradient Degradation

Instagram made a deliberate product call: for user-facing requests, a hard binary rejection is the wrong tool. A 429 is visible. It feels like the app broke. A response that's a hair slower is invisible, and invisible is almost always the better failure.

So instead of a wall, they built a slope. As a user approaches their limit, the system starts responding a little more slowly — letting queue depth rise, trimming the cost of ranking computations. The feed comes back slightly less personalized and slightly less fresh, and the user perceives "a touch sluggish today," not "error." Only when someone blows well past the limit does the system start actually rejecting anything.

This is meaningfully harder to build than binary limiting, and it's worth being clear-eyed about that before you copy it. You need state tracking across multiple threshold tiers, fallback logic in the serving layer that can produce a degraded-but-still-useful response, and enough testing to make sure the degraded path doesn't quietly become its own reliability hazard. What you buy with all that complexity is a user experience that's genuinely better and a limit that most people who hit it never consciously notice. For a consumer product where engagement is the business, that's a trade worth the engineering.

Uber: Priority Queuing Under Surge

Uber's problem during a surge has a different shape entirely: the requests are not equal. A request updating an in-progress ride match matters more than a new ride request, which matters more than someone idly checking historical trip prices. A flat rate limit that drops the first kind to make room for the third isn't just suboptimal — it's actively choosing the wrong thing to protect.

So they don't apply a flat limit. They tier it. Tier 1 — active ride management, in-progress match updates, driver location — always passes, no matter what. Tier 2 — new ride requests during surge — is limited to whatever current capacity allows. Tier 3 — exploratory stuff like price estimates, historical data, browsing surge zones — gets shed first the moment things tighten.

This is rate limiting as business policy compiled down into infrastructure. The algorithm is almost beside the point; what carries the design is the classification — deciding which requests are load-bearing for the actual business and being willing to fail the rest to protect them. It requires understanding your product deeply enough to rank your own traffic, and then it requires the spine to defend that ranking when a PM walks over to ask why the price-estimate endpoint is throwing errors during the busiest hour of the week. ("Because we'd rather drop a price preview than drop a ride in progress" is the correct answer, and you should have the graph ready.)

Redis: The Standard Choice for Distributed Limiting

The token bucket from earlier, moved into Redis: the bucket's tokens and last-refill time live in a Redis hash, and the whole refill-and-spend calculation runs as a Lua script executed on the Redis server itself. (Full code: Appendix A.6.)

The Lua script is the part that's easy to skip and expensive to skip. It makes the read-modify-write a single atomic operation, which is the only thing standing between you and a race where two instances both read the same token count, both decide there's room, and both spend it. A non-atomic version will let through more requests than your limit allows under exactly the concurrent load you built the limiter to survive — which is to say it fails precisely when tested. Atomicity here is not an optimization. It's the feature.

Pros: centralized, consistent across every instance, works across data centers with replication. Cons: 1–5ms of network latency on every check, Redis is now a dependency of every request path you protect, and it's a single point of failure unless you've made it not one.

Nginx: Edge-Level Protection Before Your Application

You don't write this one — you configure it. A couple of limit_req_zone lines declare zones keyed on whatever you want to limit (the client IP, an API-key header) along with each zone's rate, and a limit_req line inside your location block applies them, with a burst allowance so short spikes queue instead of getting rejected outright. (Full config: Appendix A.7.)

Nginx limiting runs in the worker process before a single line of your application code executes, which makes it fast and lets it absorb request rates that would make a Redis round-trip cry. It's also an independent line of defense against floods that never bother to authenticate — the unauthenticated junk dies at the edge and never learns your app exists.

The limitation is the same one fixed windows had: each Nginx instance keeps its own state, so total throughput across N instances is N × limit. For broad IP-based protection that's a feature — you want each edge node enforcing locally without a coordination round-trip. For per-API-key accuracy, where you need one global view of a key's usage, it's the wrong layer, and trying to force it to be the right one will hurt.

In-Process Token Bucket (Single Instance or Low-Volume)

For a service that runs as a single instance, or where approximate per-instance limiting is genuinely good enough, it's the in-memory token bucket from earlier with one addition: a lock around the refill-and-spend so two threads can't both read the same count and both decide there's room. (Full code: Appendix A.8.)

The lock keeps concurrent requests from corrupting the token count. Latency is excellent — nanoseconds, no network — and the catch is the one you already know: state is local. Multiple instances each enforce the limit on their own, and your real fleet-wide ceiling is N × limit. That's totally fine for a single instance or a low-stakes limit, and quietly wrong if you forget it and start quoting the per-instance number as the system guarantee.

Decision Framework by Scale

• < 10k RPS: In-process token bucket per instance. Simple, fast, no dependencies. Don't overthink it.
• 10k–100k RPS: Redis-backed sliding window counter. Centralized accuracy at an acceptable latency cost.

• 100k RPS: Nginx edge limiting for the broad strokes, plus Redis for per-entity granularity on the specific high-value endpoints that earn it. Routing every request at this scale through Redis means betting your entire API's availability on one dependency's worst day, which is a bet you'll regret on the day it loses.

These don't have clean answers, and that's the point. You can paste any of them into an AI and get a confident, well-structured response in about four seconds — and it will read perfectly and quietly skip the one thing that actually matters: your system, your constraints, your blast radius. The value isn't in the answer, it's in the arguing. Sit with each one long enough to disagree with your first instinct before you go looking for someone — silicon or otherwise — to agree with you.

Exercise 1: The Shared IP Problem

You rate limit by source IP. One of your largest customers is an enterprise with 3,000 employees, all egressing through the same corporate proxy. They hit your rate limit constantly, despite each individual user being within normal usage. Several of these employees are key contacts at an account worth seven figures annually.

Design a rate limiting strategy that handles this. What information do you need that you don't currently have? What are the trade-offs of using it? How do you handle the transition without disrupting their usage during the migration?

Hint: This is the exact reason Stripe rates by API key rather than IP. But API keys can be shared or leaked—so Stripe also has IP-based limiting as a separate secondary layer with different thresholds and different purposes. The two layers solve different problems and shouldn't be conflated.

Exercise 2: The Retry Storm

Your rate limiter returns HTTP 429 with no Retry-After header. Clients that receive a 429 retry after a fixed 1-second delay. At moderate load, this creates a feedback loop: rate limit → 1-second wait → synchronized retry storm → rate limit.

Design the signaling strategy your 429 responses should include to break this loop. What information should a well-behaved client receive? What headers, what values? How do you handle clients that ignore the Retry-After header entirely?

Hint: Retry-After handles the well-behaved clients. For the rest, you need a second line of defense. Chapter 5's discussion of jitter in retry backoff is directly relevant—the retry storm problem and the rate limiting signal problem are two sides of the same coin.

Exercise 3: The Burst Problem

Your SLA allows 10,000 requests per second. Actual traffic profile:

• Normal: 5,000 RPS
• Evening peak (8pm–10pm): 18,000 RPS for approximately 20 minutes
• Sustained maximum without degradation: 14,000 RPS

Design a token bucket configuration that accommodates the evening peak without causing sustained overload. What capacity and refill_rate settings would you choose? How do you verify empirically that the configuration works before deploying it?

Hint: Token bucket capacity governs burst duration: a full bucket at capacity C with refill rate R can sustain a burst of (C / (burst_rate - R)) seconds before the bucket empties. At 18,000 RPS with a refill rate of 10,000 RPS, the net consumption rate is 8,000 tokens/second. With capacity = 240,000, the bucket empties in 30 seconds. Whether that's acceptable depends on how long your evening peaks actually sustain at 18,000 RPS.

Exercise 4: The Observability Question

A customer calls your support line claiming they're being rate limited unfairly. They say they're sending only 50 requests per minute but hitting the limit constantly.

What data do you look at to answer this? Do you currently have that data? What would you need to add to your rate limiting implementation to have it? How would you distinguish between a misconfigured rate limit, a customer miscounting their own requests, and a customer sharing their API key with multiple systems?

There's no single right answer here. This is the kind of question that exposes gaps in your observability when you encounter it in production for the first time. The answer to "do you have this data?" is frequently no.

→ Chapter 2 (Idempotency): Rate limiting causes client retries. Retries are safe only if the handlers being retried are idempotent. Before deploying aggressive rate limiting on a high-value endpoint, verify that endpoint handles duplicate requests correctly—otherwise you've solved the load problem while making the correctness problem worse.

→ Chapter 3 (Multi-Level Caching): Rate limit checks can be layered with caching. Check rate limit state from a local in-process cache first (fast, slightly stale), fall through to Redis only on cache miss. This hybrid approach reduces the per-request cost of distributed rate limiting at the expense of some accuracy during the cache TTL window.

→ Chapter 4 (Circuit Breakers): Rate limiting is proactive load shedding—you decide to reject requests before observing failure. Circuit breakers are reactive—they detect that a downstream service is failing and stop sending requests to it. The two patterns are complementary: rate limiting protects your service from too much good traffic; circuit breakers protect your service from broken dependencies.

→ Chapter 5 (Retries): The retry behavior you get from clients is directly shaped by how you signal rate limiting. Your 429 response design is an input to the retry algorithm in Chapter 5. The two chapters should be read together—they are two sides of the same protocol.

→ Chapter 12 (Dead Letter Queues): When rate limiting rejects requests that represent durable work (write operations, critical notifications), those requests may need to go somewhere rather than being silently dropped. DLQs provide a mechanism for work that couldn't be processed immediately to be deferred and retried later, trading latency for reliability.

The chapter keeps the code out of the narrative on purpose. Here it is, collected, runnable, and annotated. Each entry is the full version of something described in plain language in the body — read the prose for the idea, come here when you want to type it out. All of it is illustrative rather than production-hardened: the in-memory versions skip the distribution problem, and none of them handle every edge you'll meet in the real thing.

A.1 — Fixed Window (the naive solution)

1counter = 0
2window_start = time.now()
3 
4def handle_request():
5    global counter, window_start
6    
7    if time.now() - window_start > 60:
8        counter = 0
9        window_start = time.now()
10    
11    if counter < LIMIT:
12        counter += 1
13        process_request()
14    else:
15        reject()

A.2 — Token Bucket

1class TokenBucket:
2    def __init__(self, capacity, refill_rate):
3        self.capacity = capacity
4        self.tokens = capacity        # Start full
5        self.refill_rate = refill_rate  # tokens per second
6        self.last_refill = time.time()
7    
8    def allow(self):
9        now = time.time()
10        # Add tokens for time elapsed since last check
11        elapsed = now - self.last_refill
12        self.tokens = min(
13            self.capacity,
14            self.tokens + self.refill_rate * elapsed
15        )
16        self.last_refill = now
17        
18        if self.tokens >= 1:
19            self.tokens -= 1
20            return True
21        return False

refill_rate is the sustained ceiling; capacity is the burst ceiling. They're independent knobs — see the Token Bucket section.

A.3 — Leaky Bucket

1class LeakyBucket:
2    def __init__(self, capacity, leak_rate):
3        self.queue = deque()
4        self.capacity = capacity
5        self.leak_rate = leak_rate     # requests processed per second
6        self.last_leak = time.time()
7    
8    def allow(self, request):
9        self._leak()  # Process pending requests first
10        
11        if len(self.queue) < self.capacity:
12            self.queue.append(request)
13            return True   # Accepted into queue
14        return False       # Queue full, reject
15    
16    def _leak(self):
17        now = time.time()
18        elapsed = now - self.last_leak
19        requests_to_drain = int(elapsed * self.leak_rate)
20        for _ in range(min(requests_to_drain, len(self.queue))):
21            self.queue.popleft()
22        self.last_leak = now

A.4 — Sliding Window Log

1class SlidingWindowLog:
2    def __init__(self, limit, window_seconds):
3        self.limit = limit
4        self.window = window_seconds
5        self.log = []  # sorted list of timestamps
6    
7    def allow(self):
8        now = time.time()
9        cutoff = now - self.window
10        
11        # Remove entries outside the window
12        self.log = [t for t in self.log if t > cutoff]
13        
14        if len(self.log) < self.limit:
15            self.log.append(now)
16            return True
17        return False

Exact, and memory-hungry: storage grows with requests-per-entity × window. Fine at bounded volume, impractical at millions of entities.

A.5 — Sliding Window Counter

1class SlidingWindowCounter:
2    def __init__(self, limit, window_seconds):
3        self.limit = limit
4        self.window = window_seconds
5        self.current_count = 0
6        self.previous_count = 0
7        self.current_window_start = time.time()
8    
9    def allow(self):
10        now = time.time()
11        elapsed = now - self.current_window_start
12        
13        if elapsed >= self.window:
14            # Rotate windows
15            self.previous_count = self.current_count
16            self.current_count = 0
17            self.current_window_start = now
18            elapsed = 0
19        
20        # How much of the previous window overlaps with our sliding window?
21        overlap_ratio = 1.0 - (elapsed / self.window)
22        estimated_count = (
23            self.previous_count * overlap_ratio 
24            + self.current_count
25        )
26        
27        if estimated_count < self.limit:
28            self.current_count += 1
29            return True
30        return False

Two integers per entity, error typically under 10%. The production default.

A.6 — Token Bucket in Redis (atomic Lua)

1# Token bucket in Redis using atomic Lua script
2lua_script = """
3local key = KEYS[1]
4local capacity = tonumber(ARGV[1])
5local refill_rate = tonumber(ARGV[2])
6local now = tonumber(ARGV[3])
7 
8local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
9local tokens = tonumber(bucket[1]) or capacity
10local last_refill = tonumber(bucket[2]) or now
11 
12local elapsed = now - last_refill
13tokens = math.min(capacity, tokens + refill_rate * elapsed)
14 
15if tokens >= 1 then
16    tokens = tokens - 1
17    redis.call('HMSET', key, 'tokens', tokens, 'last_refill', now)
18    redis.call('EXPIRE', key, 3600)
19    return 1
20end
21return 0
22"""

The Lua script makes the read-modify-write atomic. A non-atomic version races under exactly the concurrent load you built the limiter to survive.

A.7 — Nginx Edge Configuration

1limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;
2limit_req_zone $http_x_api_key zone=api_key:50m rate=1000r/s;
3 
4server {
5    location /api/ {
6        limit_req zone=api burst=50 nodelay;
7        limit_req zone=api_key burst=200 nodelay;
8    }
9}

State is per-instance: total throughput across N instances is N × the rate. A feature for broad IP protection, a problem for per-key accuracy.

A.8 — Thread-Safe In-Process Token Bucket

1from threading import Lock
2import time
3 
4class ThreadSafeTokenBucket:
5    def __init__(self, capacity, refill_rate):
6        self.capacity = capacity
7        self.tokens = capacity
8        self.refill_rate = refill_rate
9        self.last_refill = time.time()
10        self.lock = Lock()
11    
12    def allow(self):
13        with self.lock:
14            now = time.time()
15            elapsed = now - self.last_refill
16            self.tokens = min(
17                self.capacity,
18                self.tokens + self.refill_rate * elapsed
19            )
20            self.last_refill = now
21            
22            if self.tokens >= 1:
23                self.tokens -= 1
24                return True
25            return False

Nanosecond latency, no network — but local state, so each instance limits independently. This is also the fail-safe mode the recovery section refers to: when the central store is unreachable, every server falls back to this.

Next: Chapter 2 — Idempotency: building systems where the second time is indistinguishable from the first.