A shared rate limit can keep an API’s total traffic within capacity while letting one customer consume the available requests and leave everyone else throttled. That is the failure Sergey Shinder describes: the limit protected the service overall, but did not allocate capacity fairly. His account offers a useful design lesson, though its incident figures are the author’s report and have not been independently verified.
What happened in the reported incident
In a DEV Community article, author Sergey Shinder says a customer began a historical API backfill. Within ten minutes, 112 other customers were being rejected. The edge was using one shared token bucket configured for 2,000 requests per second across the service, rather than separate buckets keyed to customers.
As an Amazon Associate I earn from qualifying purchases.
In a shared token bucket, requests draw from the same pool. As tokens become available, a caller producing sustained high traffic can consume them before quieter callers make their requests. Shinder says the backfill ran at around 40 requests per second. He reports that availability for the hour was 91% in aggregate, but closer to 30% for the 112 customers who were not doing anything unusual. Those are figures reported in his account, not independently corroborated service telemetry.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhy a service-wide limit did not make allocation fair
A global cap answers a capacity question: how much traffic should the service accept in total? It does not answer an allocation question: how much of that capacity should each customer be able to use? If every request competes for one pool, the caller that keeps arriving fastest can dominate it. The cap may still constrain total load, yet individual customers can experience very different outcomes.
As Shinder puts it, “A limit protects the service. It says nothing about who gets what, and where you have not said it, the answer is whoever pushes hardest.”
How Shinder says the limits were changed
Shinder reports replacing the single shared customer-facing bucket with per-customer buckets sized from each customer’s trailing 30-day peak multiplied by a factor. The global bucket remained as a backstop to protect total service capacity. He also describes distinguishing interactive calls from batch work, identifying the limit responsible for a rejection, tracking throttling by customer, and displaying the worst tenant’s success rate alongside aggregate availability.
Rank #2
These are the author’s reported design choices, not universal defaults. A per-customer policy needs a defensible sizing factor and a reliable way to identify which customer made each request. A trailing peak can reflect a real usage pattern, but it can also preserve capacity for a past burst; the multiplier and measurement window are policy decisions that should be monitored and adjusted against actual capacity and customer needs.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsChoose the limit scope deliberately
“Rate limit” is not one fixed scope. Envoy’s current local rate-limit documentation, which identifies version 1.40.0-dev, describes token buckets that can apply to routes or virtual hosts. Depending on configuration, a bucket may be shared across workers in an Envoy process or allocated per downstream connection. Those scopes have different effects: a process-wide bucket pools traffic handled by that process, while connection-level allocation does not itself establish a customer-wide quota.
Rank #3
Envoy also documents descriptors that match request attributes such as caller cluster and path, allowing distinct buckets for matching combinations and a default bucket for other requests. This illustrates a way to scope limits by caller or request class; it does not establish that Shinder used Envoy or that his exact design was tested with it.
For any implementation, write down what identity the policy uses and what requests share a bucket. A limit keyed to a customer can isolate customers only if the system consistently maps requests to the right customer. If identity is missing, ambiguous, or shared across unrelated customers, the configured scope may not match the fairness policy operators intend.
Balance customer isolation with service protection and priority
Different controls solve different problems. A per-customer bucket limits one customer’s ability to consume shared capacity; a global bucket can still cap the service’s total accepted traffic. Workload classes add another policy dimension: interactive requests can receive priority over batch jobs from the same customer, but token buckets do not create that priority automatically. Operators must define which classes outrank others and how the service behaves when capacity is scarce.
Recommended Free Tools
| Control | What it helps address | What it does not decide by itself |
|---|---|---|
| Shared global bucket | Caps traffic against a service-wide threshold. | How capacity is divided among customers or request classes. |
| Per-customer bucket | Separates one customer’s allowance from another’s. | How to identify customers and size each allowance. |
| Workload-class limits or priority | Can protect interactive work from batch traffic under an explicit policy. | Which work should rank higher and what happens during contention. |
| Global backstop plus customer buckets | Combines customer isolation with a limit on total service traffic. | How to allocate capacity when the global backstop itself is reached. |
Make throttling explainable and visible
When Envoy’s local rate-limit filter rejects a request, it returns HTTP 429 by default; the response status is configurable. The filter can optionally emit a Retry-After header. Envoy documents this delay as time until the next token is available in the rejecting bucket, subject to the configured behavior. It is not a promise that an entire customer account or the service will be fully usable after that interval.
Best Value
- Organize Your Thoughts: Keep all your book reviews and stats in one place, making it easier to look back and reflect on your reading history.
- Enhance Your Reading Experience: Detailed review sections help you dive deeper into each book and appreciate its nuances.
- Stay Motivated: Reading challenges and daily trackers ensure you stay on top of your reading goals and progress.
A useful rejection response should make clear which limit was reached and provide retry guidance that matches that limit. Operators also need enough observability to see whether a limit is protecting total capacity or disproportionately affecting one customer. Envoy documents counters for requests checked, rate-limited decisions, and enforced rejections. Shinder additionally reports tracking the throttled fraction per customer and showing the worst tenant’s success rate next to aggregate availability. Aggregate availability alone can conceal a severe problem for an individual tenant.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
