Thundering herd: what it is, the Slack case, and how to prevent it
I studied thundering herd: the Slack incident, why it's not just 'high traffic', and the three techniques that prevent mass synchronization: backoff with jitter, circuit breaker, and load shedding.
Recently I focused on studying thundering herd, mainly how it happened at Slack, how to prevent it, and why it’s sneakier than it looks. I decided to document what I learned here.
What is thundering herd?
A thundering herd is when many clients or processes become synchronized and “wake up” at the same time to hit the same resource.
The key point: it’s not just high traffic. It’s high traffic at the SAME moment.
A system can handle 2,000 requests per second spread over time and die on those same 2,000 requests concentrated into 50 milliseconds. The difference between the two scenarios isn’t the volume. It’s the synchronization.
That’s exactly what happened at Slack: around 2.3 million queries executed at the SAME moment. Each query alone was cheap. Together and simultaneous, they took the service down.
Where it shows up day to day
What struck me while studying is that synchronization is born from ordinary places:
- Caches expiring together: thousands of keys with the same TTL expire at the same instant and every miss hits the database at once (the famous cache stampede)
- Mass retries: a service errors for 2 seconds and every client that failed returns together after the same timeout
- Identical crons: several schedules configured “at 00:00” all fire at the same time
- Reconnection after restart/deploy: the service goes down, all clients detect it and reconnect at the same instant it comes back
That last one hits close to home. I work with pipelines triggered by Schedulers: multiple independent schedules that, if misconfigured, become a punctual herd of jobs. The queue absorbs part of the problem, but the origin of the synchronization is something I created.
How to prevent it
1. Exponential backoff with jitter
Instead of every client retrying at the same time after an error, each retry waits a growing (exponential) amount of time plus a random factor, the jitter.
const attempt = 3;
const baseMs = 1000;
const jitter = Math.random() * 500;
const delay = Math.pow(2, attempt) * baseMs + jitter;
// 1s → 2s → 4s → 8s → ... + a small random value
Backoff alone still synchronizes: if everyone failed together and uses the same sequence, they come back together on every round. What breaks the synchronization is the jitter, which spreads requests over time.
Result: fewer spikes, more stability.
2. Circuit breaker
A “fuse” between the client and the service. When the system notices a service is failing too much, it opens the circuit and temporarily stops sending requests.
The basic states:
- Closed: everything works normally, requests pass through
- Open: too many failures, calls are blocked immediately
- Half-open: tests a few requests before letting everything through again
This keeps an already-overloaded service from being pressured even harder by retries. The logic that sticks:
Failing fast is much cheaper than failing slowly.
3. Load shedding
Basically: intentionally reject part of the requests.
When the system hits a critical limit, it starts saying “no” to some users or processes instead of trying to serve everyone and falling over. Practical examples:
- Block non-priority users
- Return a fast error (503) instead of processing until it blows up
- Limit queue sizes
- Drop the heaviest requests first
It sounds counterintuitive, but the math is simple: it’s better to lose part of the traffic than to lose everything.
Bonus: don’t create the herd in the first place
The three techniques above handle the herd after it exists. But the cheapest prevention is at the origin:
- Cache TTL with jitter so keys don’t expire together
- Random offsets on crons that don’t need to run at the same minute
- Staggered restarts/deploys per batch of instances, instead of taking everything down and bringing everything up at once
If clients never synchronize, there’s no herd to dissipate.
Summary of what I learned
| Technique | What it does | When to use |
|---|---|---|
| Backoff + jitter | Spreads retries over time | Any client that retries |
| Circuit breaker | Stops hitting a sick service | Between services that call each other |
| Load shedding | Intentionally rejects traffic | At the edge of systems under pressure |
| Prevention at origin | Avoids synchronization | TTLs, crons, and deploys |
Thundering herd taught me something that goes beyond the topic: in distributed systems, the enemy is rarely volume. It’s accidental coordination, thousands of individual, rational decisions that, added together, become a herd.
If you enjoyed the topic, worth studying alongside: cache stampede, retry storm, and singleflight/request coalescing are variations of the same problem.