When retries become a storm

Retries are supposed to ride out a blip — but a correlated failure makes the whole fleet fail at once, and then retry at once. The obvious fix makes it worse: every retry adds load to a service that's already drowning, and naive retry multiplies the offered load by the retry count. Backoff spaces the rounds out, yet each still arrives as one synchronised wave. Only jitterRandomising each backoff delay within its window, so requests that failed together don't retry together. Full jitter picks a uniform delay in [0, the backoff target]. breaks the herd apart.

The storm
A failure tripsrequestsThe herd: requests caught by the same outage, so they all fail — and start retrying — at roughly the same instant.at once, each retrying up totimes with ams base delay.
Total retry load: 6,000 extra attempts — 7× the original requests, however you schedule them. The only question is whether they arrive in a wall or a trickle.
Retry load over time
0.1×peak vs herd

At its worst moment, Exp + jitter hits the service with about 102 retries at once over a 6.3 s drain.

621/window
02 s4 s6 s6.3 s
Jitter is the only knob that lowers the peak: the same 6,000 retries arrive as a smooth trickle of about 102 at a time — 59× below the immediate-retry spike. Spread plus de-correlation is what tames the storm.
Policy by policy
StrategyPeak loadDrains in
Immediate
6× herd
instant
Fixed delay
1× herd
600 ms
Exponential
1× herd
6.3 s
Exp + jittershown
0.1× herd
6.3 s

Read the peak column: immediate retry stacks every round into one spike; fixed and exponential backoff cut that to a single herd per round but never below it — they only move the rounds apart in time. Jitter is the one that drops the peak, because it's the only one that stops the herd retrying in unison. In practice, pair it with a retry budget and a circuit breaker so the herd shrinks too.

jitter timings seeded from your inputs — this link reproduces the same storm