TL;DR: Exponential backoff with jitter spreads client retries over time so a recovering service doesn't get flattened by a synchronised wave. We added full jitter to our CI agents and a retry budget, and our internal 503 rate during recovery dropped from minutes of saturation to seconds.

A few months back at Buildkite a metadata service we run on ECS had a 40 second blip, and the recovery took six minutes instead of forty seconds. The service came back healthy, then immediately fell over again, then recovered, then fell over. Our build agents had all retried at the same fixed interval, so every two seconds a few thousand requests arrived in lockstep. That synchronised wave is a retry storm, and exponential backoff with jitter is the standard fix.

I'd read about this for years and never actually watched it bite us. Once it did, the maths got a lot more interesting. Here's what we changed and why it worked.

What is a retry storm

A retry storm happens when many clients fail at the same moment, then retry on the same schedule, producing a repeating spike of traffic that prevents the downstream service from recovering. Because the retries are correlated in time, each recovery attempt is immediately overwhelmed, so a short outage stretches into a long one. Jitter breaks the correlation by randomising when each client retries.

The key word is correlated. One client retrying every two seconds is nothing. Three thousand clients that all failed at 12:04:01 and all retry at exactly +2s, +4s, +8s is a denial-of-service attack you wrote yourself. The downstream never sees the gaps between the spikes, so its CPU and connection pool stay pinned.

Why fixed-interval retries make it worse

Our old agent retry config looked sensible on paper. Retry up to 5 times, wait 2 seconds between attempts, give up after that. The problem is the 2 seconds is identical for every agent, so the herd stays in formation through the entire recovery window.

Plain exponential backoff helps a bit. If you double the wait each time (2s, 4s, 8s, 16s) the spikes spread out as attempts progress. But the first retry after a shared failure still lands at the same instant for everyone, and that first wave is usually the one that re-saturates a fragile service. The fix that actually matters is adding randomness on top of the exponential growth.

Marc Brooker's AWS post on the topic ran the simulation properly years ago, and the result is blunt: full jitter both reduces the number of calls and finishes the work faster than no jitter or partial jitter. We just hadn't applied it to our own internal clients.

Full jitter versus equal jitter

There are a few flavours. The two we considered were equal jitter and full jitter.

import random

BASE = 0.2      # seconds
CAP = 20.0      # max backoff

def equal_jitter(attempt):
    backoff = min(CAP, BASE * (2 ** attempt))
    return backoff / 2 + random.uniform(0, backoff / 2)

def full_jitter(attempt):
    backoff = min(CAP, BASE * (2 ** attempt))
    return random.uniform(0, backoff)

Equal jitter keeps half the backoff fixed and randomises the other half, so retries are spread but still loosely clustered. Full jitter picks any value between zero and the current cap, which spreads the herd the widest. We went with full jitter for the metadata client because the downstream cared far more about peak concurrency than about any single request finishing quickly.

One catch worth calling out: with full jitter a retry can fire almost immediately after a failure, since the lower bound is zero. That's fine when your concern is the aggregate wave. If you also need a guaranteed minimum gap, clamp the lower bound to a small floor like BASE.

Backoff alone isn't enough

Jitter spreads the retries, but it doesn't cap the total work a struggling service has to absorb. For that we added two more controls.

First, a retry budget. Each agent tracks the ratio of retries to fresh requests over a sliding 10 second window, and once retries exceed 20 percent of traffic it stops retrying and fails fast. The Google SRE book section on addressing cascading failures describes this pattern, and it's the single change that put a hard ceiling on storm amplitude.

Second, a circuit breaker in front of the client. After a threshold of consecutive failures it opens for a few seconds and returns errors locally without touching the network. That gives the downstream a guaranteed quiet window to recover, instead of a steady trickle of probes from thousands of agents.

Trade-offs and limitations

Jitter makes individual request latency less predictable, which annoys anyone trying to set a tight p99 SLO on a single call. You're trading per-request determinism for system-wide stability, and that's usually the right trade for internal infrastructure, but not always for a user-facing path with a strict deadline.

Retry budgets need per-client state, so a brand new agent with no history can still pile on during the first window. Circuit breakers add a threshold and a timeout you have to tune, and a breaker set too aggressively will trip on normal noise and hurt availability. We game-day these settings now rather than trusting the defaults, because a control you've never tested under failure is just a guess.

Wrapping up

Exponential backoff with jitter is cheap to add and it solved a problem that fixed-interval retries had quietly created for us. If I were starting over I'd reach for full jitter by default, pair it with a retry budget, and only add a circuit breaker once I'd measured a real cascading failure. Then I'd run a game day to break it on purpose, because the only retry logic I trust is the kind I've watched fail safely.

Further reading

• Exponential Backoff And Jitter, AWS Architecture Blog
• Addressing Cascading Failures, Google SRE Book
• Handling Overload, Google SRE Book
• Retry guidance and the retry budget concept in gRPC