Retry storms

Why does every client politely trying again turn a brief hiccup into an outage that will not end?

Retrying a failed request is the most obviously correct thing a client can do. Networks drop packets; servers restart; a request that failed a moment ago will probably succeed now. Every sensible client library does it, and each one is behaving well.

So here is the puzzle worth stopping at. A service wobbles for three seconds. Every client does the polite, correct thing. Twenty minutes later the service is still down, and nothing is wrong with it except the traffic. The original fault is long gone and cannot be found in any log. What kept it down was the recovery behaviour of well-meaning clients — which means the outage was, in a real sense, caused by the fix.

Work the arithmetic, because this is one of those cases where the intuition only arrives after the multiplication.

A service handles a thousand requests a second comfortably. Something makes it briefly slow — a garbage collection pause, a cold cache, a failing-over database. Requests start timing out. Each client, on timeout, retries once. What is the offered load now? Two thousand a second. If the retry policy allows three attempts, four thousand.

Now notice the direction of the feedback. That extra load makes the service slower still, so more requests time out, so a larger share of them are retried. The multiplier is not fixed at three — it climbs as the failure rate climbs, and each round feeds the next. That is the signature of exponential growth, and it is why a wobble does not decay but amplifies.

Then ask what a timeout actually means to the server, because this is the part people miss. A client that gives up at two seconds and retries has not cancelled anything. The server is usually still working on the abandoned request — holding a thread, a connection, a database lock — while also accepting the retry. So the useful work per unit of effort collapses: the service is at full capacity, and much of what it completes is answers nobody is waiting for any more. Engineers call the useful fraction goodput, and the whole phenomenon is a modern echo of the congestion collapse Nagle described in the ARPANET literature in 1984.

Now add layers, since almost nothing runs alone. If a browser retries a gateway, the gateway retries a service, and that service retries a database, the multipliers do not add — they multiply. Three attempts at each of three tiers is up to twenty-seven database requests for one user's click. This is retry amplification, and it is why the tier furthest from the user, usually the one least able to shed load, feels the storm most violently.

And here is the property that makes this genuinely dangerous rather than merely annoying: the system can settle into a state that sustains itself after the trigger has gone. Bronson and colleagues named this class of behaviour metastable failure in 2021: the retry load is now large enough to keep the service failing, and the failures are what generate the retry load. Removing the original cause changes nothing, because the original cause is no longer part of the loop. That is the answer to "why will it not end."

So what breaks the loop? Three things, and it is worth seeing that they attack different terms.

Backoff with jitter attacks the rate. Waiting exponentially longer between attempts thins the traffic, but plain exponential backoff has a trap: every client that failed at the same instant retries at the same instant, and you get synchronised waves. Randomising the wait — full jitter, in the framing of the well-known AWS analysis — spreads them out. The randomness is not a polish on the idea, it is most of the benefit.

A retry budget attacks the multiplier. Instead of each request deciding independently, the client tracks retries as a proportion of overall traffic and stops retrying above a ceiling — the Google SRE book uses about ten percent. Under a broad outage, retries are futile anyway, so the budget concedes quickly rather than doubling the load.

Not retrying at all attacks the decision. A request that failed because the input was invalid will fail identically forever. Retry only errors that are plausibly transient, and never a non-idempotent operation that has no safe replay — which is why idempotency and retry policy are really one design problem.

A few corrections worth making explicitly.

The first is that this is not a capacity problem in disguise. Doubling the servers raises the load at which the storm starts, but once inside the loop a larger service simply fails at a larger number. Capacity moves the threshold; only breaking the feedback removes the behaviour.

The second is that timeouts are part of the mechanism, not a defence against it: a short timeout abandons requests the server is still processing, turning capacity into waste faster than a long one does.

The third is that retry policy is a commons problem: under a correlated failure the expected value of another attempt is low and its cost to the shared resource is real, so the polite individual strategy is the destructive collective one.

Finally, the recovery is asymmetric. A service in a metastable state usually cannot recover by being left alone, because the load never falls. Operators have to shed load deliberately — refuse traffic at the edge, drain a queue, or restart clients — and only then let it back in gradually. Being surprised by that at three in the morning is what makes this concept worth learning in advance.

• Circuit breakers, which stop a client attempting a call it can predict will fail.
• Load shedding at the edge, and why refusing work quickly protects what you accept.

Every term the collection defines is gathered in the glossary.