Retry Logic That Makes an Outage Worse

A retry storm has a signature. It shows up in the graphs as a cliff, not a slope: error rate ticks up, clients begin re-sending, and within seconds the volume of requests hitting the failing service is several times what it was before anything broke. The service, already struggling, now spends its remaining capacity rejecting work it has already rejected. Recovery takes longer than the original fault deserved.

The uncomfortable part is that every one of those retries was written by someone trying to be careful. Retries are the first resilience pattern most teams reach for, and they come with a story that sounds like engineering maturity: networks are flaky, so try again. That story is true often enough to survive review. It is also the mechanism by which a small outage becomes a large one.

A retry is a bet that the next attempt lands somewhere better

Consider what a retry actually assumes. It assumes the failure was local and transient: a dropped packet, a momentary blip, a single overloaded node. If that assumption holds, a second attempt costs almost nothing and saves the request.

When a service is genuinely overloaded, the assumption inverts. The failure is not local, and it is not transient. Every retry adds load to a system that is failing precisely because it has too much load. The bet pays off in the easy case and compounds the loss in the hard one, which is the worst possible payoff profile for something wired into every client by default.

There is a second assumption buried underneath: that the client can tell the difference between those two situations. Most cannot. A timeout looks identical whether the server never received the request or received it, processed it, and died before replying.

Three clients retrying is a multiplier, and multipliers stack

The arithmetic is unforgiving. Suppose one service calls another and each call is retried three times. A single user action becomes three requests. Add a second hop with its own retry policy and the same action becomes nine. A third hop, which is common in any service mesh of modest depth, turns it into twenty-seven.

No one designed a twenty-seven-fold amplification. Each team configured a reasonable local policy, reviewed it in isolation, and shipped it. The multiplication happens at the seams between teams, where no one is looking, and it only becomes visible during the exact incident when visibility is worst.

This is why retry storms so often arrive in the middle of a deploy or a partial failure: the amplification was always latent, and it needed only a trigger to reveal itself. The trigger is usually small. A single replica restarting can be enough if the retries pile onto the remaining replicas faster than they can drain.

The synchronized retry is the specific killer

Independent retries spread over time are survivable. The damage comes from retries that fire together, because a struggling service needs a gap to recover and a synchronized burst removes it. Three mechanisms create that synchronization, and they tend to appear together.

The first is immediate retry. A client that retries the instant it sees a failure produces a burst timed to the failure itself, which is the moment the service is least able to absorb it.

The second is a fixed delay. A constant backoff of one second, applied by every client, means the retries simply re-synchronize on each round. The burst is spaced out but not spread out.

The third is correlated failure. When a shared dependency goes down, all clients fail at the same instant and retry on the same schedule, producing a wave that arrives together and re-forms after every interval. This is the pattern that produces the second, larger spike on a graph minutes after the first.

Retrying a request that already succeeded

There is a quieter failure mode that does not show up as extra load at all. A client sends a write, the server commits it, and the response is lost in transit. The client times out and retries. Now the write happens twice.

The duplicate charge, the double-shipped order, the two copies of the same record: these are retry bugs that surface days later as data-quality complaints, long after the code review that approved the retry has been forgotten. The fix requires idempotency keys so the server can recognize the second attempt as a repeat of the first, and most retry implementations skip that step because it is more work than wrapping a call in a loop.

The same problem applies to reads that trigger side effects, which is more common than it sounds. A retried request that increments a counter, sends a notification, or starts a job will do all of that twice, and the second copy is just as real as the first.

Retry budgets convert retries from a local decision into a shared one

The pattern that actually contains this damage is the retry budget. Instead of every client retrying every failure, a client tracks how many of its requests are already retries and caps that fraction. A common shape: retries may not exceed ten percent of recent successful requests, with a hard floor so that a client making very few calls can still retry a couple of times.

The effect is that retry volume scales with healthy traffic instead of with failure. When the service is fine, retries are cheap and plentiful. When it is failing, the budget shrinks and clients stop amplifying the outage without any human deciding to stop.

The budget also fixes the seam problem. Each hop enforces its own budget, so the twenty-seven-fold amplification cannot form: the second and third hops are bounded by the same rule as the first. The setting lives in one place per client, which is far more auditable than a retry count scattered across call sites.

Backoff needs jitter, and the jitter needs to be real

Exponential backoff without jitter is a partial fix. It spaces attempts further apart, but every client multiplies by the same factor on the same schedule, so the retries stay clumped. Full jitter, where the delay is drawn uniformly from zero to the current backoff ceiling, spreads the attempts across the whole interval and breaks the synchronization.

The catch is that jitter must be applied by the client, not assumed. A base class that computes a backoff and then adds a random number in the low milliseconds will still clump when the backoff ceiling is seconds long. The spread has to scale with the ceiling, or it is decoration.

It is also worth capping the backoff ceiling. An unbounded exponential that reaches minutes turns a brief outage into a long tail of clients waiting on timers that will fire long after the service has recovered, each one arriving to find a healthy system and a stale request.

A circuit breaker stops the retries the budget cannot

Budgets bound the volume of retries; they do not stop a client from hammering a dependency that is clearly down. That is the breaker's job. After a threshold of consecutive failures, the breaker opens and calls fail fast instead of being attempted, which removes both the original request and its retries from the failing service's load.

The half-open state matters more than the open one. A breaker that opens and then slams fully closed on the first successful probe will oscillate during a marginal recovery, producing repeated bursts that keep the service down. Allowing a small trickle of probe traffic before a full close, and backoff on the reopen, gives the system room to actually heal.

Breakers and budgets are complementary, and shipping one without the other leaves a gap. A budget with no breaker still sends retries at a failing endpoint at up to its allowance. A breaker with no budget still lets high retry counts through whenever the breaker is closed, which is exactly the early phase of an outage when the amplification does the most damage.

Timeouts and retries have to be sized against each other

A retry policy is only as good as the timeout underneath it. If a request timeout is thirty seconds and the client retries three times, a single logical request can occupy a connection and a worker for a minute and a half, and the retries arrive stacked behind the original.

The paired constraint is a deadline that travels with the request. The server can then stop working on requests whose clients have already given up, freeing capacity for the retries that are still live. Without it, the failing service burns cycles on work whose result no one will read, which is the purest form of self-inflicted load.

A service that has been retried against for ten minutes is not going to be saved by an eleventh minute of retries. The deadline is the mechanism that makes the client admit this on the server's behalf.

The retry that should have been a queue

Some work does not belong in a synchronous request at all. If a call takes long enough that a timeout is plausible, the retry is a symptom: the operation was too large for the request path. Moving it to a queue, where it can be attempted with backoff by a worker that owns the retry policy, removes the retry from the client entirely and puts it somewhere with a global view.

That change also fixes idempotency. A queue delivers at least once, so the consumer must be idempotent by design, which is a more natural place to enforce it than a scattered set of client code paths.

The remaining retries are the ones that genuinely should be retried: a short read against a service that is briefly unavailable, a connection reset, a lookup that failed for a reason the client can see. Those are worth a small number of attempts with jitter and a budget, and they are worth stopping at the first sign that the dependency is down rather than merely slow.

The default is the problem

Most of the retry storms in production trace back to a library default: three attempts, no budget, no jitter, no deadline. The defaults were chosen to make the happy path robust, and they do. They also make the unhappy path worse in a way that no individual team chose.

The fix is not to remove retries. It is to make the retry policy a deliberate, per-dependency decision with a budget, jitter, a deadline, and a breaker, reviewed by the same people who own the dependency's capacity. That is more work than a decorator, and it is the difference between a client that rides out a blip and a client that turns a blip into an incident.

The next time a graph shows a second, larger spike minutes after the first, the retry policy is the first place to look. The spike is the shape of a synchronized retry wave, and it was written months earlier by someone trying to be careful.