Back to blog

Why does your retry logic turn small failures into outages?

Thắng Đoàn
Thắng Đoàn

You added retries to be resilient, yet the outage got worse. Retries multiply demand aimed at a system that is already struggling.

Retries multiply load when it hurts most

When a call takes too long, the client holds its resources longer, as the AWS Builders Library explains. If the client then retries, the slow service receives more work, not less. Even a small percentage of failures can grow into a flood of extra requests. Amazon's guidance is explicit: avoid magnifying a small percentage of failures into a complete outage.

The herd arrives all at once

Wikipedia describes the thundering herd problem: many processes wake simultaneously, only one can use the resource, and the rest waste CPU and fail again. Naive retries create the same pattern. If a timeout fires for many clients at the same moment, they all retry together. Synchronization, not volume alone, is what kills the struggling service.

Backoff with jitter breaks the stampede

Exponential backoff means each failure multiplies the wait before the next attempt, per Wikipedia's article on the algorithm. After a failed call, a client might wait 1 second, then 2, then 4. Amazon adds jitter, meaning random variation in the delay, so clients no longer retry in lockstep. Randomness is the fix for the herd.

  • Timeout first: cap how long you wait before you retry at all.
  • Backoff second: multiply the pause after each failure.
  • Jitter third: randomize the pause to desynchronize clients.

What this does not solve

Backoff with jitter manages demand during transient failures. It does not fix a service that is down, a bug, or a capacity shortage that retries cannot wait out. If the failure is not transient, retries just delay the honest signal that something is broken. You still need other ways to detect and handle failures that will not resolve on their own.

Which of your services would receive the most simultaneous retries if one dependency slowed down today?
Do your retry policies include jitter, or are all your clients retrying on the same clock?

Pick one client that calls a failure-prone service and check whether its retry delay includes random jitter.

Sources

AWS Builders Library: Timeouts, retries, and backoff with jitter

Wikipedia: Thundering herd problem

Wikipedia: Exponential backoff

Share:

Recommended for you

Enjoyed this article?

Subscribe for new articles. No spam. Unsubscribe anytime.

By subscribing you agree to receive the newsletter. No spam, and you can unsubscribe anytime.