Why does your retry logic turn small failures into outages?
You added retries to be resilient, yet the outage got worse. Retries multiply demand aimed at a system that is already struggling.
Retries multiply load when it hurts most
When a call takes too long, the client holds its resources longer, as the AWS Builders Library explains. If the client then retries, the slow service receives more work, not less. Even a small percentage of failures can grow into a flood of extra requests. Amazon's guidance is explicit: avoid magnifying a small percentage of failures into a complete outage.
The herd arrives all at once
Wikipedia describes the thundering herd problem: many processes wake simultaneously, only one can use the resource, and the rest waste CPU and fail again. Naive retries create the same pattern. If a timeout fires for many clients at the same moment, they all retry together. Synchronization, not volume alone, is what kills the struggling service.
Backoff with jitter breaks the stampede
Exponential backoff means each failure multiplies the wait before the next attempt, per Wikipedia's article on the algorithm. After a failed call, a client might wait 1 second, then 2, then 4. Amazon adds jitter, meaning random variation in the delay, so clients no longer retry in lockstep. Randomness is the fix for the herd.
- Timeout first: cap how long you wait before you retry at all.
- Backoff second: multiply the pause after each failure.
- Jitter third: randomize the pause to desynchronize clients.
What this does not solve
Backoff with jitter manages demand during transient failures. It does not fix a service that is down, a bug, or a capacity shortage that retries cannot wait out. If the failure is not transient, retries just delay the honest signal that something is broken. You still need other ways to detect and handle failures that will not resolve on their own.
Which of your services would receive the most simultaneous retries if one dependency slowed down today?
Do your retry policies include jitter, or are all your clients retrying on the same clock?
Pick one client that calls a failure-prone service and check whether its retry delay includes random jitter.
Sources
AWS Builders Library: Timeouts, retries, and backoff with jitter
Recommended for you
- Automation
Why do your webhook automations double-fire?
Webhooks arrive as POST requests, and POST is not idempotent. If your handler does not deduplicate by event ID, a repeated delivery becomes a repeated action.
- Automation
Why do your retries keep doing the same job twice?
Networks fail, so your automation retries. Without idempotency, every retry risks a duplicate charge, ticket, or order, and the HTTP spec will not save you.
- Automation
What breaks when you automate everything in n8n?
Visual workflow tools promise speed, but every node you drag instead of code you write accrues a debt your team pays later.
- Automation
Why do your unattended automations break at 2 AM?
Workflow automations fail overnight not because logic is wrong, but because nobody designed them to tolerate failure when no one is watching.
Enjoyed this article?
Subscribe for new articles. No spam. Unsubscribe anytime.