Retry policy mistakes that turn a blip into an outage

A correct retry policy repeats only calls that can safely be repeated, only after errors that can change, with a wait that grows and carries a random part, under a hard limit on tries and on total time, and with a shared budget so that a sick dependency is not buried under its own clients. Each piece guards against a different failure. Leave one out and a ten-second blip can become an hour-long outage, because the retries themselves are traffic. The rest of this page goes through the pieces in the order a request meets them, then the storms you get when they are missing.

Decide first whether to retry at all

Two questions come before any timer. Did the failure come from something that may clear up, and is the call safe to send twice?

The first question is about the error. A connection reset, a timeout, a 503 or a 429 are signals that waiting might help. A 400, a 401, a 403 or a 404 will answer the same way tomorrow, so repeating them only adds load. Google's SRE book states the rule bluntly: permanent errors and malformed requests are never worth a retry, because neither will ever succeed (SRE book, cascading failures). A useful default is an allow-list of retryable outcomes rather than a deny-list of the ones you remembered.

The second question is about the call. The HTTP specification calls a method idempotent when several identical requests have the same intended effect on the server as a single one, and it lists PUT, DELETE and the safe methods as idempotent (RFC 9110, section 9.2.2). POST is not on that list. A POST that creates an order, sends an email or moves money is the one that duplicates when a response is lost after the work was done. Amazon's guidance is that calls with side effects are unsafe to retry unless the API offers idempotency, and some of its APIs take an explicit token for exactly that purpose (Amazon Builders' Library). In practice: the client generates one key per logical operation, sends the same key on every attempt, and the server stores the outcome under that key and replays it. Without the key, the right policy for a write is not to retry.

Wait longer each time, and cap the wait

If a call failed because the other side is busy, trying again at once is the worst answer. Exponential backoff spaces the attempts out: the wait doubles after each failure, so the client backs away from the service quickly. Left alone, the doubling would eventually mean waiting for hours, which no caller can use, so the wait gets a ceiling. Amazon calls this capped exponential backoff and notes the downside: once every client sits at the ceiling, they all retry steadily at that rate (Amazon Builders' Library). The cap protects the caller's patience; it does not protect the server. That is the job of the next piece.

Add randomness, or the clients move as a herd

Picture a thousand clients that all failed at the same second because a deploy restarted the server. With plain backoff they all wait the same time and all return together, and the server takes a second wave just as it recovers. Randomness fixes this by spreading the returns over a window. Amazon describes jitter as randomness added to the wait to spread retries around in time, and says backing off alone often helps less than expected when failures come from overload or contention (Amazon Builders' Library). Google's guidance is the same: always use randomized exponential backoff, because retries that are not scattered across the window can ripple in step and feed on themselves (SRE book, cascading failures).

The simplest form picks the actual wait uniformly between zero and the current ceiling. A few lines show the whole idea:

function pauseBeforeAttempt(n, baseMs, ceilingMs) {
  const ceiling = Math.min(ceilingMs, baseMs * 2 ** n);
  return Math.random() * ceiling;
}

Jitter is not only for retries. Periodic jobs, cache expiry and scheduled work all benefit from a random offset, for the same reason: it keeps many machines from acting on the same tick.

Obey Retry-After when the server speaks

A server that knows when it will be ready can say so. The Retry-After header holds either a calendar timestamp or an integer number of seconds, and next to a 503 response it tells you how long this service expects to stay out of reach (RFC 9110, section 10.2.3). Rate limiters send it with 429 as well. A client that ignores the header and runs its own timer is guessing when the server has already given an answer. The safe pattern is to take the larger of your computed wait and the header value, then still add a little jitter on top, so that a thousand clients given the same header do not return in the same millisecond. Parse both forms, and treat a garbage value as absent rather than as zero.

Stop: limits on tries and on total time

Every loop needs two exits. A cap on attempts stops a single request from retrying forever; Google's text says plainly not to retry a request indefinitely (SRE book, cascading failures). A cap on total elapsed time matters just as much, since three tries with long waits can outlast the user, the upstream deadline or the process itself. Pass the remaining deadline down to each attempt, and give up when the time is gone even if tries remain. A retry that finishes after the caller has stopped listening is pure waste.

Share a budget, so the herd cannot grow

Per-request limits do not stop the whole fleet from multiplying load. A budget does. The SRE book suggests an allowance for the whole process, for example a few dozen retries a minute, and once it is spent the request simply fails (SRE book, cascading failures). Amazon reaches the same idea with a token bucket kept locally: successful calls refill it slowly, failures drain it, and when it is empty retries stop (Amazon Builders' Library). The effect is that retries cost a small, bounded share of normal traffic. When the dependency is truly down, the budget empties fast and your clients go quiet, which is what a struggling server needs.

The storms, by cause

Most retry storms come from a short list.

Retries at every layer. If a user action passes through a browser, a frontend, a backend and a database client, and each layer retries, the attempts multiply rather than add. Amazon walks through a deep stack where every layer tries three times and arrives at 243 times the normal load on the database (Amazon Builders' Library); Google's example, four attempts at each of three layers, gives 64 attempts on the database for one click (SRE book, cascading failures). Pick one layer to own retries, normally the one nearest the user that can still tell whether the call is safe to repeat, and let the others fail fast.

No jitter after a shared event. A restart, a certificate rotation or a cache flush makes many clients fail together. Without randomness they come back together too, repeatedly.

Retrying overload as if it were a glitch. A 503 from a server that is already drowning is a request to slow down. If the answer to it is a fast loop, the retries become the load that keeps the service down. The SRE book describes how retry volume can grow each second until the backend collapses, and that digging out may need the frontends to cut load drastically until the retries stop (SRE book, cascading failures).

Repeated writes. A POST without an idempotency key, retried after a lost response, creates a second charge or a second record. The failure is quiet: nothing crashes, and the damage shows up in a support ticket.

Retrying what cannot change. A missing-page error or a validation error retried five times is five wasted calls and a longer wait before the caller sees the real message.

If your own design was built with a library's defaults, it is worth reading how a model-facing fallback loop breaks in a similar way; our note on empty success responses in a Gemini fallback chain is one case where the definition of failure decided everything.

When this does not apply

This page assumes a client that talks to a remote service over a network and can pick its own timing. Several situations break that assumption. Interactive tools where a person is waiting may prefer one fast retry and then an honest error over a long, polite wait. Message queues and job runners usually have their own redelivery rules and dead-letter handling, and layering a second retry loop on top reproduces the multiplication problem above. A few protocols, such as DNS resolvers and some database drivers, have built-in retry behaviour that you should configure rather than wrap. Some servers publish their own retry guidance, and where that differs from this page, follow the server. Finally, the advice here comes from public writing by two large operators and a standards document, not from our own load tests: it describes what they report and what the specification says, and your traffic may need different numbers.

Checking your own code

If you want a second pair of eyes on a retry loop before it ships, our retry-backoff-review skill takes a single retry loop, or a short note on the way your client repeats failed calls, and lists the handful of mistakes above that cause outages or double writes: waits with no randomness, loops with no limit, repeated writes with no key, a Retry-After header nobody reads, and retries on errors that cannot change. When it finds nothing, it says that in one line. The store itself is at https://aiskills402.com.