Retries: When They Save You and When They Make It Worse

Retrying a failed operation is the most common reliability pattern and one of the easiest to get catastrophically wrong. Done well, transient failures disappear before anyone notices. Done badly, your own clients finish off a service that was already struggling.

Only retry what can be retried

A timeout is worth retrying; a 400 is not, and never will be. Retrying a deterministic failure is just a slower way to fail while generating load. The first decision is always whether this class of error can succeed on a second attempt at all.

The rules that matter

• Exponential backoff with jitter — synchronised retries are a self-inflicted thundering herd.
• A hard cap on attempts, and a defined destination for what exhausts it.
• Idempotency, or your retry charges the customer twice.
• A circuit breaker, so you stop hammering something that is clearly down.

These are policy decisions, not code details — which is why they belong somewhere visible. A retry branch with "+72 h, max 3" written on it is auditable in a way that a constant buried in a service is not.