Timeouts, retries, and the failure you made worse

Your service is only as available as the things it calls

A handler that calls a payment provider, a mail API and a model endpoint has four ways to fail, not one. If each dependency is up 99.9% of the time and you need all of them, your ceiling is about 99.6% before you have written a single bug.

Most of the difference between a service that degrades and one that collapses is three settings and one decision. The settings are timeouts, retries and backoff. The decision is what to do when the answer never comes.

Every outbound call needs a timeout, and most do not have one

The defaults are worse than people assume. Python’s requests waits forever unless you pass timeout=. Node’s fetch has no default timeout. Many database drivers will wait indefinitely for a connection. “Forever” is not a hypothetical: a network can drop packets in a way that leaves the socket open and silent, and your worker sits there until something else kills it.

What that does to a live site is specific and worth picturing. Each stuck request holds a worker. Requests keep arriving. Within a minute every worker is waiting on the same dead dependency, and your site is down, including all the pages that never needed that dependency at all.

Set a timeout on every outbound call, and make it smaller than the time your own caller is willing to wait. If your page must answer in 3 seconds, a dependency cannot have a 10-second timeout.

resp = session.post(url, json=payload, timeout=(2, 5))  # connect, read

Then decide, per call, whether a timeout means failure or a fallback. A recommendation service that times out should return the popular items. A payment that times out must not be treated as success.

Retries help, and then they attack you

Retrying a failed call is usually right: most failures are transient. Three rules keep it from becoming the outage:

1. Only retry safe operations. GETs, yes. A POST that charges a card, only with an idempotency key, so the second attempt returns the first result rather than charging again.
2. Back off exponentially, with jitter. Fixed-interval retries from a thousand clients synchronise into waves. Random jitter spreads them. The formula is min(cap, base * 2**attempt) * random().
3. Cap the total. Three attempts, then fail. A retry budget (refuse to let retries exceed a small percentage of total requests) is the version that survives contact with real traffic.

The failure mode to understand is the retry storm. A dependency slows down, every caller retries, the dependency now receives three times its normal load while already struggling, and it never recovers. The retries did that, not the original fault. This is why a struggling system sometimes only recovers after clients are stopped entirely.

Retries multiply through layers, too. Three retries at the browser, three in your gateway and three in your service is 27 requests for one user action. Retry at one layer, deliberately chosen: usually the innermost one that knows whether the call is safe.

The circuit breaker

When a dependency is clearly broken, calling it again is a way of spending your own capacity to learn something you already know. A circuit breaker counts recent failures and, past a threshold, stops calling for a while, failing instantly instead.

Three states, and the third is the one people miss: closed (normal), open (failing fast, no calls made), and half-open: after a cooling period, let a single request through. If it succeeds, close; if not, stay open. Without the half-open state you have an on-off switch that never turns itself back on.

A breaker turns a 30-second hang for every user into an immediate, cheap failure that frees your workers to serve everything else. Libraries exist in every language; a counter, a timestamp and twenty lines also work.

Decide in advance what degraded looks like

For each dependency, write down the answer to “what does the user see when this is down”:

• Search unavailable → show the browse page, with a line saying search is temporarily down.
• Recommendations unavailable → show the most recent items.
• Mail provider unavailable → queue the message, tell the user it will be sent, retry in the background.
• Payment unavailable → say so plainly, and do not create an order in an ambiguous state.

A feature flag per dependency makes this operational rather than theoretical: during an incident you turn off the broken feature and the rest of the site works. That is often the fastest available fix, faster even than a rollback.

The two-line summary

Timeouts stop one slow dependency from consuming your service. Backoff, jitter and breakers stop your recovery attempts from becoming the second outage. Both are configuration you write on a calm afternoon, and neither can be added while everything is on fire.