Failures and antipatterns · Intermediate

Retry storms: recovery traffic that expands a failure

Count retry amplification across layers and budget attempts against deadlines and backend capacity.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Recovery adds load
  2. Count a three-layer example
  3. Use one retry budget
  4. Retries need semantic safety
  5. Ask Logeia about this articleCheck your understanding

Recovery adds load

Retries help recover from transient failures but also add requests to a struggling dependency. If the dependency is already slow because it lacks capacity, immediate retries can consume the spare resources needed for recovery. Recovery traffic then becomes part of the outage.

Count attempts across the client, API and internal-service layers. A small local retry setting can amplify through several independently retrying layers. The useful unit is the downstream work generated by one user operation, not merely the number configured in any one component.

Count a three-layer example

Suppose three layers each permit three total attempts, including the initial call. In a simple nested failure case where every layer uses all attempts, the deepest dependency can receive 3 × 3 × 3 = 27 calls. Reading the number three at one layer seriously understates the possible work.

Actual behavior depends on call structure and cancellation propagation. First distinguish total attempts from retries after the first attempt. Use request identifiers to correlate downstream calls and locate the layer where repeated work multiplies. The arithmetic is an illustrative bound for the stated model, not a measurement of every deployment.

Use one retry budget

Exponential backoff with jitter reduces attempt density and synchronization. Bound both attempts and the whole operation’s deadline. Per-attempt timeouts alone can still allow the combined operation to exceed the user’s latency budget, so consider remaining time before starting another attempt.

Not every error benefits from a retry. Invalid input or missing authorization usually requires a different response rather than repetition. Classify failures, and consider concentrating retry responsibility at one appropriate layer to avoid independently multiplying attempts throughout the stack.

Retries need semantic safety

A timeout does not prove that the remote operation was never applied. A server can commit an update and lose its reply, making a retry duplicate the effect. State-changing operations need suitable semantics, such as a deduplicated idempotency key, before automatic retry is safe.

Test recovery after the dependency returns, including the ratio of retries to original requests and the workload seen by the deepest service. A higher success rate alone can hide excessive amplification or duplicated business operations. Verify capacity protection and correctness together.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 16

Choose a learning path