Failures and antipatterns · Intermediate

Cascading failures

Trace how one slowdown spreads through retries, queues and shifted traffic, then protect useful work with admission and deadline policies.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Local trouble can exhaust other resources
  2. Multiply attempts across retrying layers
  3. Separate the trigger from amplifiers
  4. Reduce admitted work enough to recover
  5. Restore traffic gradually and verify useful completion
  6. Ask Logeia about this articleCheck your understanding

Local trouble can exhaust other resources

A cascading failure spreads an initial slowdown or fault through dependencies, retries, queues and shifted traffic. A slow database can occupy application connections and workers, filling an upstream queue and causing unrelated requests to miss deadlines. Those components need not have suffered independent hardware failures. Resource retention while waiting is itself an amplifier. Record the trigger separately from the mechanisms that enlarged its effect. The order in which dashboards became red is not a complete causal model. Trace which requests held which resources, for how long, and why those resources were unavailable for other useful work.

Multiply attempts across retrying layers

Assume a hypothetical three-layer request path in which each layer makes at most three attempts and invokes the lower layer on every attempt. One original request can produce up to twenty-seven attempts at the bottom. If a setting means three retries in addition to the first attempt, the factor is four instead, so terminology matters. These numbers illustrate multiplication rather than report a measured outage. Locally reasonable retry behavior at several layers can collectively overload the dependency. Backoff without jitter can also reconcentrate attempts at the same future instant. Review settings as an end-to-end request budget, including the initial attempt, instead of judging each client library’s retry count independently.

Separate the trigger from amplifiers

Align arrivals, original work, retry attempts, active operations, queued operations, expired deadlines and per-dependency latency. Distinguish errors that triggered the event from errors caused by overload afterward. Removing failed health-check targets can push more load onto the remaining nodes; clearing caches can increase source demand. If the client has already abandoned a request but the server continues expensive work, scarce resources produce no useful result for that client. Measure successful useful completions separately from attempted work. This helps determine whether apparent activity represents recovery or an increasingly busy system spending more effort on requests that can no longer succeed within their required window.

Reduce admitted work enough to recover

Limit per-request retries and the aggregate retry budget, and retry only at selected layers. Candidate controls include exponential backoff with jitter, bounded queues, admission control, workload isolation and propagated deadlines. Check expiry before beginning another expensive stage rather than processing work whose usefulness has already ended. Admission rejection helps little if clients immediately repeat indefinitely. Distinguish permanent invalid requests from transient overload instead of assigning every error the same retry policy. A circuit breaker also needs a reopening policy that avoids every client testing recovery simultaneously. Each mechanism must reduce actual offered work or retained resources at the constrained boundary, not merely move the queue somewhere less visible.

Restore traffic gradually and verify useful completion

During recovery, restore traffic gradually rather than releasing all clients as soon as the initiating cause is repaired. Include queued work and cache refills in the load estimate. Verify that completion capacity exceeds admission long enough to drain old work and that the retry fraction falls. Restarting every node together can discard warm state and create another synchronized surge. In an isolated test environment, introduce dependency delay, lost replies and partial failure to check where propagation stops. A lower mean CPU percentage alone is not recovery. Use successful requests, deadlines, queue age and important business throughput as completion evidence, preserving the distinction between reduced activity and restored service.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 16

Choose a learning path