Design patterns and optimization · Intermediate

Trace circuit-breaker suppression and recovery

Trace closed, open and half-open states while separating failure windows, bounded probes, local refusals and fallback semantics.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Stop repeated calls to an unhealthy dependency
  2. Trace a threshold and a recovery window
  3. Bound probes and define state scope
  4. Preserve the meaning of fallbacks and refusals
  5. Verify the state machine deterministically
  6. Define what a successful probe actually checks
  7. Solve the refusal-count exercise
  8. Ask Logeia about this articleCheck your understanding

Stop repeated calls to an unhealthy dependency

A persistently unresponsive notification dependency can consume caller resources on every order. A circuit breaker temporarily suppresses calls based on observed outcomes and uses limited probes to assess recovery. Read timeouts and bulkheads first. It is a caller-side control, not a direct oracle of remote health: network failure and dependency failure can produce indistinguishable observations.

Use closed for normal calls, open for immediate refusal and half-open for restricted probes. Write their behavior explicitly rather than relying on the everyday meaning of open and closed. A local open-state refusal differs from an attempted downstream failure. Feeding refusals back into the same failure denominator can keep a breaker open because of its own decisions. Define which actual attempts, errors and latency outcomes contribute before choosing thresholds.

Trace a threshold and a recovery window

For a teaching policy, open after at least six relevant failures among the last ten actual calls. After five seconds, permit one half-open probe. Ten calls with six failures trigger suppression of subsequent requests. Five elapsed seconds do not prove recovery; only a probe satisfying the declared success conditions can justify closing. A failed probe reopens the breaker. These thresholds are illustrative, not empirically selected production settings.

Specify count versus time windows, minimum observations and treatment of slow successes. Invalid input and authorization refusal should not automatically count as dependency malfunction. Classify connection errors, timeouts and overload using the real API contract. Calls already in flight may continue after the transition, so open does not imply instantaneous zero downstream activity. The trace needs both local state changes and the operations already admitted under the previous state.

StateNew request behaviorExample transition
ClosedNormal callRelevant failures reach threshold
OpenRefuse without sendingWait interval permits a probe
Half-openLimited probe callsOutcome selects closed or open

Bound probes and define state scope

Letting every request become a probe after five seconds creates a recovery burst. Atomically limit half-open probe admission, decide what other callers receive and bound probe duration. Across many callers, one local probe each can still be many remote calls. Jittered waits or a gradual ramp may help, but must be checked against aggregate dependency capacity.

Process-local state can reset on restart or scale-out. Globally shared state is not automatically superior because it adds another dependency. Decide whose observations are shared and at what scope. One tenant's failures may not justify blocking every tenant, and one endpoint's failure may not describe an entire service. Breaker keys and bulkhead units need not match. Record both boundaries so a shared state or reservation does not unintentionally broaden the impact of a narrow failure.

Preserve the meaning of fallbacks and refusals

If orders continue while notifications are suppressed, retain required notification work through a recoverable contract such as an outbox. Treating a skipped notification as completed changes requirements. A cached read fallback needs an explicit freshness allowance and cannot replace an inventory decision requiring current state. A fallback that calls another shared dependency may merely move the overload, so inspect that path too.

Immediate client retries can keep load cycling outside the breaker. Combine its refusal semantics with budgets, deadlines and backoff. A business operation applied before opening remains applied; the breaker does not undo unknown results. Stable-ID recovery is still needed. Observe fast refusal, bounded occupancy and correct business outcomes separately, and state which objective the mechanism actually improves. One favorable response-time metric cannot stand in for all three.

Verify the state machine deterministically

Control the clock and call outcomes to test just before and at thresholds, before and after the wait, concurrent probe admission, probe success and probe failure. Advancing a model clock is more reproducible for state transitions than a long sleep. Integration with a real service is a separate test; a passing state model does not recreate every network failure. Assert local refusal count and actual sends independently.

Observe state, transition reason, attempted-call failures and latency, local refusals, probes and occupancy separately. Sparse traffic can delay recovery confirmation, so open duration is not identical to remote outage duration. A dependency reliably handled by deadlines and limited retries may not need the added state complexity. Compare erroneous suppression costs as well as benefits. Verify restoration after dependency recovery and reduced load, not merely the initial ability to open.

Define what a successful probe actually checks

A successful HTTP probe that bypasses the database may not establish recovery of normal database-dependent work. Conversely, a recovery test should not create real payments merely to exercise the route. Specify what a side-effect-free probe verifies and how normal traffic resumes gradually under its result. Save probe scope separately from the full operation's scope. This prevents a healthy lightweight endpoint from being mistaken for evidence about every downstream dependency. The half-open state supplies a place for controlled evidence; it does not determine the correct probe contract automatically. Classify its success using the service behavior the breaker is intended to protect.

Solve the refusal-count exercise

While open, one hundred requests are refused locally. Should they become one hundred new downstream failures in the observation window? Not under this design: they were never sent and provide no new dependency outcome. Count them as refusals separately from the recovery probe. Otherwise the breaker can keep filling its failure window with consequences of its own suppression.

If ten processes each permit one half-open probe, the dependency can receive ten probes. A local limit of one is not a global limit of one. Compare the needed aggregate control with the cost of shared coordination. Understanding the pattern means tracing sends, classified outcomes and occupied slots, not only memorizing three state names. Preserve that trace when combining a breaker with other admission and retry policies.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 16

Choose a learning path