Study guides · Intermediate

Partial Failures, Clocks, Timeouts, and Ordering

Reason about missing replies, partial failures, clocks and retries through an ambiguous payment result.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. A missing reply has several explanations
  2. A timeout is a decision deadline
  3. Separate wall time from causal order
  4. Make a retry identify the same operation
  5. Use a lost order response as the shared example
  6. Correct retries still consume finite capacity
  7. Place isolation and admission on the failure trace
  8. Follow the readings and solve a changed-identity case
  9. Ask Logeia about this articleCheck your understanding

A missing reply has several explanations

A distributed system can suffer partial failure: one process or communication path fails while other components continue. If a payment request receives no reply, the server may have stopped, the request may never have arrived, or the operation may have committed before its reply was lost. A client-side timeout cannot distinguish these cases. Unlike an idealized local function call, the caller does not observe every relevant state at once. Represent an unknown result explicitly instead of turning every missing response into proof that no side effect occurred.

A timeout is a decision deadline

A timeout establishes when a component stops waiting and chooses another action. It may mistake a slow or paused peer for a failed one. Shorter timeouts improve reaction time but can cause unnecessary retries and extra work. Independent per-hop timeouts can also exceed the caller’s overall budget when requests pass through several services. Propagate deadlines where appropriate and combine them with bounded retries and backoff. Check whether work continues after the caller stops waiting; cancellation and timeout are related interfaces, but one does not automatically guarantee the other has completed.

Separate wall time from causal order

Clocks on different machines can disagree and be adjusted. A smaller wall-clock timestamp is therefore not sufficient proof that one event causally preceded another. Causal order can be built from local program order, message-send-before-receive relationships and transitivity. Lamport clocks preserve that ordering implication, but the converse is false: ordered clock values do not prove a causal path exists. Designs that depend on real-time bounds need additional assumptions about clock uncertainty, pauses and communication. Use logical order for the question it answers rather than treating it as a replacement for every physical-time requirement.

Make a retry identify the same operation

A payment retry can duplicate an operation unless retries identify the same logical action. Assign an operation ID and store the deduplication decision consistently with the result so a repeated request can return the earlier outcome. Specify how long IDs remain valid, what happens when an ID is reused with different parameters and how recovery preserves the record. A status endpoint also needs a defined state machine to distinguish pending, completed and unknown work. Retry safety depends on these durable application rules, not on the network promise that messages will never be duplicated.

Use a lost order response as the shared example

Take order-7, decrement inventory from ten to nine and persist the result, then lose only the response. The caller observes silence without knowing whether inventory is nine or ten. A fresh-identity retry can decrement it again. Stable operation meaning, payload matching and atomic local recording connect a retry to the earlier result.

The table separates histories that look similar to the caller. Missing arrival, stopping before application and losing a post-application response are different server states. A result query also needs an explicit committed-state contract. This chapter uses modeled orders rather than real payments. The later local-process experiment controls application and response boundaries with temporary storage, providing concrete evidence for one history without claiming to reproduce every network failure.

Stopped boundaryEffectCaller observation
Before arrivalNoneNo response
Before applicationNot yet appliedNo response
After application, before replyInventory nineNo response

Correct retries still consume finite capacity

Deduplicated effects do not make attempts free: authentication, connections, reads and locks still cost resources. Three layers with three attempts each can produce 27 downstream attempts. Backoff spaces work, jitter spreads it and retry budgets limit extra count. Idempotency does not authorize unlimited retries.

A request also has one end-to-end waiting allowance. Fresh 300 ms timeouts at every stage can exceed it. Pass remaining time and specify where cancellation is actually checked. Already committed inventory is not restored merely because waiting stops. Observe why another attempt was refused: expired time, exhausted budget and a final input error mean different things. Preserve useful completion as well as response latency when assessing these controls.

Place isolation and admission on the failure trace

A slow notification destination can occupy every shared connection and block healthy inventory. Bulkheads isolate reservations, bounded admission rejects excess work and circuit breakers restrict calls or probes after relevant failures. Splitting only downstream pools still leaves parent-worker exhaustion possible. Trace the actual resource held at every stopped boundary.

Suppressed required notifications are unfinished work, not successful delivery. Retain send intent where needed. Fallback staleness and omitted optional output belong to the business contract. Place mechanisms at admission, application, response and cleanup boundaries rather than listing pattern names as a complete solution. This identifies which observed failure each one changes and which uncertainty remains.

A fast acknowledgment can leave notifications, compensation or abandoned computation unfinished. After order-7's inventory reaches nine, specify who resumes its unsent notification and how normal admission returns after dependency recovery. Evaluate completion of accepted work as well as refusal counts.

A successful local three-service experiment does not synchronize clocks on different hosts. Its shared monotonic domain differs from cross-host deadline encoding. Logical event order is another quantity again. When a premise changes, identify which conclusion remains supported rather than carrying every earlier result into the enlarged setting. That habit connects the specialized patterns through one failure history without attributing guarantees to mechanisms that did not provide them.

Follow the readings and solve a changed-identity case

Start with the lost-response lab and verify inventory nine plus one recorded identity. Continue to retry budgets and deadlines, then compare budgets and effects in the three-service experiment. Logical clocks address a different question: causal order rather than elapsed waiting. Counts, duration and causality remain different quantities even in one order example.

The exercise proposes a new operation ID after the lost response. That can create another order effect, so recover the original identity's outcome instead. Conflicting content needs rejection rather than unconditional old success. Being able to explain the injected boundary and retained state is the intended foundation for the more specialized readings.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 16

Choose a learning path