Failures and antipatterns · Intermediate

Split brain

Investigate competing writers by separating quorum decisions, leadership terms and fencing of external resources.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Distinguish competing leaders from conflicting commits
  2. Reason about a five-node partition
  3. Check leases and external-write fencing
  4. Avoid creating new histories during recovery
  5. Reconcile authority and uncertain outcomes
  6. Ask Logeia about this articleCheck your understanding

Distinguish competing leaders from conflicting commits

Split brain describes a failure in which different sides treat themselves as legitimate writers and accept conflicting changes. Distinguish two processes claiming leadership from two incompatible histories actually being committed. An old leader can remain unaware of a newer term while a correct consensus protocol prevents it from committing new work. Begin with the authority behind each successful operation: membership, term and the participating votes. Network silence does not prove that another machine has stopped. The diagnostic question concerns which effects were accepted under which authority, rather than how many dashboards displayed the word leader at one moment.

Reason about a five-node partition

Consider a hypothetical fixed five-member majority cluster partitioned into groups of three and two. With a majority of three, the larger side may continue under the protocol while the smaller side cannot commit new changes. Letting the smaller side independently redefine two votes as sufficient breaks the original safety assumptions. This example simplifies fixed membership and is not a complete dynamic-reconfiguration procedure or a measured outage. Membership changes themselves can require consensus. Adding a member can change the necessary quorum, so an improvised reconfiguration during failure can destroy a state that was still recoverable. Follow the product’s supported membership and recovery protocol instead of reasoning from machine counts alone.

Check leases and external-write fencing

Selecting a new owner after a presumed lease expiry does not necessarily stop a paused former owner from later writing an external resource. Fencing makes the resource reject obsolete authority, for example by checking a monotonically advancing generation token. Merely logging a token does not enforce rejection. Clock-based leases depend on timing assumptions and serve a different role from durable terms or resource-side checks. Separate the consensus protocol’s internal history from external effects such as file updates or notifications. A correctly elected leader does not automatically give every external service knowledge of the latest term. Identify the actual boundary that prevents the old owner from performing a conflicting effect.

Avoid creating new histories during recovery

Preserve node and cluster identities, membership, terms or generations, successful operation identifiers, destinations and the partition interval. Before and after restoring connectivity, verify which history is authoritative. Force-initializing both sides independently and exposing them under the same service name can create two histories rather than repair one. Use the specific product’s backup and quorum-loss procedure and distinguish committed changes from attempted but unconfirmed work. Restoring the network, electing leadership and stopping stale external writers are separate tasks. Recovery should re-establish a consistent authority relationship, not merely remove a health warning while leaving both paths able to affect the same business resource.

Reconcile authority and uncertain outcomes

After recovery, reconcile acknowledged operations by business identifier and investigate missing, duplicate or conflicting effects. A blind latest-timestamp overwrite can lose work because of clock differences or distinct operations. The convergence policy depends on data semantics and the protocol. In an isolated test environment, exercise partitions, delayed former owners, membership changes and external effects separately. A minority side refusing writes can be an intentional safety property despite reduced availability. Do not define success as both sides always accepting writes. State the required consistency and business recovery policy first, then test that the system preserves those guarantees under the failures it claims to tolerate.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 17

Choose a learning path