Design patterns and optimization · Intermediate

Isolate resource occupancy with bulkheads

Compare a shared pool with reserved dependency pools, tracing stalled work, idle capacity, parent waits and release paths.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Protect one dependency from another stalled dependency
  2. Compare a shared pool with reserved pools
  3. Specify acquisition order and release on failures
  4. Choose the partitioning unit from the failure you contain
  5. Verify containment while the dependency is stopped
  6. Relate reservations to actual resource consumption
  7. Solve the idle-reservation exercise
  8. Ask Logeia about this articleCheck your understanding

Protect one dependency from another stalled dependency

Suppose inventory queries and notifications share ten outbound slots. If a notification destination stops replying, those calls can hold every slot and prevent healthy inventory queries from starting. A bulkhead partitions a resource so failure in one group has a bounded effect on another. Read concurrency limits, queues and partial failure first. Separate functions do not imply separate pools; identify the actual resource being isolated.

Reserve six slots for inventory and four for notifications. Four stalled notification calls cannot take the six inventory slots. That protects this pool boundary, not every resource in the system. CPU, database access, networking and parent workers may remain shared. Record isolated and shared resources together. A named bulkhead is not evidence that all failures are independent; its guarantee is only as broad as the boundary implemented and verified.

Compare a shared pool with reserved pools

With one shared pool, ten notifications arriving first can occupy every slot and delay later inventory work. With six-plus-four partitioning, notification five waits in its own bounded queue or is refused; it does not enter the inventory reservation. Bound queues as well as active slots, because unlimited waiting notifications can still exhaust memory. Trace slot acquisition at admission or execution and release on every exit path.

The tradeoff appears when there are no notifications and ten inventory requests. Strict reservation admits only six inventory calls even though four slots are idle. Isolation can reduce normal utilization. Lending idle capacity is possible, but needs a policy for reclaiming it and protecting reservations when the other workload returns. Test failure histories to ensure lending has not recreated the original fully shared pool under precisely the conditions the bulkhead was intended to contain.

Arrival historyShared ten slotsInventory six, notification four
Ten notifications, then inventoryInventory can wait for every slotSix inventory slots remain
Ten inventory requests onlyUp to ten activeUp to six active
Notifications stallCan retain shared capacityContained to notification reservation

Specify acquisition order and release on failures

A parent holding shared capacity while waiting for a child pool can still spread blockage. A dedicated notification pool fails to protect inventory if blocked notification parents retain every common worker. Identify synchronous waits, asynchronous waits and downstream connection limits. When an operation needs several reservations, define acquisition order and avoid circular waiting. Added isolation can introduce its own coordination complexity.

Release capacity after exceptions, expiration, cancellation and pre-call refusal as well as success. Verify a guaranteed cleanup boundary such as finally. Small leaks can exhaust a pool under low traffic over time. Decide whether retries retain the slot or release and reacquire it, and what reserved capacity means during backoff. The check is not merely that a semaphore exists, but that every failure path returns resource counts to the expected range without defeating the isolation policy.

Choose the partitioning unit from the failure you contain

Partitions can follow dependencies, features or caller groups. Dependency-only pools do not stop one large caller from consuming every slot within a pool. Conversely, a large independent pool per caller can exceed total physical capacity. Sum reservations and decide whether an aggregate limit is also needed. Choose the unit from the unwanted influence between workloads, not from an arbitrary organizational boundary.

Connect full-pool behavior to load shedding and bounded deadline-aware waiting. A circuit breaker suppresses calls based on observed dependency failure, while a bulkhead limits resource occupancy. Even low-failure but very slow calls can exhaust a pool. These mechanisms are not interchangeable. Map their acquisition order and observations so a refusal can be attributed to a full reservation, open breaker or expired request rather than every condition becoming one opaque service error.

Verify containment while the dependency is stopped

Use a control point to stop notification progress, then observe held slots, inventory completions, queues and shared resources. Check release after demand falls and recovery after progress resumes. Synchronize on admission or acquisition rather than assuming an arbitrary sleep proved a worker had stopped. Save refusals, waits and completions, not only the fastest surviving inventory call. The six-to-four allocation is a teaching model, not a measured optimal production split.

A small low-load service may reasonably retain one pool with strict deadlines. Partitioning adds configuration, monitoring and reserved-capacity cost. Apply it where a failure trace shows the unwanted propagation, then verify the boundary. Separate processes or nodes can provide stronger physical isolation, but shared databases and other dependencies still require review. A diagram or pattern name alone does not identify the resource that can be exhausted.

Relate reservations to actual resource consumption

Six inventory slots and four notification slots do not imply at most ten database connections if one call internally opens ten connections. Define the unit represented by a slot. Worker count, request count and connection count are different quantities. Trace the resources one admitted operation can retain and sum the relevant worst-case occupancy against the dependency's capacity. This check can reveal a second shared bottleneck even when the outer semaphore works exactly as implemented. Isolation should protect a named resource and workload relationship; it cannot be inferred from a small round number of application tasks alone.

Solve the idle-reservation exercise

With six inventory slots, four notification slots, no notifications and ten inventory arrivals, strict reservation executes at most six inventory operations. Four wait or are refused in the inventory policy; they do not automatically borrow notification slots. Explain that utilization cost together with protection when notifications stall. Changing the split to eight and two trades capacity between workloads and needs both normal and failure conditions considered.

If waiting notification parents hold all ten common workers, the downstream split alone does not protect inventory admission. Change parent waiting or the shared boundary. Annotate where work stops and which resource remains held. The pattern is understood when isolated resources, shared resources, refusal and release can all be followed through the same failure history.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 16

Choose a learning path