Design patterns and optimization · Intermediate

Shed load to protect useful completion

Use concurrency, bounded queues, deadlines and priorities to reject excess work while preserving result semantics and recovery.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Reject work that cannot complete usefully
  2. Use waiting cost as well as queue length
  3. Align refusal semantics with retry behavior
  4. Make priority and fairness explicit
  5. Test recovery as well as rejection
  6. Count useful completion among admitted requests
  7. Solve the overflow-queue exercise
  8. Ask Logeia about this articleCheck your understanding

Reject work that cannot complete usefully

When order arrivals exceed capacity, putting everything in a queue accumulates work that cannot finish before callers' deadlines. Waiting requests retain connections or memory, and retries can amplify the overload. Load shedding rejects some new work early to protect useful completion and recovery. Read bounded queues and deadline propagation first. The objective is not an attractive error rate but sustained useful work under excess demand.

For example, allow ten active operations and twenty waiting ones. When both are full, reject the thirty-first outstanding request rather than moving it to an unbounded side list. Even a free slot can be wasted on work whose remaining deadline makes completion implausible. Admission before acquiring scarce database connections is often a meaningful boundary, though classification and authentication still cost resources. The admission layer itself therefore needs a capacity model rather than being assumed infinitely cheap.

Use waiting cost as well as queue length

In a fixed-duration teaching model, ten parallel workers each take fifty milliseconds and twenty jobs already wait. A newly queued job behind those groups needs roughly 150 ms through completion. An eighty-millisecond remainder is therefore unlikely to produce useful completion. This is an admission model, not a real-machine bound. Variation, priorities and uneven service times require a correspondingly different estimate.

A count cap alone treats a large aggregation and a tiny lookup as equal jobs. Resource use can be exhausted at a low count. Combine request class, estimated cost, concurrency and queue delay, leaving margin for estimation error. CPU utilization alone can miss connection exhaustion or lock waits. Observe the scarce resource and completed-work delay together. The model should explain what is being protected and why admission occurs at that boundary, rather than merely specifying a threshold with no workload meaning.

Align refusal semantics with retry behavior

An admission refusal before business effects can state that work was not accepted. A timeout after downstream execution may instead leave an unknown result and must not receive the same classification. Follow the actual HTTP or RPC semantics for overload responses. Distinguish caller-specific limits from service-wide temporary overload. Any retry still obeys deadline and extra-attempt budget; synchronized immediate retries can return the rejected load at once.

Rejecting new work cannot restore a service whose accepted operations wait forever. Bound execution, downstream waiting and retries too. Load shedding, backpressure and retry budgets can cooperate while serving different roles: refusing admission, asking producers to slow down and bounding extra attempts. Name those responsibilities separately and inspect every path. Installing a named mechanism at one boundary does not establish that total work is bounded across the whole system.

Make priority and fairness explicit

A service can protect inventory reservation by dropping optional recommendations first. Do not let an untrusted client header grant unlimited highest priority. Define trusted classes, per-caller limits and treatment of older waiting work. Even the highest-priority pool has finite capacity and needs a refusal outcome. Zero refusals is not a coherent objective under unlimited arrivals.

If a degraded response omits optional results or serves stale cached data, include that behavior and freshness condition in the product contract. It cannot turn an uncommitted order into a successful one. Hidden loss of required work can make latency and success metrics look better without improving outcomes. Observe refusals, useful completion and delay by class, including caller skew and recovery after overload. Threshold flapping may justify margin or smoothing, but the control policy must remain tied to the workload and resource it protects.

Test recovery as well as rejection

Check just below and above limits, short bursts, sustained overload and return to lower load. Save rejection cost, completion of admitted work, maximum queue size and retained connections. A load generator that waits for each success before sending again can reduce offered load during slowdown and hide the intended stress. Record arrival behavior and sender waiting. The numerical examples here are capacity models, not a report of a production load test.

A small low-load service may need only a fixed concurrency cap and a short bounded queue. Identify the scarce resource before adding elaborate cost estimation or priorities. Distinguish insufficient capacity from unbounded admission that prevents recovery. Bulkheads can isolate stalled dependencies, while circuit breakers suppress repeated failing calls. Select them from a trace of stopped work and retained resources, not as an automatic list of mechanisms every service must install.

Count useful completion among admitted requests

One hundred refusals can coexist with ten admitted requests completing within their deadlines, which may be the intended protection. Zero refusals achieved by indefinite waiting can instead conceal abandoned work. Evaluate completed useful requests and deadline behavior at the same offered arrival rate. Retain total submissions so missing or timed-out requests do not disappear from the denominator. This also keeps a degraded response from silently being counted as complete required work. The policy's success depends on the stated service objective and finite capacity, not on making one isolated error percentage as small as possible.

Solve the overflow-queue exercise

With ten active and twenty waiting requests, moving the thirty-first to a temporary side queue only bypasses the cap unless that queue has its own explicit bound and expiry. Return to useful completion rather than the appearance of acceptance. A genuinely asynchronous later-processing contract needs durability, visible state, deadlines, identity and cancellation. An in-memory side list does not provide those properties.

If clients retry overload responses immediately without limits, early refusal can still impede recovery. Coordinate count, spacing and time budget while preserving refusal meaning. Draw admission, acceptance and effect-commit boundaries and describe what each response establishes. That trace is the learning outcome: a policy that protects finite capacity and communicates a result callers can recover from correctly.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 16

Choose a learning path