Separate a new placement map from moving live data
A service partitions data across A, B and C, then adds D. Computing D's assignment does not put the assigned data on its disk. Partitioning selects responsibility; live rebalancing must move data and authority while reads and writes continue correctly. Read replication and data placement first. Separate mapping, copying, catch-up and cutover rather than assuming another server immediately creates safe distributed capacity.
Use ranges A [0,40), B [40,80) and C [80,120). Move B's latter range [60,80) to D. Of fixed example keys ten, thirty, fifty, seventy, ninety and 110, only seventy moves. Boundary sixty belongs to D and eighty to C under lower-inclusive, upper-exclusive intervals. This is a worked placement model, not an executed production migration. Draw the map used by readers and writers before tracing data transfer.
Catch changes that occur after the initial copy
Copy key seventy at stock nine from B to D, then let B accept an update to eight. Cutting over merely because the copy finished can expose stale nine as authoritative. Save a copy baseline position and apply subsequent changes, or use another verified mechanism such as a brief write pause or range lock. Choose what the actual storage system supports. Copied bytes and caught-up changes are different progress measures.
Deletion is a change too. Repeatedly copying only existing rows can leave a deleted key stranded at D. Define sequence and deletion handling so a stated position corresponds to a coherent state. Moving related tables also requires relationships and transaction boundaries considered together. This example uses one range; a phrase such as copy keys does not automatically include all dependent state. Define migration unit and correctness checks before execution.
Transfer authority as well as bytes
Label old placement generation seven and new placement eight. After D catches the required position, commit the authoritative map change and direct new work to D. A stale client writing to B needs a defined refusal or redirect contract. Letting B and D independently commit because each believes itself owner is not solved by simply sending to both.
Generation labels do not replace agreement about which map is authoritative. If B can still write without observing the new authority, a larger number alone is not fencing. Control what the actual storage boundary accepts. Keeping old data temporarily after cutover can support stale-route handling and recovery, but removal has its own readiness conditions. Trace authority, retry and read semantics before reclaiming B's copy. Correct migration transfers responsibility, not merely a file.
| Stage | Write authority | D behavior |
|---|---|---|
| Before copy | B, generation 7 | Not serving new ownership |
| Copy and catch-up | B retains changes | Apply through required position |
| Committed cutover | D, generation 8 | New authoritative owner |
| Old-copy cleanup | D | Validate old routes before removal |
Budget migration load under live traffic
Copying consumes network, storage, CPU and cache resources. Adding D can temporarily slow B through copy reads and retained change logs. Observe bytes, concurrent ranges, bandwidth limits and catch-up rate. If updates arrive faster than D applies them, merely waiting will not reach the cutover condition. Throttle work, reduce migration units or plan a bounded pause under explicit policy.
Equal key counts do not imply equal traffic. A single hot key seventy can dominate the benefit of moving the range, while many cold keys may change little. Measure bytes, request rate, update rate and hot-key concentration separately. Consistent hashing can reduce reassignment without solving safe live movement or splitting one hot key. Placement properties and migration safety are distinct claims, even when one mechanism supports the other.
Keep replication, partitioning and recovery state separate
If each range has three replicas, the number of ranges differs from a key's replica count. Four owners do not automatically imply a four-member quorum. Key seventy's replica set and membership protocol determine that question. Placement metadata also needs its own authoritative storage. Updating a map row does not prove old and new replicas share a safe committed history.
A temporary migration copy is not automatically a backup. Recovery from erroneous deletion remains another requirement after old copies are reclaimed. Pending outbox work and processed request identities may belong to migrated state too. Losing deduplication records while preserving business rows can duplicate effects after cutover retries. Inventory the complete required state: business data, constraints, progress and replay protection. This makes the migration unit match the application contract instead of treating only the obvious primary table as data.
Plan recovery by the actual failed stage
Before cutover, the old owner may remain authoritative and a partial copy can restart. After committed cutover, blindly restoring the old map can lose updates accepted at D. A generic rollback label is insufficient. Check committed placement and update positions. A restarted migration owner needs stable job identity and generation, and concurrent controllers must not move one range in contradictory directions.
Test boundary keys, live updates and deletions, stale-generation retries, and stops immediately before and after cutover. Compare values, versions, identities and deletion state as well as counts. The hash lab verifies placement changes only; it does not perform the copying or authority transfer described here. Separate its model evidence from a real storage migration test. The stage table supports that later test design without claiming the entire procedure has already been executed.
Verify contents at a comparable position, not just counts
Equal row counts do not make migration correct if key seventy is nine at one owner and eight at the other. Compare keys, values, versions, deletions and relevant constraints at a corresponding baseline position. Range hashes or sampling can reduce comparison cost, but introduce collision or coverage assumptions that must remain visible. A count check must not be reported as complete content equality.
Live writes complicate inspection: reading B at one position and D later can reveal a legitimate catch-up difference rather than corruption. Save the compared positions and judge against the cutover requirement. Verification scans also consume resources. Unlimited concurrent validation can disrupt the ordinary workload migration was intended to protect, so budget it with copying and change replay. The migration record should connect a specific ownership generation and apply position to each comparison, making restart or an unexpected mismatch diagnosable. This supplies a concrete acceptance condition beyond a progress bar reaching one hundred percent.
On mismatch, preserve configuration generation, copy baseline, last applied change and inspection position. Repeating until a comparison passes can hide what changed. A reproducible failed comparison is useful evidence for deciding whether cutover is permitted, not merely a progress obstacle to discard.
Solve the update-after-copy exercise
B is copied at stock nine, then commits eight before cutover. Without catch-up or another mechanism preserving that update, D becomes authoritative with stale state. Copy completion alone is not sufficient. Under the half-open range convention, key sixty belongs to D and eighty to C; no boundary should be assigned to both or neither.
After D accepts a new write, restoring generation seven alone may route back to B without that update. A reverse cutover is another data-and-authority migration, not something made safe merely by retaining an old copy. The learning outcome tracks map, contents, change position, authority and stale requests at each stage. The hashing articles next study reassignment volume as a distinct question from live-movement correctness.