Trace two stops around one durable send intent
Commit order-7 and event-7 in one SQLite sender transaction, then start the relay three times. Stop it before sending, stop it after receiver acknowledgment, and finally let it mark delivery complete. Two deliveries reach the receiver, but its effect counter increases only once. Read outbox and inbox first. This uses actual processes, restart and local sockets with disposable databases, not a production broker durability or performance certification.
Sender and receiver use separate database files and separate atomic boundaries. Orders and send intent commit together at the sender; identity and business effect commit together at the receiver. The remote boundary intentionally leaves a duplicate after application but before sent marking. Preserve deliveries two and effects one as separate facts. Hiding the duplicate and calling it one delivery would obscure the exact failure the experiment is meant to demonstrate.
Prepare readiness and disposable files
Save outbox.py and run it as a file with Python 3.12 or later, retaining the spawn-compatible main guard. The consumer binds an available loopback port and reports readiness through a pipe before relays begin. No long startup sleep is used. sender.db and receiver.db live in TemporaryDirectory and require no external connection settings. No real email, payment or messaging service is contacted.
The consumer handles two deliveries. Each relay is a short-lived process reading one pending row. Deliberate stop points use os._exit(23), distinguished from the final normal zero exit. Exit 23 is an expected injected stop, not automatically a failed test; unexpected codes or expired waits fail assertions. Finally terminates surviving processes before temporary storage is removed. The lifecycle checks make the restart experiment bounded even when a child cannot complete its intended stage.
Stop after acknowledgment but before sent marking
The first relay reads pending work and exits before network transmission; pending remains one. The second sends event-7 and exits after receiving ack. The consumer commits identity insertion and effect increment together before acknowledging, so one effect already exists while the sender still has one pending row. The third relay resends the same ID and marks sent after its acknowledgment.
The consumer increments effects only when INSERT OR IGNORE inserted a new identity. Identity and effect are not split into independently committed operations. This fixed event carries one known kind and payload, so the compact protocol sends only its ID. An arbitrary-payload extension must also reject conflicting content for a reused ID. A unique key does not automatically solve identity reuse or retention periods. Keep those additional contracts distinct from the atomicity exercised by this example.
import json,multiprocessing as mp,os,socket,sqlite3,tempfile
from pathlib import Path
def consumer(path,ready):
with socket.socket() as listener:
listener.bind(('127.0.0.1',0));listener.listen();listener.settimeout(10)
ready.send(listener.getsockname()[1]);ready.close()
for _ in range(2):
connection,_=listener.accept()
with connection,connection.makefile('r') as stream:
event=json.loads(stream.readline())
db=sqlite3.connect(path)
try:
with db:
fresh=db.execute('INSERT OR IGNORE INTO inbox VALUES(?)',(event['id'],)).rowcount
if fresh:db.execute('UPDATE effects SET count=count+1')
connection.sendall(b'ack\n')
finally:db.close()
def relay(path,port,stop):
db=sqlite3.connect(path)
row=db.execute('SELECT id FROM outbox WHERE sent=0 ORDER BY id LIMIT 1').fetchone()
assert row
if stop=='before-send':os._exit(23)
with socket.create_connection(('127.0.0.1',port),timeout=5) as connection:
connection.sendall((json.dumps({'id':row[0]})+'\n').encode())
with connection.makefile('r') as stream:assert stream.readline()=='ack\n'
if stop=='after-ack':os._exit(23)
with db:db.execute('UPDATE outbox SET sent=1 WHERE id=?',row)
db.close()
def pending(path):
with sqlite3.connect(path) as db:return db.execute('SELECT count(*) FROM outbox WHERE sent=0').fetchone()[0]
if __name__=='__main__':
context=mp.get_context('spawn')
with tempfile.TemporaryDirectory() as directory:
sender=str(Path(directory)/'sender.db');receiver=str(Path(directory)/'receiver.db')
with sqlite3.connect(sender) as db:
db.executescript('CREATE TABLE orders(id TEXT PRIMARY KEY);CREATE TABLE outbox(id TEXT PRIMARY KEY,sent INTEGER);')
with db:
db.execute('INSERT INTO orders VALUES(?)',('order-7',))
db.execute('INSERT INTO outbox VALUES(?,0)',('event-7',))
with sqlite3.connect(receiver) as db:
db.executescript('CREATE TABLE inbox(id TEXT PRIMARY KEY);CREATE TABLE effects(count INTEGER);INSERT INTO effects VALUES(0);')
parent,child=context.Pipe(duplex=False);worker=context.Process(target=consumer,args=(receiver,child));worker.start();child.close()
try:
assert parent.poll(10);port=parent.recv()
for stop in ['before-send','after-ack','none']:
process=context.Process(target=relay,args=(sender,port,stop));process.start()
try:
process.join(10);assert process.exitcode==(0 if stop=='none' else 23)
finally:
if process.is_alive():process.terminate();process.join()
assert pending(sender)==(0 if stop=='none' else 1)
worker.join(10);assert worker.exitcode==0
with sqlite3.connect(receiver) as db:
assert db.execute('SELECT count FROM effects').fetchone()==(1,)
assert db.execute('SELECT count(*) FROM inbox').fetchone()==(1,)
with sqlite3.connect(sender) as db:assert db.execute('SELECT count(*) FROM orders').fetchone()==(1,)
print('relay_stops=2 deliveries=2 pending=0 orders=1 inbox_ids=1 effects=1')
finally:
if worker.is_alive():worker.terminate();worker.join()
parent.close()
Inspect durable state after every relay process
Join each relay and assert pending one after both injected stops and zero after completion. Check consumer exit zero, one received identity, one effect and one order. The final report states two relay stops, two deliveries and one effect. These expectations have been executed. The experiment measures no performance ranking and does not cover every possible power-loss scenario.
The sender state looks identical after the first and second stops, while the receiver differs. Sender pending rows alone cannot reveal whether the destination already applied work. Nor does one received ID alone prove the matching effect exists; assert the effect count to catch identity-only commits. Concurrent-consumer extensions need additional race and locking tests. Sequential receipt here establishes the controlled restart history, not every concurrency guarantee of a larger design.
Do not turn a process stop into a power-loss certification
os._exit stops the relay at a selected program boundary. It does not power off the operating system or fail the storage medium. The experiment inspects files after SQLite commit returns under this environment. Claims about media failure or simultaneous loss of all nodes require additional storage, synchronization, replication and backup evidence. The injected exit remains useful precisely because its narrower boundary is known.
What is directly checked is that send intent survives relay restart and that receiver identity corresponds to one effect. Preserve exit 23 together with pending state rather than hiding the failed process from the record. A follow-up can repeat the same stop twice and predict state before running, but that should be labeled another schedule. If consumer identity were kept only in memory, restarting that process would introduce a different duplication history not covered by the current two-delivery sequence. Likewise, adding a real notification after the receiver transaction would create another effect boundary. Keep the current local atomicity claim intact while stating what each extension would need to verify; this makes a passing experiment informative without presenting it as universal crash tolerance.
If the consumer process is restarted in another trial, retain that additional stop position explicitly. It changes the recovery history rather than merely repeating the current relay-only test.
Solve changed-atomicity and identity exercises
Split identity insertion and effect increment into separate commits. A stop after identity can suppress a missing effect on retry; reversing the commits can duplicate an effect. Correct sender persistence cannot repair broken receiver atomicity. Return to each with block to identify the precise transaction containing each operation.
Generating a fresh event ID on every relay restart also allows the consumer to treat retries as distinct effects. Resending the original intention requires stable identity. At completion all children are joined and temporary databases removed. Permanent sent-row retention is outside this lab; production policies must align audit and retention with replay duration. The learning outcome is explaining deliveries, identity, effects and pending state as four distinct values through one restart history.