UPGRADE AND ROLLBACK CERTIFICATION FAILED at replication factor 3. What broke is listed under Failures.
Measured with scripts/docker-upgrade-chaos.sh, the rolling upgrade and rollback drill. See Running the chaos drills for the other drills and what each one certifies.
Faults injected
p1: rolling every node from OLD to NEW (pin 7)
p1: mixed cluster (node 3 on NEW, nodes 1 and 2 on OLD)
p1: splitting the metadata ring from 4 to 5 vnodes on NEW
p2: rolling every node from NEW back to OLD (pin 7)
p3: rolling every node from OLD to NEW (pin 7)
p3: finalizing: the pin removed node by node
p3: switching the canary flag on
p3: node 3 started on OLD after the flip must refuse its data directory
Invariants
Measure
Value
Acknowledged writes still readable
53,005
Post-chaos produce
2,717 records/s
Failures
no produce was acknowledged through p3: node 3 without the pin, while malachi3 was down (stuck at 1465)
no produce was acknowledged through the refused rollback of node 3 (quorum on 2/3) (stuck at 3630)
nodes 1 and 2 did not serve a clean produce while node 3 was down
no produce was acknowledged through p3: node 3 back on NEW, while malachi3 was down (stuck at 3630)
Run details
Measure
Value
acked_writes_by_phase
phase1: 18950, phase2: 12592, phase3: 21463
negative_control
false
old_machine_version
7
old_ref
v0.22.4
old_rule
the newest release whose code differs from this tree, which is release v0.22.6's
old_sha
f5070bcb07367c783871697b8903ec750807c6b4
old_version
0.22.4
vnode_rule
sharded: v0.22.4 resumes its vnode members after a restart