Pure primary-failover policy for active segments: when an active segment's primary is no longer alive, or its live replicas can no longer reach the acknowledgement quorum, the segment is sealed and writing rolls to a fresh one, which is what NorthGuard does ("we just seal it, make a new one, move the producers over to that new segment").

Sealing rather than promoting is what makes this safe. A batch is acknowledged once a majority holds it durably, so a replica outside that majority can be behind. Promoting such a replica would let it append at offsets the dead primary had already assigned and acknowledged, leaving two replicas with different records under the same offset, which nothing downstream detects: the expected_first chain in replication rejects a gap rather than a conflict, and the integrity scrub verifies each copy against itself. Sealing removes that possibility by construction, because an offset the sealed segment already assigned is never handed out again. The freshly sealed segment is then picked up by Malachi.Cluster.SelfHealing, which already re-replicates sealed segments.

The fence is what makes the seal binding rather than advisory, and Malachi.Cluster.HealCoordinator applies it in two steps, in this order. It first MEASURES every live replica with Malachi.Cluster.ReplicationServer.durable_stats/4, which reports what a replica holds and leaves it writable. Only if those answers reach the seal quorum (seal_quorum/1) does it FENCE them with Malachi.Cluster.ReplicationServer.seal/4, and the fence answers are what plan/5 then seals on, so the point is recorded after every replica behind it has stopped accepting writes. A returning old primary is refused an append by each fenced replica it reaches, including by its own store after a restart, and so cannot close a quorum.

Measuring before fencing is not a nicety. A fence has no inverse: nothing unseals a replica's store. Fencing whatever answers, before knowing whether enough did, therefore closes replicas of a segment this pass may then decline to seal, and they keep refusing writes after their primary comes back, shrinking every later write quorum for nothing. Below the seal quorum the pass therefore leaves every replica untouched, and the range stays blocked until enough return.

Note what this still does not claim: a replica the pass never reached is not fenced.

The seal point, and when a range is left blocked

The seal is placed at the highest durable end any replica reports, and only when at least the seal quorum of the replica set answered the probe: n - q + 1 answers for n replicas and an acknowledgement quorum q (Malachi.Cluster.ReplicaTracker.quorum_size/1). Both halves are forced, not chosen.

The argument, per record rather than per replica, because the difference matters: take an acknowledged record at offset o. It lives on at least q replicas of the full set, the answering replicas are at least n - q + 1 of it, and two such subsets of n replicas always share one, since together they count more than n. So some answering replica holds o. A replica's log is contiguous (replication rejects a gap), so that replica's reported end is above o, and the highest end among the answers is at least that. The covering replica may be a different one for each record; no single answer need hold the whole segment, which is exactly why the seal takes the maximum instead of trusting one replica's view.

Any lower point, including the offset a majority of the ANSWERS agree on, can sit below an acknowledged record that only the dead primary and one survivor ever held, and sealing there would discard it: exactly what this policy exists to prevent. Below the seal quorum, no such intersection is guaranteed, the committed end is unknowable, and the segment is left alone with its range no longer accepting writes.

With three replicas the seal quorum is two, a majority. With two it is one: an acknowledgement needed both replicas, so either one alone holds every acknowledged record, and a segment placed on two brokers (a roll while a third was down) is sealed on its survivor instead of blocking its range for as long as the other is gone. With one replica it is one, which a dead primary cannot give.

That block is not a latch. The caller (Malachi.Cluster.HealCoordinator) is a periodic level-triggered loop, so the next pass re-evaluates: as soon as enough answer again, the seal is emitted and writing rolls to a new segment on its own. If they never return, the range stays blocked, which is the CP choice, and recovering it is a deliberate operator decision rather than something this policy takes on the operator's behalf.

The probing itself lives in the caller: candidates/3 names the segments to probe, and plan/5 turns the probe results into commands, so the policy stays pure and testable without processes.

A copy that failed

A replica does not have to die for its copy to be lost to a segment. When a storage operation fails, Malachi.Cluster.ReplicationServer takes that copy out of service for good (it never writes it again, and answers every request for it with an error), and the segment can no longer count on it. NorthGuard treats that exactly like a failed broker, "we just seal it, make a new one, move the producers over", and so does this policy: a segment with a FAILED copy is a candidate too, whether the copy is the primary's or a follower's, with the caller learning which copies failed from Malachi.Cluster.ReplicationServer.failed_segments/3 (failure_probes/2 says whom to ask).

The argument above carries over unchanged, because a failed copy is removed from the answers just as a dead primary is: it is never probed, never fenced, and never becomes the head. The consequence carries over too: without the failed copy the seal quorum must still answer, so one failed copy seals the segment on the others at three replicas and at two, while at one the range stays blocked until the failed server restarts.

Live replicas below the quorum

A primary that is alive does not make a segment writable on its own: every append needs the acknowledgement quorum. A segment whose live, non-failed replicas fall below it (a follower gone from a segment placed on two brokers) refuses every write while its primary looks healthy, so it is a candidate too, and it is sealed on whoever answers under the same seal quorum.