Closes the self-healing loop for sealed segments: the decision comes from Malachi.Cluster.Placement and the execution from Malachi.Cluster.Catchup.

Given the current Malachi.Metadata, the set of live brokers, and the replication factor, heal_sealed/4 finds the sealed segments that are under-replicated (a replica left), picks the healed replica set with Placement.place/3, and backfills each newly added replica with the segment's records (copied from a surviving replica via Catchup.run/6). It returns the :set_segment_replicas commands whose backfill succeeded, for the caller to apply through the control plane (Malachi.Metadata / the replicated DS-RSM), plus any segments it could not heal.

Only sealed segments are handled here: their offset range [start_offset, start_offset + length) is fixed, so a backfill is a well-defined copy. The active segment grows as it is written, so a follower that falls behind on it rejoins via the write-path catch-up trigger (a separate slice) rather than a one-shot backfill.

Brokers are Malachi.Cluster.ReplicationServer references. The live-broker set is supplied by the caller (membership is a separate concern); a segment whose every replica is dead cannot be backfilled and is reported as failed rather than silently dropped.

Besides the metadata-level pass, heal_sealed/4 also runs a physical integrity pass over sealed segments whose replica set looks healthy: a live node can lose a sealed copy without the metadata noticing (a deleted or truncated file, a swapped disk), leaving the cluster silently below its replication factor while every replica is "alive". The pass probes each live replica's stored bytes (Malachi.Cluster.ReplicationServer.stored_bytes/3, a read-only stat that never opens the log, so steady state costs no descriptors); only a replica whose bytes fall short of the sealed byte_size is then opened for its exact durable end (Malachi.Cluster.ReplicationServer.durable_end/4) and re-backfilled from an intact replica via Malachi.Cluster.Catchup, resuming at the truncation point rather than recopying the segment. In-place corruption that keeps the byte size is out of this probe's reach by construction, since it compares sizes: that is what Malachi.Cluster.Scrubber verifies checksums for.

The same probe answers the opposite question at no extra cost. A copy whose bytes RUN PAST the sealed byte_size is reported as :unsettled, for Malachi.Cluster.SealedOverrun to bring back down to the recorded length and fence there. Until that has happened a copy can be past the recorded length for two very different reasons, one benign (a follower that never sealed its store still carries its preallocated tail) and one not (it holds a record the seal excludes), and the byte count alone cannot tell them apart. It does not have to: both end at the same place, and a copy that has been settled is byte-exact and is no longer selected. The exception is a copy that still measures above the recorded size after it is settled, such as one that holds exactly its seal's records: it is returned again in the unsettled list until it is confirmed, on every pass in which its segment is fully live, not under-replicated and the copy is not latched, and whose segment's latch state was readable this tick, unless its own probe times out on that tick. A sealed size is immutable, so the copy is confirmed once and not selected again. Nothing is repaired from this list. Settling such a copy costs a directory fsync per pass that selects it until it is confirmed, round-robin and bounded by the settle batch. On a copy whose log is not yet sealed, its first settle also costs the seal marker write, and, when the log has an active segment, a data sync and a roll.

A copy that failed in storage

A sealed copy its replication server has latched as failed (see "Storage failures" in Malachi.Cluster.ReplicationServer, and the :failed option) is a lost replica on a live broker: it refuses every request, so it is neither a backfill source nor a placement target for that segment, and the segment is healed as if that broker had left it. The healed set leaves the copy out, and the caller deletes it once the new set is recorded (Malachi.Cluster.HealCoordinator).

A failed copy is REPLACED, never merely dropped. With no spare broker to take its place it stays in the set, moved behind the healthy replicas so reads avoid it, and the segment is reported as failed with {:no_spare_broker, copies} on every pass. Dropping it would give up the one chance it has left: the latch lives in memory, and after a restart a copy whose sealed bytes were intact serves again.