Malachi.Cluster.OrphanedFence (malachi v0.22.7)

View Source

Pure policy for the state where a segment's store is fenced while the control plane still calls it active: the range is closed to writes and nothing reconciles the two halves.

Malachi.BrokerServer's split and merge path closes the segment on the primary and only then records the seal through the control plane. Those two steps cannot be made atomic, and that is not an oversight: a fence has no inverse, which is the argument the whole fence design rests on, so a failed metadata write cannot be answered by unsealing the store. On a generic error from the second step (an ra timeout is the ordinary way to get one) Malachi.Broker.record_seal/5 returns the broker unchanged, so no roll is owed and nothing retries.

What that leaves is a produce loop with no exit: ensure_segment/2 finds no cached segment, adopts the one the metadata still calls active, the store refuses the batch with {:error, {:sealed, N}}, the cache is dropped, and the next produce adopts the same segment again. A successor is never opened. Nothing else rescues it either: Malachi.Cluster.Failover takes a segment only when its primary is dead, a copy failed, or its live replicas fall below the acknowledgement quorum, and here the primary is alive, healthy and among enough replicas, while healing and retention only touch sealed segments.

So this is the level-triggered half. candidates/2 names the segments to ask about, the caller asks each primary which of them are already fenced, and plan/3 turns those answers into seal commands. Level-triggered rather than a retry at the source, because a retry only converges while the process that owed the seal is alive: Malachi.BrokerServer seeds offsets and sequence floors on restart, not rolls, so a restart loses the debt for good.

Why no seal quorum here

Malachi.Cluster.Failover seals only once its seal quorum answered, and it has to: its primary may be dead, so the committed end is only knowable from the answers every acknowledgement quorum shares a replica with. Here the primary is ALIVE and its store is fenced, and the end it reports is the very number the fencing node itself would have recorded had its metadata write landed. The fence is a latch, so nothing can extend the segment past it. That is the same rule, from the same source, and it needs no new argument.

Why the pass may not fence

The probing this policy drives must not seal or fence, which is why it asks Malachi.Cluster.ReplicationServer.fenced_segments/3 (which reports a latch) rather than seal/4 (which sets one). That probe can still latch a copy whose log fails to open, except for a failure that may clear by the next request, which is refused and reopened on the next request; see its moduledoc. Malachi.Cluster.HealCoordinator learned this the hard way on the failover path: a pass that fenced while probing closed replicas of segments it then declined to seal, and at replication_factor: 2 that wedged a range permanently, because below a majority nothing was sealed and nothing unseals a store. This pass visits EVERY active segment, so the same mistake here would be that one multiplied by the whole workload.

Why it acts on the first observation

A segment can be legitimately fenced and not yet sealed for a moment, during a normal split or merge. Acting inside that window is harmless, and deliberately so rather than by luck. The fence is a latch, so the end this pass reads is the end the splitting node read; and when the splitter's own record_seal/5 then lands it is answered {:error, {:already_sealed, existing}}, which Malachi.Broker converges on and reports as :ok. Both sides agree on the same length and the split proceeds. Waiting for a second observation would buy nothing and would cost the coordinator a piece of per-segment state that a leadership change resets anyway.

Summary

Types

What a fenced store reports for a segment: where it ended and its size on disk.

Fence answers per segment: %{segment_id => answer}. A segment that is not fenced is absent.

Functions

The active segments worth asking about, grouped by the primary to ask: [{primary, [{segment_id, start_offset}]}], sorted so a pass is deterministic.

The seal commands for the segments whose store answered FENCED, at now_ms (passed in so every replica applies the same timestamp, as the produce path already does for its own seals). Returns a sorted, deterministic list.

Types

answer()

@type answer() :: {non_neg_integer(), non_neg_integer()}

What a fenced store reports for a segment: where it ended and its size on disk.

answers()

@type answers() :: %{optional(term()) => answer()}

Fence answers per segment: %{segment_id => answer}. A segment that is not fenced is absent.

Functions

candidates(metadata, live_brokers)

The active segments worth asking about, grouped by the primary to ask: [{primary, [{segment_id, start_offset}]}], sorted so a pass is deterministic.

Grouped rather than flat because the question is asked in one batched call per primary (Malachi.Cluster.ReplicationServer.fenced_segments/3): this runs over every active segment on every pass, so a call per segment would make the poll's cost scale with the number of ranges.

Only segments whose primary is LIVE. A dead primary is Malachi.Cluster.Failover's case and sealing it from here would need that module's seal quorum. The two can still meet on one segment: a failover that fences a live primary (a failed copy, or live replicas below the acknowledgement quorum) leaves it fenced, and Malachi.Cluster.HealCoordinator drops this module's seal for a segment its own pass already sealed, so it is neither sealed twice nor counted as an orphaned fence. A segment with an empty replica set has no primary to ask and is skipped, as Malachi.Broker.active_roll/2 skips it for the same reason.

plan(metadata, answers, now_ms)

The seal commands for the segments whose store answered FENCED, at now_ms (passed in so every replica applies the same timestamp, as the produce path already does for its own seals). Returns a sorted, deterministic list.

The length comes from the store's answer, never from a counter, which is the rule Malachi.Broker.record_seal/5 already states: a sealed length has to be a consequence of closing the segment rather than a measurement racing it.

No {:set_segment_replicas, ...} alongside the seal, unlike Malachi.Cluster.Failover: that one moves a live holder to the head because the head it seals is a broker that just died. Here the head is alive and holds everything the seal promises, so reads already route correctly.

A segment that is no longer in metadata (retention dropped it between the probe and this call) is skipped rather than sealed at a start offset nobody can check.