Malachi.Cluster.RaResume (malachi v0.22.7)

View Source

Resume-first lifecycle for ra members: restart the local server if it ever existed on this node, and only fall back to forming/joining when the name was never started here.

Order matters for durability. :ra.start_cluster (and :ra.start_server) REGISTER a fresh, empty uid for the server name before they can fail on an already-formed cluster, which orphans the persisted Raft log under the old uid; a subsequent restart then resurrects an AMNESIAC member over the empty uid. One amnesiac is masked (the leader replicates state back into it), but two rebooting nodes can form an empty-log quorum, elect an empty leader, and truncate the surviving member's real state: the storage-chaos harness caught the whole control plane wiped this way, with every health check green throughout. Resuming first means a registered uid is never overwritten, so a member always comes back with its history.

Summary

Functions

registered?(system, arg)

Whether server_id's name has ever been started on this node in system: its uid is in ra's directory, which survives a restart with the data directory. The question resume_or/3 answers by trying, asked without starting anything, for a caller that must decide before the member comes up whether what it finds on disk belongs to it (Malachi.Storage.DataDirGuard).

resume_or(system, server_id, never_started_fun)

Restarts server_id in system if this node has ever started it; calls never_started_fun (which should form or join the cluster) only when the name is unknown to this node. Any other restart failure (a registered member that cannot come back: corrupt dir, bad system state) is returned as {:error, reason} rather than falling through to formation, because re-forming over a registered member is exactly the amnesia path described in the moduledoc.

Functions

registered?(system, arg)

@spec registered?(atom(), :ra.server_id()) :: boolean()

Whether server_id's name has ever been started on this node in system: its uid is in ra's directory, which survives a restart with the data directory. The question resume_or/3 answers by trying, asked without starting anything, for a caller that must decide before the member comes up whether what it finds on disk belongs to it (Malachi.Storage.DataDirGuard).

resume_or(system, server_id, never_started_fun)

@spec resume_or(atom(), :ra.server_id(), (-> result)) ::
  :ok | result | {:error, term()}
when result: term()

Restarts server_id in system if this node has ever started it; calls never_started_fun (which should form or join the cluster) only when the name is unknown to this node. Any other restart failure (a registered member that cannot come back: corrupt dir, bad system state) is returned as {:error, reason} rather than falling through to formation, because re-forming over a registered member is exactly the amnesia path described in the moduledoc.

Returns :ok when resumed (or already running), otherwise never_started_fun's result or the restart error.