Keeps this node's per-vnode coordinators in sync with the vnodes it currently leads (1C-b-ii). On a level-triggered reconcile. Right after start (handle_continue), every :interval ms, and on a :topology_changed nudge - it reads the vnode placement, compares the vnodes this node leads now with the ones it already runs coordinators for, then starts coordinators for newly-led vnodes and stops them for vnodes it no longer leads. Generic and testable via seams:

• :placement - (-> {:ok, placement} | {:error, reason}), the vnodes the cluster has right now (e.g. the live ring this node tracks). Read once per reconcile, so the two derivations below can never disagree about which vnodes exist within a pass;

• :leading - (placement -> [vnode_id]), the vnodes this node currently leads (e.g. leading_vnodes/4 over live ra membership and leadership);
• :spawn - (vnode_id -> pid), starts that vnode's coordinators (e.g. a Supervisor under a DynamicSupervisor) and returns the pid of that (sub)tree, which the manager monitors and later stops by;
• :stop - (pid -> any), stops a vnode's coordinators;
• :version_servers - optional (placement -> [{machine, server_id}]), the local vnode members whose machine version to watch (default none);

• :resume - optional (placement -> [{vnode_id, :ok | {:error, reason}}]), brings this node's stopped vnode members back and says which it tried (e.g. Malachi.Application.resume_local_vnodes/3; default none). One that raises skips the resume for that pass (see "A resume that raises");

• :interval - reconcile period in ms (default 5_000);
• :name - optional registered name.

Why the placement is a seam and not a list

Every other input here is level-triggered, and this one used to be the exception: the placement was resolved once at boot and captured. A vnode this node gains afterwards, through a rebalance that adds it as an ra member or through a split that creates a new vnode, was then invisible, so none of its per-vnode work ran until the node restarted for some other reason: its segments never expired, its under-replicated ones were never repaired, its consumer-group coordinator never started, and its member was never watched for a machine version it had stopped applying.

An unreadable placement is inert

A read that fails leaves the running set, and the last version statuses, exactly as they are. The alternative, treating a failed read as "this node leads nothing", would stop every coordinator on the node over a transient read error, which is worse than the gap above. The state is logged on the transition into and out of that condition (not on every pass) and reported on every pass through the [:malachi, :cluster, :vnode_reconcile] telemetry event.

An empty placement is not a failed read: a cluster genuinely has no vnodes before its ring is published, and a node that hosts none of them correctly runs no coordinators.

Idempotent: a transient leadership flap just starts/stops coordinators; the underlying work is idempotent and routed through ra, so a brief double-run only redoes work (the same reasoning as 1C-a, hence no lease). reconcile_now/1 reconciles synchronously and returns the running vnode ids (a manual trigger, e.g. for tests).

Self-healing on death: the manager monitors each spawned pid. If a vnode's coordinator tree dies on its own (e.g. its supervisor exhausts its restart intensity), the manager drops it so the next reconcile starts it again while this node still leads it, rather than leaving it silently down until leadership changes. The interval paces the retry, so a coordinator that keeps dying respawns at most once per reconcile with a log each time instead of spinning. A deliberate stop demonitors first, so it is never mistaken for a death.

Resume first: every pass starts by bringing this node's stopped vnode members back (:resume), before anything reads leadership. ra does not restart a node's registered servers when the node comes back, and a stopped member reads as not hosted, so without this a restarted node would neither rejoin its vnodes nor ever see that it had not. A member brought back is logged each time; one that cannot be is logged once while it keeps failing, and again only after it recovered and failed anew.

A resume that raises: the production resume reads the ra directory, which raises while the ra system restarts. That skips the resume for the pass rather than crashing the manager, which shares a one_for_all supervisor with every coordinator it runs and would otherwise restart them all on every tick the cause lasts. The rest of the pass still runs, so the coordinators follow whatever leadership it reads. The cause is logged once while it lasts and again only after a resume completed; the last per-member outcomes are kept, so a member that is still failing is not logged anew.

Version watch: on the same tick and over the same placement snapshot, every member returned by :version_servers is checked with Malachi.Cluster.MachineVersion.check/3, whether this node leads it or not, since a follower is just as able to stop applying its log. The last status per member is kept here, so a member that stays stuck is logged once, and a member no longer listed is forgotten.