Malachi.Cluster.Rebalance (malachi v0.22.7)

View Source

Executes a rebalancing plan (from Malachi.Application.rebalance_plan/2) against the vnodes' ra clusters. For each change it adds the joining members before removing the leaving ones (add-before-remove, so a vnode never drops below quorum mid-move) through injected add_member / remove_member seams, so the executor is testable without ra, and R3-b supplies the real ra ops.

It is idempotent: the seams must treat an already-present add / already-gone remove as :ok, so an interrupted commit can be re-run. It is fail-fast: within a change the first failing add stops it (the removes are not attempted, protecting quorum); across a plan the first failing change stops the run, returning what was applied so a later commit resumes. Between changes it re-checks the leader? seam and stops if leadership was lost mid-commit (the lease holder dropped the lease).

Summary

Types

change()
failure()
member_op()

Functions

apply_change(map, add_member, remove_member)

Applies one change: adds every add member, then removes every remove member (never the reverse), stopping at the first failure. Returns :ok or {:error, {step, vnode_id, node, reason}}.

apply_plan(plan, add_member, remove_member, leader? \\ fn -> true end)

Applies a whole plan one change at a time, fail-fast. Before each change it checks leader? (default always) and stops if leadership was lost. Returns {:ok, applied_vnode_ids} or {:error, {applied_vnode_ids, failure}}; idempotent, so re-running resumes.

ra_add_member(vnode_id, new_node, current_members)

Adds new_node to vnode vnode_id's ra cluster (the real add_member seam apply_plan/4 uses in production): starts the ra server on new_node (via :erpc, as a member of the existing cluster) then adds it to the consensus, routing the change through current_members (any of them reaches the leader). Idempotent. An already-running server or already-present member counts as :ok. On success ra replicates the vnode's state (log/snapshot) to the new member automatically.

ra_remove_member(vnode_id, leaving_node, current_members, opts \\ [])

Removes leaving_node from vnode vnode_id's ra cluster: removes it from the consensus (routing through current_members), and once the group has committed that, deletes its server. Idempotent. A non-member counts as :ok. Within one call, a leave that took longer to commit than ra's call timeout is heard as a non-member, and that member is stopped rather than deleted (see the comment on not_member below). A leave still uncommitted when the retries run out answers an error and touches nothing; the planner then never lists that member again, since it plans from the group's live membership, so it stays registered on its node, which only resumes it when its group cannot answer.

Types

change()

@type change() :: %{vnode_id: atom(), add: [node()], remove: [node()]}

failure()

@type failure() :: {:add | :remove, atom(), node(), term()} | :lost_leadership

member_op()

@type member_op() :: (atom(), node() -> :ok | {:error, term()})

Functions

apply_change(map, add_member, remove_member)

@spec apply_change(change(), member_op(), member_op()) :: :ok | {:error, failure()}

Applies one change: adds every add member, then removes every remove member (never the reverse), stopping at the first failure. Returns :ok or {:error, {step, vnode_id, node, reason}}.

apply_plan(plan, add_member, remove_member, leader? \\ fn -> true end)

@spec apply_plan([change()], member_op(), member_op(), (-> boolean())) ::
  {:ok, [atom()]} | {:error, {[atom()], failure()}}

Applies a whole plan one change at a time, fail-fast. Before each change it checks leader? (default always) and stops if leadership was lost. Returns {:ok, applied_vnode_ids} or {:error, {applied_vnode_ids, failure}}; idempotent, so re-running resumes.

ra_add_member(vnode_id, new_node, current_members)

@spec ra_add_member(atom(), node(), [node()]) :: :ok | {:error, term()}

Adds new_node to vnode vnode_id's ra cluster (the real add_member seam apply_plan/4 uses in production): starts the ra server on new_node (via :erpc, as a member of the existing cluster) then adds it to the consensus, routing the change through current_members (any of them reaches the leader). Idempotent. An already-running server or already-present member counts as :ok. On success ra replicates the vnode's state (log/snapshot) to the new member automatically.

ra_remove_member(vnode_id, leaving_node, current_members, opts \\ [])

@spec ra_remove_member(atom(), node(), [node()], keyword()) :: :ok | {:error, term()}

Removes leaving_node from vnode vnode_id's ra cluster: removes it from the consensus (routing through current_members), and once the group has committed that, deletes its server. Idempotent. A non-member counts as :ok. Within one call, a leave that took longer to commit than ra's call timeout is heard as a non-member, and that member is stopped rather than deleted (see the comment on not_member below). A leave still uncommitted when the retries run out answers an error and touches nothing; the planner then never lists that member again, since it plans from the group's live membership, so it stays registered on its node, which only resumes it when its group cannot answer.

Deleted, not stopped: a stopped server stays registered on its node, and a node that comes back resumes its registered vnode members (Malachi.Application.resume_local_vnodes/3). Once the group has let the member go, its log describes a membership the group no longer has, and deleting it is what keeps a restart of that node from bringing it back.

Only after the commit (:ra.leave_and_delete_server/3, which waits for consensus), never on the append that :ra.remove_member answers on: until the leave commits, the leaving member may hold the only other copy of an entry the group acknowledged, with the member the rebalance just added still catching up, and deleting its log then would leave that entry on the leader's disk alone.