Malachi.Cluster.VnodeSplit (malachi v0.22.7)

View Source

Runs a vnode split end to end, under the lease. Only the lease leader acts (one writer of the ring version). It reads the cluster's current Malachi.Cluster.RingTopology from the membership, records the split intent (begin_split, published before any migration so a coordinator that takes over the lease mid-split can reconcile it: B2), splits the ring over real Raft - starting the new vnode's cluster and migrating the displaced topics' metadata, fenced and copy-first (Malachi.Cluster.ReplicatedDSRSM.split_vnode/4), then advances the topology (which moves the ring forward and clears the intent) and publishes it. On a logical migration failure, split_vnode has already rolled back in-process, so it just clears the now-stale intent (clear_pending) and returns the error. From there gossip disseminates the topology and every node adopts the new ring for both metadata and consumer-group routing (Int-1 / VS-2b).

Every publication goes through the :publish seam (Malachi.Cluster.TopologyPublisher), which writes the durable ring before gossiping it, so a reshard survives a full-cluster restart. A publication the store refuses stops the split where it stands rather than disseminating a ring the cluster did not accept.

The lease seam carries a token, not a boolean

:lease answers {:ok, fence} | :error. The fence rides into every durable write, so a coordinator that lost the lease without noticing is refused by the store's compare-and-set rather than left to discover it on the next renew. That is the case a lease cannot cover on its own.

This ties the split slices together: the migration (VS-2a/VS-2c), the versioned disseminable ring (VS-2b), the durable ring (issue #32), and the runtime adoption (Int-1).

Summary

Types

lease()

The lease with its fencing token: {:ok, fence} while this node leads, :error otherwise.

option()

Required :publish (a Malachi.Cluster.TopologyPublisher.publish/0) and :lease. :lease defaults to a permanently-held lease at fence 0, which is what unit tests of the split machinery want and what the application never uses: it always wires the real holder.

Functions

reconcile(membership, opts)

Reconciles an interrupted split on lease takeover (or coordinator restart): if the topology carries a pending-split intent: a split whose coordinator crashed after recording it but before completing: completes it forward (ReplicatedDSRSM.complete_split/4: resume the migration idempotently from wherever it stopped) and publishes the finished topology (advance, which moves the ring forward and clears the intent). This is NorthGuard's "carrying it out to the end": the coordinator that takes over drives the split to completion rather than undoing it. If it cannot complete now (a vnode is unreachable), the intent is kept pending so a later takeover/restart retries: the partial state is left intact, never cleared, so no progress is lost (an explicit abort is ReplicatedDSRSM.abort_split/3). A no-op when nothing is pending. Only the lease leader acts ({:error, :not_leader} otherwise). Idempotent.

split(membership, new_vnode_id, token, nodes, opts)

Splits the vnode owning token's region by adding new_vnode_id at token (a new ra cluster on nodes), migrating the displaced topics, and publishing the advanced topology. Returns :ok, {:error, :not_leader} if this node does not hold the lease, {:error, :no_topology} if the cluster has no ring yet, {:error, {:conflict, ring}} if the durable store refused the publication, or the split_vnode/4 error (a ring/start/migration failure).

Types

lease()

@type lease() :: (-> {:ok, non_neg_integer()} | :error)

The lease with its fencing token: {:ok, fence} while this node leads, :error otherwise.

option()

@type option() ::
  {:publish, Malachi.Cluster.TopologyPublisher.publish()} | {:lease, lease()}

Required :publish (a Malachi.Cluster.TopologyPublisher.publish/0) and :lease. :lease defaults to a permanently-held lease at fence 0, which is what unit tests of the split machinery want and what the application never uses: it always wires the real holder.

Functions

reconcile(membership, opts)

@spec reconcile(GenServer.server(), [option()]) :: :ok | {:error, :not_leader}

Reconciles an interrupted split on lease takeover (or coordinator restart): if the topology carries a pending-split intent: a split whose coordinator crashed after recording it but before completing: completes it forward (ReplicatedDSRSM.complete_split/4: resume the migration idempotently from wherever it stopped) and publishes the finished topology (advance, which moves the ring forward and clears the intent). This is NorthGuard's "carrying it out to the end": the coordinator that takes over drives the split to completion rather than undoing it. If it cannot complete now (a vnode is unreachable), the intent is kept pending so a later takeover/restart retries: the partial state is left intact, never cleared, so no progress is lost (an explicit abort is ReplicatedDSRSM.abort_split/3). A no-op when nothing is pending. Only the lease leader acts ({:error, :not_leader} otherwise). Idempotent.

Because the intent is part of the durable topology, it now also survives a full-cluster restart: a split interrupted by everything going down is still there to be completed when the cluster returns.

split(membership, new_vnode_id, token, nodes, opts)

@spec split(
  GenServer.server(),
  Malachi.Cluster.HashRing.vnode_id(),
  Malachi.Cluster.HashRing.token(),
  [node()],
  [option()]
) :: :ok | {:error, term()}

Splits the vnode owning token's region by adding new_vnode_id at token (a new ra cluster on nodes), migrating the displaced topics, and publishing the advanced topology. Returns :ok, {:error, :not_leader} if this node does not hold the lease, {:error, :no_topology} if the cluster has no ring yet, {:error, {:conflict, ring}} if the durable store refused the publication, or the split_vnode/4 error (a ring/start/migration failure).