Malachi.Cluster.RingTopology (malachi v0.22.7)

View Source

The cluster's routing topology as a versioned, gossip-disseminable value: a HashRing (which vnode owns which hash arc) plus each vnode's node placement (%{vnode_id => [node]}), tagged with a monotonic version.

This is how a vnode split propagates. NorthGuard keeps "very minimal global states" and spreads them over its SWIM dissemination path (the meetup transcript: "we also use this dissemination for spreading some minimal global like cluster metadata"); the ring is exactly such minimal global state. So rather than a dedicated Raft cluster for the ring (strongly consistent but heavier and less faithful), the ring rides the existing gossip and every node converges on the latest version.

Convergence is a CRDT join: merge/2 keeps the higher version (last-version-wins), with a deterministic term-order tiebreak on the rare same-version clash, so merge is commutative, associative and idempotent, and every node reaches the same topology regardless of gossip order. Only the rebalancing leader advance/3s the version (one writer), so a same-version clash should not arise in normal operation; the tiebreak just keeps the math total.

The topology also carries an optional pending-split intent (begin_split/4): the durable record of a split that is migrating but not yet complete. It rides the same gossip, so if the coordinator driving a split crashes, whichever node takes over the lease reads the intent and reconciles the interrupted split (see pending/0). advance/3 (complete) and clear_pending/1 (abort) both clear it.

Summary

Types

pending()

A split in flight: the durable intent recorded before a split migrates, so a coordinator that takes over the lease after the previous one crashed mid-split can reconcile it (nil when none is pending). Carries what a reconciler needs to identify and undo the split: the new vnode, its ring token, and its placement nodes (the old ring/placements are the topology's own, since a pending split has not yet advanced the ring).

placements()
t()

Functions

advance(ring_topology, ring, placements)

A new topology at version + 1 with ring/placements: the single-writer bump the rebalancing leader applies to complete a split (or any ring change). The monotonic version is what makes merge/2 converge. Completing a ring change also clears any pending-split intent: the split it recorded is now reflected in the ring, so there is nothing left to reconcile.

begin_split(topology, new_vnode_id, token, nodes)

Records a pending split at version + 1 without changing the ring: the intent that a split of new_vnode_id at token onto nodes is now in flight. Written (and gossiped) before the migration so a coordinator that takes over the lease mid-split can find and reconcile it. The ring still routes by the pre-split placement until advance/3 completes the split (or clear_pending/1 aborts it).

clear_pending(topology)

Drops a pending-split intent at version + 1, keeping the ring unchanged, how an aborted split is published (the reconciler rolled the migration back, so the ring stays as it was and the intent is gone). Contrast advance/3, which clears the intent by moving the ring forward to the completed split.

merge(a, b)

CRDT join of two observations of the topology: the higher version wins; a same-version clash is broken deterministically by Erlang term order (so the result is independent of argument order). Idempotent (merge(a, a) == a), commutative and associative.

new(ring, placements)

The initial (version 0) topology for ring and its vnode placements; no split pending.

servers(ring_topology)

The %{vnode_id => server_id} routing map derived from the placements: each vnode's server id is {vnode_id, a_member} (any member of its placement: ra resolves the live leader). A vnode with an empty placement is skipped (it has no cluster to route to). Used to point routing at the topology.

vnode_placement(ring_topology)

The topology as the vnode placement the boot path speaks: [{vnode_id, token, nodes}], one entry per vnode on the ring, ordered by token so it is stable across nodes.

Types

pending()

@type pending() ::
  %{
    new_vnode: Malachi.Cluster.HashRing.vnode_id(),
    token: Malachi.Cluster.HashRing.token(),
    nodes: [node()]
  }
  | nil

A split in flight: the durable intent recorded before a split migrates, so a coordinator that takes over the lease after the previous one crashed mid-split can reconcile it (nil when none is pending). Carries what a reconciler needs to identify and undo the split: the new vnode, its ring token, and its placement nodes (the old ring/placements are the topology's own, since a pending split has not yet advanced the ring).

placements()

@type placements() :: %{required(term()) => [node()]}

t()

@type t() :: %Malachi.Cluster.RingTopology{
  pending: pending(),
  placements: placements(),
  ring: Malachi.Cluster.HashRing.t() | nil,
  version: non_neg_integer()
}

Functions

advance(ring_topology, ring, placements)

@spec advance(t(), Malachi.Cluster.HashRing.t(), placements()) :: t()

A new topology at version + 1 with ring/placements: the single-writer bump the rebalancing leader applies to complete a split (or any ring change). The monotonic version is what makes merge/2 converge. Completing a ring change also clears any pending-split intent: the split it recorded is now reflected in the ring, so there is nothing left to reconcile.

begin_split(topology, new_vnode_id, token, nodes)

@spec begin_split(
  t(),
  Malachi.Cluster.HashRing.vnode_id(),
  Malachi.Cluster.HashRing.token(),
  [node()]
) ::
  t()

Records a pending split at version + 1 without changing the ring: the intent that a split of new_vnode_id at token onto nodes is now in flight. Written (and gossiped) before the migration so a coordinator that takes over the lease mid-split can find and reconcile it. The ring still routes by the pre-split placement until advance/3 completes the split (or clear_pending/1 aborts it).

clear_pending(topology)

@spec clear_pending(t()) :: t()

Drops a pending-split intent at version + 1, keeping the ring unchanged, how an aborted split is published (the reconciler rolled the migration back, so the ring stays as it was and the intent is gone). Contrast advance/3, which clears the intent by moving the ring forward to the completed split.

merge(a, b)

@spec merge(t(), t()) :: t()

CRDT join of two observations of the topology: the higher version wins; a same-version clash is broken deterministically by Erlang term order (so the result is independent of argument order). Idempotent (merge(a, a) == a), commutative and associative.

new(ring, placements)

@spec new(Malachi.Cluster.HashRing.t(), placements()) :: t()

The initial (version 0) topology for ring and its vnode placements; no split pending.

servers(ring_topology)

@spec servers(t()) :: %{required(term()) => {term(), node()}}

The %{vnode_id => server_id} routing map derived from the placements: each vnode's server id is {vnode_id, a_member} (any member of its placement: ra resolves the live leader). A vnode with an empty placement is skipped (it has no cluster to route to). Used to point routing at the topology.

vnode_placement(ring_topology)

@spec vnode_placement(t()) :: [{term(), Malachi.Cluster.HashRing.token(), [node()]}]

The topology as the vnode placement the boot path speaks: [{vnode_id, token, nodes}], one entry per vnode on the ring, ordered by token so it is stable across nodes.

This is the shape Malachi.Application already threads through the sharded control plane (it is what place_vnodes/4 returns), so a ring restored from the durable store drops straight into the same wiring that an environment-derived one does. A vnode on the ring with no recorded placement is skipped: there is no cluster to address it through, the same rule servers/1 applies.