Malachi.Cluster.ClusterFlags (malachi v0.22.7)

The pure state of the cluster's feature flags: the set of features an operator has switched on, replicated by Malachi.Cluster.ClusterFlagsMachine over a dedicated ra cluster, exactly as Malachi.Cluster.Ring sits behind RingMachine and Malachi.Cluster.Lease behind LeaseMachine.

A flag is the operator's commitment that a format-changing or protocol-changing feature may start being used. It exists because finishing a rolling upgrade and committing to the new behaviour are two different moments: while every node runs the new build but no flag is on, rolling the build back is still free.

A set that only grows

There is no command to turn a flag off. Once a feature is on, peers and on-disk data may already be in the new shape, and going back is a migration, not a toggle. Being grow-only is what makes three other things safe and simple:

the local cache each node keeps may lag without any risk, because the worst a node that has not noticed yet can do is keep using the old behaviour, which every node still understands;
two operators racing the same flag converge, since enable is idempotent;
applying the log in any order reaches the same state, so a replay after a restart needs no care.

Why the machine does not validate the name

apply/2 accepts any atom. Checking the name against a compile-time registry here would look tighter and would be a determinism bug: two members can sit at the same effective machine version on two different releases whose registries differ, and they would then disagree about whether the same command is valid, which is exactly the divergence the machine version exists to prevent. The name is validated by the caller, next to the capability check that actually gates the flip (Malachi.Cluster.ClusterFlagsServer.enable/4).

Why this store and not the metadata

Malachi.Metadata is sharded one Raft group per vnode, so a flag written there would be a per-vnode fact with no single source of truth, and the sharded write path routes by topic, which a command that names no topic cannot do. The ring store is not an option either: it is only formed on a clustered node, and a single-node deployment still has to be able to turn a format-changing feature on.

Summary

Types

command()

{:enable_flag, flag} switches flag on. Idempotent, and there is deliberately no command that switches one off.

flag()

A flag, named after the capability every node must advertise for it (Malachi.Cluster.Capabilities).

reply()

:ok for a flag that is now on, or {:error, :invalid_flag} for a command whose flag is not an atom.

t()

The flags an operator has switched on.

Functions

apply(state, arg)

Applies a command, returning {new_state, reply}. Deterministic: no clock, no randomness, no node identity, so every replica reaches the same state from the same log.

command_versions()

Every command shape, mapped to the machine version that introduced it (see Malachi.Cluster.MachineVersion).

enabled(state)

Every switched-on flag, sorted, so two reads of the same state compare equal.

enabled?(state, flag)

Whether flag is switched on.

new()

A store with no flag switched on (the state every cluster starts in).

Types

command()

@type command() :: {:enable_flag, flag()}

{:enable_flag, flag} switches flag on. Idempotent, and there is deliberately no command that switches one off.

flag()

@type flag() :: atom()

A flag, named after the capability every node must advertise for it (Malachi.Cluster.Capabilities).

reply()

@type reply() :: :ok | {:error, :invalid_flag}

:ok for a flag that is now on, or {:error, :invalid_flag} for a command whose flag is not an atom.

The second one is why there is a reply type at all. Malachi.Cluster.MachineVersion's table is keyed by a command's shape, its tag and arity, which cannot tell {:enable_flag, :batch_format} from {:enable_flag, "batch_format"}. A clause that only accepted the atom would let the binary through admission and then raise here, on every replica, on every replay of the log, which is the exact failure that module's own documentation gives as the reason the gate exists.

t()

@type t() :: %Malachi.Cluster.ClusterFlags{enabled: %{optional(flag()) => true}}

The flags an operator has switched on.

Functions

apply(state, arg)

@spec apply(t(), command()) :: {t(), reply()}

Applies a command, returning {new_state, reply}. Deterministic: no clock, no randomness, no node identity, so every replica reaches the same state from the same log.

{:enable_flag, flag} always answers :ok, whether or not the flag was already on. An operator who re-runs the command, and a coordinator retrying after a timeout it never saw the answer to, both get the same answer as the first caller.

A flag that is not an atom is refused with the state untouched. Nothing this repository writes produces one, because Malachi.Cluster.ClusterFlagsServer resolves the operator's text to a known atom before submitting, but refusing is the only safe answer available: raising here would take down every replica and keep doing it on every replay, since the log is never truncated.

command_versions()

Every command shape, mapped to the machine version that introduced it (see Malachi.Cluster.MachineVersion).

enabled(state)

@spec enabled(t()) :: [flag()]

Every switched-on flag, sorted, so two reads of the same state compare equal.

enabled?(state, flag)

@spec enabled?(t(), flag()) :: boolean()

Whether flag is switched on.

new()

@spec new() :: t()

A store with no flag switched on (the state every cluster starts in).