Malachi.Storage.DataDirGuard (malachi v0.22.7)

View Source

The boot check that a log data directory and the control plane this node is about to start describe the same data.

ra keeps a node's control plane under the node's name: its log lives under MALACHI_RA_DATA_DIR/<node>, and every member is recorded as {cluster, node()}. What a node acknowledged survives a restart only while the segments on disk and that control plane come back together, and when they do not, the orphan sweep used to settle it by deleting the segments (#273). Two things are checked before the broker starts, after a first one on the directory's own record:

  • a log directory of another control plane. A start that passes records the control plane's name at the root of the log directory (malachi.cluster), just before it writes a seed or once every check has passed, and a later start under another name refuses, before the ring store is touched. A start refused before its seed is written records nothing; one that fails after the seed check passed (the seed write, or a later membership check) keeps the name it seeded under, since the seed's vnodes carry that name and whether the write landed may not be known. Every other check reads ra by node name, and the ring store is one per ra directory and node name whatever the cluster is called, so a renamed control plane over the same directories would read the old one's state as its own; a member that held no segment would pass every other check and could seed the ring its peers then trust.

  • segments nobody knows. The log directory already holds segment directories, and the control plane that would describe them is about to be formed rather than resumed: the ring store (Malachi.LogRing, which every clustered node runs, sharded or not) was never started on this node; or, for an unsharded control plane, its metadata member was never started here; or a sharded ring is about to be seeded, forming every vnode now (check_seed/3); or, for a sharded node configured alone, one of its vnodes was never started here. A cluster is not held to that, since a rebalance moves vnodes off a node without republishing the ring, and a vnode with other replicas keeps its history on them; a vnode whose only replica is this node (MALACHI_LOG_VNODE_REPLICATION_FACTOR=1 on a cluster) has no such protection. The node name changed (a container recreated without a fixed hostname, iex --name with another name), the ra directory was lost, the cluster was renamed, or the segments were written by a release that kept a single node's metadata in memory. On a single node a control plane formed now is empty, every one of those directories would read as an orphan, and a topic created again under an old name would read the old bytes as its own. A cluster member is refused too, though its peers may still hold the control plane: it would come back an amnesiac voter, and adopting is then the operator's way back, the sweep removing only what no owner lists. A node that joins a cluster for the first time has nothing on disk to refuse.

  • a single node told it has peers. The control plane was started here as a cluster of one, and the configuration now lists other nodes. Growing a one-member cluster in place is not supported: this node would resume its ring store (and its metadata, sharded or not) alone while the others formed clusters of their own over the same names. The ring store member, which every clustered node runs and which was formed over the same nodes as the metadata, is asked for its recorded membership; a member that does not answer within the ring boot timeout refuses the start too, since a check that cannot be made must not read as passed.

Each refuses the start through Malachi.StartupRefusal (exit 78) with a line that names the causes and the way out. Segments nobody knows can be adopted with MALACHI_ADOPT_ORPHANED_LOG_DIR=true, which is the operator saying the directories are not wanted: the start goes on, the adopted names are logged, and the orphan sweep removes them on its own schedule. The others cannot.

A third refusal guards the switch to sharding: a node whose unsharded metadata member ran here never starts sharded. A sharded ring is durable and outranks MALACHI_LOG_VNODES once written, while the topics stay described by the unsharded metadata nobody would read again, so the sweep would take their segments. Converting a control plane in place is not supported. The refusal is made twice, each with its own way out: before this node writes a seed (check_seed/3), when nothing was written and removing MALACHI_LOG_VNODES boots it unsharded again; and once the topology is known (check/4), for a sharded ring already recorded (seeded by another node, or by an earlier release that converted control planes in place, or by an earlier release on this node under another cluster name over the same ra directory), which no setting undoes and which only forming the cluster again gets past.

The checks run in steps, because the boot starts the ring store before it can tell whether the control plane is sharded. check_ring/4 runs before the boot touches the ring store, so its refusal leaves nothing behind: had the ring store been formed and the start then refused, a later attempt could find it registered and pass, depending only on whether ra's directory reached the disk before the halt. check_seed/3 runs once the ring store is started but before a seed is written, so its refusal leaves an empty ring store and no seed. check/4 runs once the topology is known, after the ring store was resumed or seeded, for everything that needs the topology.

decide_ring/1, decide_seed/1 and decide/1 are the decisions over plain facts; check_ring/4, check_seed/3 and check/4 gather them and act.

Summary

Types

What decide/1 needs to know once the topology is known, all of it read from this node. member_known? says whether the unsharded metadata member had ever been started here, sharded or not; missing_vnodes names the vnodes of a sharded node configured alone that never were; ring_members is the ring store member's recorded membership, :unknown when it did not answer, or nil when it was not asked; adopted? says the operator's adoption was already taken by check_ring/4 or check_seed/3.

What decide_ring/1 needs to know, read before the boot touches the ring store. recorded_cluster is the control plane name the log directory's cluster marker holds: nil when there is none yet, and :unreadable when the file does not parse.

What decide_seed/1 needs to know, read before the boot writes a sharded seed.

Functions

The last step, once the topology is known (after check_seed/3 when a seed was due): gathers the facts for the control plane cluster over nodes and the log directory dir, decides, and acts: :ok and {:adopt, dirs} return (the second after logging the names), a refusal halts through Malachi.StartupRefusal.refuse!/2.

The first step, before the boot touches the ring store: refuses the start when the cluster marker in dir names another control plane than cluster or does not parse, whatever the operator adopts (a marker that cannot be read at all raises, recorded_cluster/1); then when dir holds segment directories and the ring store was never started on this node (ring_known?, read just before), or logs and answers {:adopt, dirs} when the operator adopted them. When it answers {:adopt, dirs}, pass adopted?: true to check_seed/3 and check/4, so the adoption is not taken twice. It never records the marker.

The step before the boot writes a sharded seed into an empty ring store, so a refusal here leaves no seed behind. A seed forms every vnode now, empty, and outranks the environment from then on, so it refuses the start when the unsharded metadata member of cluster was ever started on this node (that metadata would never be read again) or when dir holds segment directories, which no vnode formed now can know (the ring store is one per ra directory and node name, so a cluster renamed over the same directories gets here). The operator's adoption applies to the segments, not to the unsharded member.

The cluster marker's path in the log directory dir.

The decision over facts

The decision check_ring/4 makes before the boot touches the ring store: a log directory whose cluster marker names another control plane than cluster, or does not parse, refuses, whatever the operator adopts (the ring store is one per ra directory and node name, so it cannot tell the two apart); segment directories on a node whose ring store was never started here refuse ({:adopt, dirs} instead when adopt?).

The decision check_seed/3 makes: a sharded seed over an unsharded member that ran here refuses ({:refuse, :reshard_unsupported}, whatever the operator adopts); over segment directories it refuses them as unknown, or adopts them when adopt?, unless check_ring/4 already did; otherwise :ok.

The control plane name the cluster marker in dir records: nil when there is none, :unreadable when the file does not hold exactly one cluster=<name> line. Any other read error raises, since a marker that cannot be read must not pass as absent.

Whether the ring store was ever started on this node: the fact check_ring/4 needs, read before the boot starts the store and so registers it.

The names at the root of dir that are segment directories (Malachi.Retention.Orphans.segment_directory?/1).

Types

decision()

@type decision() ::
  :ok
  | {:adopt, [String.t()]}
  | {:refuse, {:unknown_segments, [String.t()]}}
  | {:refuse, {:grow_unsupported, [node()]}}
  | {:refuse, :membership_unknown}
  | {:refuse, :reshard_unsupported}
  | {:refuse, :resharded}
  | {:refuse, {:cluster_renamed, String.t() | :unreadable}}

facts()

@type facts() :: %{
  self: node(),
  configured_nodes: [node()],
  sharded?: boolean(),
  member_known?: boolean(),
  missing_vnodes: [atom()],
  ring_members: [node()] | :unknown | nil,
  segment_dirs: [String.t()],
  adopt?: boolean(),
  adopted?: boolean()
}

What decide/1 needs to know once the topology is known, all of it read from this node. member_known? says whether the unsharded metadata member had ever been started here, sharded or not; missing_vnodes names the vnodes of a sharded node configured alone that never were; ring_members is the ring store member's recorded membership, :unknown when it did not answer, or nil when it was not asked; adopted? says the operator's adoption was already taken by check_ring/4 or check_seed/3.

ring_facts()

@type ring_facts() :: %{
  :ring_known? => boolean(),
  :segment_dirs => [String.t()],
  :adopt? => boolean(),
  optional(:cluster) => String.t(),
  optional(:recorded_cluster) => String.t() | :unreadable | nil
}

What decide_ring/1 needs to know, read before the boot touches the ring store. recorded_cluster is the control plane name the log directory's cluster marker holds: nil when there is none yet, and :unreadable when the file does not parse.

seed_facts()

@type seed_facts() :: %{
  member_known?: boolean(),
  segment_dirs: [String.t()],
  adopt?: boolean(),
  adopted?: boolean()
}

What decide_seed/1 needs to know, read before the boot writes a sharded seed.

Functions

check(cluster, nodes, dir, opts \\ [])

@spec check(atom(), [node()], Path.t(), keyword()) :: decision() | term()

The last step, once the topology is known (after check_seed/3 when a seed was due): gathers the facts for the control plane cluster over nodes and the log directory dir, decides, and acts: :ok and {:adopt, dirs} return (the second after logging the names), a refusal halts through Malachi.StartupRefusal.refuse!/2.

The ring store member, started by the boot before this runs, is asked for its recorded membership only when nodes lists other nodes, the one case that needs it; asking resumes it if it is not running, which is idempotent (Malachi.Cluster.RaResume.resume_or/3).

Options

  • :sharded? - whether the control plane is sharded (default false);
  • :vnodes - the sharded placement, [{vnode_id, token, nodes}] (default []): on a node configured alone, every one of them must have been started here before;
  • :adopted? - check_ring/4 or check_seed/3 already took the operator's adoption (default false);
  • :adopt? - accept unknown segment directories (default from :adopt_orphaned_log_dir);
  • :members_deadline_ms - how long the member gets to report its membership, retried (default from :log_ring_boot_timeout_ms, 60000);
  • :members_fun - how one attempt asks it, (server_id, timeout) -> ra members reply (default a local :ra.members/2 query), the seam a test uses;
  • :sync_fun - how the cluster marker's directory is fsynced (default Malachi.Storage.Directory.sync/1), the seam a test uses;
  • :halt_fun - how a refusal halts (default System.halt/1), the seam a test uses.

check_ring(cluster, dir, ring_known?, opts \\ [])

@spec check_ring(atom(), Path.t(), boolean(), keyword()) ::
  :ok | {:adopt, [String.t()]} | term()

The first step, before the boot touches the ring store: refuses the start when the cluster marker in dir names another control plane than cluster or does not parse, whatever the operator adopts (a marker that cannot be read at all raises, recorded_cluster/1); then when dir holds segment directories and the ring store was never started on this node (ring_known?, read just before), or logs and answers {:adopt, dirs} when the operator adopted them. When it answers {:adopt, dirs}, pass adopted?: true to check_seed/3 and check/4, so the adoption is not taken twice. It never records the marker.

Options

  • :adopt? - accept unknown segment directories (default from :adopt_orphaned_log_dir);
  • :halt_fun - how a refusal halts (default System.halt/1), the seam a test uses.

check_seed(cluster, dir, opts \\ [])

@spec check_seed(atom(), Path.t(), keyword()) :: :ok | {:adopt, [String.t()]} | term()

The step before the boot writes a sharded seed into an empty ring store, so a refusal here leaves no seed behind. A seed forms every vnode now, empty, and outranks the environment from then on, so it refuses the start when the unsharded metadata member of cluster was ever started on this node (that metadata would never be read again) or when dir holds segment directories, which no vnode formed now can know (the ring store is one per ra directory and node name, so a cluster renamed over the same directories gets here). The operator's adoption applies to the segments, not to the unsharded member.

Options

  • :adopt? - whether the operator adopted unknown segments (default the :adopt_orphaned_log_dir setting, MALACHI_ADOPT_ORPHANED_LOG_DIR);
  • :adopted? - check_ring/4 already took that adoption (default false), so it is not taken twice;
  • :sync_fun - how the cluster marker's directory is fsynced (default Malachi.Storage.Directory.sync/1), the seam a test uses;
  • :halt_fun - how a refusal halts (default System.halt/1), the seam a test uses.

cluster_marker_path(dir)

@spec cluster_marker_path(Path.t()) :: Path.t()

The cluster marker's path in the log directory dir.

decide(facts)

@spec decide(facts()) :: decision()

The decision over facts:

  • {:refuse, {:unknown_segments, dirs}}: the log directory holds segment directories and the control plane is formed now ({:adopt, dirs} instead when adopt?);
  • {:refuse, {:grow_unsupported, others}}: the ring store was started here as a cluster of one, and the configuration lists others besides this node;
  • {:refuse, :membership_unknown}: that member's membership was needed and it did not answer;
  • {:refuse, :resharded}: a sharded control plane on a node whose unsharded member ran here, whatever is on disk and whatever the operator would adopt;
  • :ok otherwise.

decide_ring(map)

@spec decide_ring(ring_facts()) ::
  :ok
  | {:adopt, [String.t()]}
  | {:refuse, {:unknown_segments, [String.t()]}}
  | {:refuse, {:cluster_renamed, String.t() | :unreadable}}

The decision check_ring/4 makes before the boot touches the ring store: a log directory whose cluster marker names another control plane than cluster, or does not parse, refuses, whatever the operator adopts (the ring store is one per ra directory and node name, so it cannot tell the two apart); segment directories on a node whose ring store was never started here refuse ({:adopt, dirs} instead when adopt?).

decide_seed(map)

@spec decide_seed(seed_facts()) ::
  :ok
  | {:adopt, [String.t()]}
  | {:refuse, :reshard_unsupported}
  | {:refuse, {:unknown_segments, [String.t()]}}

The decision check_seed/3 makes: a sharded seed over an unsharded member that ran here refuses ({:refuse, :reshard_unsupported}, whatever the operator adopts); over segment directories it refuses them as unknown, or adopts them when adopt?, unless check_ring/4 already did; otherwise :ok.

recorded_cluster(dir)

@spec recorded_cluster(Path.t()) :: String.t() | :unreadable | nil

The control plane name the cluster marker in dir records: nil when there is none, :unreadable when the file does not hold exactly one cluster=<name> line. Any other read error raises, since a marker that cannot be read must not pass as absent.

ring_known?()

@spec ring_known?() :: boolean()

Whether the ring store was ever started on this node: the fact check_ring/4 needs, read before the boot starts the store and so registers it.

segment_dirs(dir)

@spec segment_dirs(Path.t()) :: [String.t()]

The names at the root of dir that are segment directories (Malachi.Retention.Orphans.segment_directory?/1).