Malachi. Storage. DataDirGuard
(malachi v0.22.7)
View Source
The boot check that a log data directory and the control plane this node is about to start describe the same data.
ra keeps a node's control plane under the node's name: its log lives under
MALACHI_RA_DATA_DIR/<node>, and every member is recorded as {cluster, node()}. What a node
acknowledged survives a restart only while the segments on disk and that control plane come back
together, and when they do not, the orphan sweep used to settle it by deleting the segments (#273).
Two things are checked before the broker starts, after a first one on the directory's own record:
a log directory of another control plane. A start that passes records the control plane's name at the root of the log directory (
malachi.cluster), just before it writes a seed or once every check has passed, and a later start under another name refuses, before the ring store is touched. A start refused before its seed is written records nothing; one that fails after the seed check passed (the seed write, or a later membership check) keeps the name it seeded under, since the seed's vnodes carry that name and whether the write landed may not be known. Every other check readsraby node name, and the ring store is one perradirectory and node name whatever the cluster is called, so a renamed control plane over the same directories would read the old one's state as its own; a member that held no segment would pass every other check and could seed the ring its peers then trust.segments nobody knows. The log directory already holds segment directories, and the control plane that would describe them is about to be formed rather than resumed: the ring store (
Malachi.LogRing, which every clustered node runs, sharded or not) was never started on this node; or, for an unsharded control plane, its metadata member was never started here; or a sharded ring is about to be seeded, forming every vnode now (check_seed/3); or, for a sharded node configured alone, one of its vnodes was never started here. A cluster is not held to that, since a rebalance moves vnodes off a node without republishing the ring, and a vnode with other replicas keeps its history on them; a vnode whose only replica is this node (MALACHI_LOG_VNODE_REPLICATION_FACTOR=1on a cluster) has no such protection. The node name changed (a container recreated without a fixed hostname,iex --namewith another name), theradirectory was lost, the cluster was renamed, or the segments were written by a release that kept a single node's metadata in memory. On a single node a control plane formed now is empty, every one of those directories would read as an orphan, and a topic created again under an old name would read the old bytes as its own. A cluster member is refused too, though its peers may still hold the control plane: it would come back an amnesiac voter, and adopting is then the operator's way back, the sweep removing only what no owner lists. A node that joins a cluster for the first time has nothing on disk to refuse.a single node told it has peers. The control plane was started here as a cluster of one, and the configuration now lists other nodes. Growing a one-member cluster in place is not supported: this node would resume its ring store (and its metadata, sharded or not) alone while the others formed clusters of their own over the same names. The ring store member, which every clustered node runs and which was formed over the same nodes as the metadata, is asked for its recorded membership; a member that does not answer within the ring boot timeout refuses the start too, since a check that cannot be made must not read as passed.
Each refuses the start through Malachi.StartupRefusal (exit 78) with a line that names the causes
and the way out. Segments nobody knows can be adopted with MALACHI_ADOPT_ORPHANED_LOG_DIR=true, which is the
operator saying the directories are not wanted: the start goes on, the adopted names are logged, and
the orphan sweep removes them on its own schedule. The others cannot.
A third refusal guards the switch to sharding: a node whose unsharded metadata member ran here never
starts sharded. A sharded ring is durable and outranks MALACHI_LOG_VNODES once written, while the
topics stay described by the unsharded metadata nobody would read again, so the sweep would take their
segments. Converting a control plane in place is not supported. The refusal is made twice, each with
its own way out: before this node writes a seed (check_seed/3), when nothing was written and removing
MALACHI_LOG_VNODES boots it unsharded again; and once the topology is known (check/4), for a
sharded ring already recorded (seeded by another node, or by an earlier release that converted control
planes in place, or by an earlier release on this node under another cluster name over the same ra
directory), which no setting undoes and which only forming the cluster again gets past.
The checks run in steps, because the boot starts the ring store before it can tell whether the
control plane is sharded. check_ring/4 runs before the boot touches the ring store, so its refusal
leaves nothing behind: had the ring store been formed and the start then refused, a later attempt
could find it registered and pass, depending only on whether ra's directory reached the disk before
the halt. check_seed/3 runs once the ring store is started but before a seed is written, so its
refusal leaves an empty ring store and no seed. check/4 runs once the topology is known, after the
ring store was resumed or seeded, for everything that needs the topology.
decide_ring/1, decide_seed/1 and decide/1 are the decisions over plain facts; check_ring/4,
check_seed/3 and check/4 gather them and act.
Summary
Types
What decide/1 needs to know once the topology is known, all of it read from this node.
member_known? says whether the unsharded metadata member had ever been started here, sharded or not;
missing_vnodes
names the vnodes of a sharded node configured alone that never were; ring_members is the ring store
member's recorded membership, :unknown when it did not answer, or nil when it was not asked;
adopted? says the operator's adoption was already taken by check_ring/4 or check_seed/3.
What decide_ring/1 needs to know, read before the boot touches the ring store. recorded_cluster is
the control plane name the log directory's cluster marker holds: nil when there is none yet, and
:unreadable when the file does not parse.
What decide_seed/1 needs to know, read before the boot writes a sharded seed.
Functions
The last step, once the topology is known (after check_seed/3 when a seed was due): gathers the
facts for the control plane cluster over nodes and the log directory dir, decides, and acts:
:ok and {:adopt, dirs} return (the second after logging the names), a refusal halts through
Malachi.StartupRefusal.refuse!/2.
The first step, before the boot touches the ring store: refuses the start when the cluster marker in
dir names another control plane than cluster or does not parse, whatever the operator adopts (a
marker that cannot be read at all raises, recorded_cluster/1);
then when dir holds segment directories and the ring store was never started on this node
(ring_known?, read just before), or logs and answers {:adopt, dirs} when the operator adopted
them. When it answers {:adopt, dirs}, pass adopted?: true to check_seed/3 and check/4, so the
adoption is not taken twice. It never records the marker.
The step before the boot writes a sharded seed into an empty ring store, so a refusal here leaves no
seed behind. A seed forms every vnode now, empty, and outranks the environment from then on, so it
refuses the start when the unsharded metadata member of cluster was ever started on this node (that
metadata would never be read again) or when dir holds segment directories, which no vnode formed now
can know (the ring store is one per ra directory and node name, so a cluster renamed over the same
directories gets here). The operator's adoption applies to the segments, not to the unsharded member.
The cluster marker's path in the log directory dir.
The decision over facts
The decision check_ring/4 makes before the boot touches the ring store: a log directory whose
cluster marker names another control plane than cluster, or does not parse, refuses, whatever the
operator adopts (the
ring store is one per ra directory and node name, so it cannot tell the two apart); segment
directories on a node whose ring store was never started here refuse ({:adopt, dirs} instead when
adopt?).
The decision check_seed/3 makes: a sharded seed over an unsharded member that ran here refuses
({:refuse, :reshard_unsupported}, whatever the operator adopts); over segment directories it refuses
them as unknown, or adopts them when adopt?, unless check_ring/4 already did; otherwise :ok.
The control plane name the cluster marker in dir records: nil when there is none, :unreadable
when the file does not hold exactly one cluster=<name> line. Any other read error raises, since a
marker that cannot be read must not pass as absent.
Whether the ring store was ever started on this node: the fact check_ring/4 needs, read before the
boot starts the store and so registers it.
The names at the root of dir that are segment directories (Malachi.Retention.Orphans.segment_directory?/1).
Types
@type facts() :: %{ self: node(), configured_nodes: [node()], sharded?: boolean(), member_known?: boolean(), missing_vnodes: [atom()], ring_members: [node()] | :unknown | nil, segment_dirs: [String.t()], adopt?: boolean(), adopted?: boolean() }
What decide/1 needs to know once the topology is known, all of it read from this node.
member_known? says whether the unsharded metadata member had ever been started here, sharded or not;
missing_vnodes
names the vnodes of a sharded node configured alone that never were; ring_members is the ring store
member's recorded membership, :unknown when it did not answer, or nil when it was not asked;
adopted? says the operator's adoption was already taken by check_ring/4 or check_seed/3.
@type ring_facts() :: %{ :ring_known? => boolean(), :segment_dirs => [String.t()], :adopt? => boolean(), optional(:cluster) => String.t(), optional(:recorded_cluster) => String.t() | :unreadable | nil }
What decide_ring/1 needs to know, read before the boot touches the ring store. recorded_cluster is
the control plane name the log directory's cluster marker holds: nil when there is none yet, and
:unreadable when the file does not parse.
@type seed_facts() :: %{ member_known?: boolean(), segment_dirs: [String.t()], adopt?: boolean(), adopted?: boolean() }
What decide_seed/1 needs to know, read before the boot writes a sharded seed.
Functions
The last step, once the topology is known (after check_seed/3 when a seed was due): gathers the
facts for the control plane cluster over nodes and the log directory dir, decides, and acts:
:ok and {:adopt, dirs} return (the second after logging the names), a refusal halts through
Malachi.StartupRefusal.refuse!/2.
The ring store member, started by the boot before this runs, is asked for its recorded membership only
when nodes lists other nodes, the one case that needs it; asking resumes it if it is not running,
which is idempotent (Malachi.Cluster.RaResume.resume_or/3).
Options
:sharded?- whether the control plane is sharded (defaultfalse);:vnodes- the sharded placement,[{vnode_id, token, nodes}](default[]): on a node configured alone, every one of them must have been started here before;:adopted?-check_ring/4orcheck_seed/3already took the operator's adoption (defaultfalse);:adopt?- accept unknown segment directories (default from:adopt_orphaned_log_dir);:members_deadline_ms- how long the member gets to report its membership, retried (default from:log_ring_boot_timeout_ms, 60000);:members_fun- how one attempt asks it,(server_id, timeout) -> ra members reply(default a local:ra.members/2query), the seam a test uses;:sync_fun- how the cluster marker's directory is fsynced (defaultMalachi.Storage.Directory.sync/1), the seam a test uses;:halt_fun- how a refusal halts (defaultSystem.halt/1), the seam a test uses.
The first step, before the boot touches the ring store: refuses the start when the cluster marker in
dir names another control plane than cluster or does not parse, whatever the operator adopts (a
marker that cannot be read at all raises, recorded_cluster/1);
then when dir holds segment directories and the ring store was never started on this node
(ring_known?, read just before), or logs and answers {:adopt, dirs} when the operator adopted
them. When it answers {:adopt, dirs}, pass adopted?: true to check_seed/3 and check/4, so the
adoption is not taken twice. It never records the marker.
Options
:adopt?- accept unknown segment directories (default from:adopt_orphaned_log_dir);:halt_fun- how a refusal halts (defaultSystem.halt/1), the seam a test uses.
The step before the boot writes a sharded seed into an empty ring store, so a refusal here leaves no
seed behind. A seed forms every vnode now, empty, and outranks the environment from then on, so it
refuses the start when the unsharded metadata member of cluster was ever started on this node (that
metadata would never be read again) or when dir holds segment directories, which no vnode formed now
can know (the ring store is one per ra directory and node name, so a cluster renamed over the same
directories gets here). The operator's adoption applies to the segments, not to the unsharded member.
Options
:adopt?- whether the operator adopted unknown segments (default the:adopt_orphaned_log_dirsetting,MALACHI_ADOPT_ORPHANED_LOG_DIR);:adopted?-check_ring/4already took that adoption (default false), so it is not taken twice;:sync_fun- how the cluster marker's directory is fsynced (defaultMalachi.Storage.Directory.sync/1), the seam a test uses;:halt_fun- how a refusal halts (defaultSystem.halt/1), the seam a test uses.
The cluster marker's path in the log directory dir.
The decision over facts:
{:refuse, {:unknown_segments, dirs}}: the log directory holds segment directories and the control plane is formed now ({:adopt, dirs}instead whenadopt?);{:refuse, {:grow_unsupported, others}}: the ring store was started here as a cluster of one, and the configuration listsothersbesides this node;{:refuse, :membership_unknown}: that member's membership was needed and it did not answer;{:refuse, :resharded}: a sharded control plane on a node whose unsharded member ran here, whatever is on disk and whatever the operator would adopt;:okotherwise.
@spec decide_ring(ring_facts()) :: :ok | {:adopt, [String.t()]} | {:refuse, {:unknown_segments, [String.t()]}} | {:refuse, {:cluster_renamed, String.t() | :unreadable}}
The decision check_ring/4 makes before the boot touches the ring store: a log directory whose
cluster marker names another control plane than cluster, or does not parse, refuses, whatever the
operator adopts (the
ring store is one per ra directory and node name, so it cannot tell the two apart); segment
directories on a node whose ring store was never started here refuse ({:adopt, dirs} instead when
adopt?).
@spec decide_seed(seed_facts()) :: :ok | {:adopt, [String.t()]} | {:refuse, :reshard_unsupported} | {:refuse, {:unknown_segments, [String.t()]}}
The decision check_seed/3 makes: a sharded seed over an unsharded member that ran here refuses
({:refuse, :reshard_unsupported}, whatever the operator adopts); over segment directories it refuses
them as unknown, or adopts them when adopt?, unless check_ring/4 already did; otherwise :ok.
The control plane name the cluster marker in dir records: nil when there is none, :unreadable
when the file does not hold exactly one cluster=<name> line. Any other read error raises, since a
marker that cannot be read must not pass as absent.
@spec ring_known?() :: boolean()
Whether the ring store was ever started on this node: the fact check_ring/4 needs, read before the
boot starts the store and so registers it.
The names at the root of dir that are segment directories (Malachi.Retention.Orphans.segment_directory?/1).