Malachi.
View Source
The decision half of the orphan sweep: which directories under a data directory belong to no segment the control plane still lists, and which of those have been unexplained for long enough to remove.
Pure, so the one rule that matters can be stated as a property: a directory the owning vnodes report as known is never selected, whatever the sequence of operations that produced it.
Which segments a name can belong to
Malachi.Storage.Layout.segment_directory/2 maps a segment id to a directory name, and the readable
form is ambiguous: a topic may itself contain -r0-s0, so a-r0-s0-r1-s2 is either segment 2 of
range 1 of topic a-r0-s0, or nothing else the layout could have written. candidate_ids/1 does not
pick one reading: it returns every segment id the layout would turn into exactly that name, each one
witnessed by the round trip, and the sweep asks the owner of each. A name is explained when ANY of
its readings is a segment its owner knows, so an ambiguous name can only be kept by mistake, never
removed by one.
The owner is found from the id itself, through Malachi.Metadata.segment_routing_topic/1, the same
routing every segment command takes (Malachi.Cluster.ReplicatedDSRSM.known_segments/3). That is
what lets the sweep ask the vnode that can answer instead of trusting a copy of every vnode's
metadata. An id its owner does not list is then asked of every vnode, which can only keep it: a
vnode can hold metadata outside its arc (docs/ARCHITECTURE.md).
The three guards, and what each one is for
- Age. A replica creates its directory on the first
follow/4, after its segment's registration has committed on the owning vnode. The minimum age is kept anyway, for the directories no registration explains in the first place: a catch-up or a heal that lands after retention deleted the segment, which is a real orphan and can wait. - Sightings. The same directory has to be unexplained on N consecutive passes an interval apart. An answer that was briefly wrong explains a directory again on the next pass and the count restarts, because the count only survives while the directory keeps being a candidate.
- A cap per pass. A guard against this module rather than against the cluster: if the known set ever came out wrong, the operator loses a bounded number of directories and sees the counter move, instead of losing the node's data between two ticks.
Reserved names are excluded before any of that: the data directory's own format marker and, on a sharded single-node data plane, the per-shard subdirectories, neither of which is a segment.
Only a name the layout could have written
Before the guards, a candidate has to be a name Malachi.Storage.Layout.segment_directory/2 can
actually emit, which is exactly a name candidate_ids/1 finds at least one reading for.
It cannot hide an orphan, because every directory the replication path creates was named by that
function. What it does is keep the sweep off everything else that can sit under a data directory an
operator chose: a ra data directory nested inside it, a lost+found, a copy taken by hand before an
upgrade. Removal is an rm_rf and there is nothing to undo it with.
Its one failure mode is the safe one. If term_to_binary ever encodes the same term differently, an
older Base64 directory stops round-tripping and is never swept, which leaks disk instead of losing it.
Summary
Types
A directory found under the data directory: its name and how long ago it was created.
Which of the segment ids it is given the control plane lists (known), which it could not route to
an owner (unroutable), which a pending split is moving and neither owner listed (migrating), and
which no owner lists but another vnode does (misplaced). The shape
Malachi.Cluster.ReplicatedDSRSM.known_segments/3 returns.
One pass's decision: the directories to remove now, the ones still short of a guard, the sightings to carry into the next pass, and whether tracking hit its cap (which delays a removal, never causes one).
How many consecutive passes each candidate directory has been unexplained for.
Functions
Every segment id Malachi.Storage.Layout.segment_directory/2 would turn into exactly name, and
nothing else: each reading is kept only when it encodes back to name.
The names in entries worth asking the control plane about (old enough, not reserved, with at least
one reading, oldest first and at most :max_tracked of them) and how many eligible names the cap
left out.
Which of names the control plane accounts for, asking lookup about every reading of every name.
The ids among ids that segments (one vnode's segment map) lists. Runs in the caller, over the map
Malachi.Cluster.MetadataServer.segments/2 returned, never inside the vnode's leader.
One pass's decision over the unexplained names, given the sightings carried from the previous pass.
Whether name, an entry at the root of a log data directory, is a directory the storage layout could
have written for a segment: not one of the reserved names beside them (the format marker, a shard
directory) and with at least one reading. The one rule for what a segment directory is, shared by the
sweep and by the boot check that refuses to start over segments nobody knows
(Malachi.Storage.DataDirGuard).
Types
@type entry() :: {name :: String.t(), age_ms :: non_neg_integer()}
A directory found under the data directory: its name and how long ago it was created.
@type lookup() :: ([Malachi.Metadata.segment_id()] -> {:ok, %{ known: MapSet.t(), unroutable: [Malachi.Metadata.segment_id()], migrating: [Malachi.Metadata.segment_id()], misplaced: [Malachi.Metadata.segment_id()] }} | {:error, term()})
Which of the segment ids it is given the control plane lists (known), which it could not route to
an owner (unroutable), which a pending split is moving and neither owner listed (migrating), and
which no owner lists but another vnode does (misplaced). The shape
Malachi.Cluster.ReplicatedDSRSM.known_segments/3 returns.
@type review() :: %{ ready: [String.t()], held: [String.t()], sightings: sightings(), capped?: boolean() }
One pass's decision: the directories to remove now, the ones still short of a guard, the sightings to carry into the next pass, and whether tracking hit its cap (which delays a removal, never causes one).
@type sightings() :: %{required(String.t()) => pos_integer()}
How many consecutive passes each candidate directory has been unexplained for.
Functions
@spec candidate_ids(String.t()) :: [Malachi.Metadata.segment_id()]
Every segment id Malachi.Storage.Layout.segment_directory/2 would turn into exactly name, and
nothing else: each reading is kept only when it encodes back to name.
Empty for a name the layout could not have written. At most one readable reading (the tail from the
last -r), plus possibly one more when the same characters also decode as Base64.
@spec candidates( [entry()], keyword() ) :: {[String.t()], non_neg_integer()}
The names in entries worth asking the control plane about (old enough, not reserved, with at least
one reading, oldest first and at most :max_tracked of them) and how many eligible names the cap
left out.
The cap bounds the work of one pass on this node on a disk full of leftovers, and with it how many
owners get asked. It does not bound what an owner sends back: each one answers with its whole segment
map (Malachi.Cluster.MetadataServer.segments/2). A name left out is asked about on a later pass,
once the older ones are gone, and the count is what lets the caller say so.
Options
:min_age_ms- how old a directory must be before it can be a candidate (required);:max_tracked- most names to return (required).
@spec explain([String.t()], lookup()) :: {:ok, %{known: MapSet.t(String.t()), undecided: MapSet.t(String.t())}} | {:error, term()}
Which of names the control plane accounts for, asking lookup about every reading of every name.
lookup takes segment ids and answers as lookup/0 describes, or an error. A name with a known
reading is known. Otherwise a name with a reading that is unroutable, migrating or misplaced is
undecided: the caller must keep it this pass, since nobody could say, the split moving it had not
settled where it lives, or it lives on a vnode that does not own it. Every other name was asked about and is unexplained. An error from lookup is
returned as-is: a partial answer is not an answer.
@spec known_among(%{required(Malachi.Metadata.segment_id()) => term()}, [ Malachi.Metadata.segment_id() ]) :: MapSet.t()
The ids among ids that segments (one vnode's segment map) lists. Runs in the caller, over the map
Malachi.Cluster.MetadataServer.segments/2 returned, never inside the vnode's leader.
One pass's decision over the unexplained names, given the sightings carried from the previous pass.
unexplained must be names that were ASKED about and that no owner reported as known: the ones
candidates/2 returned, minus the known ones. A name that was never asked about is not unexplained,
it is unknown, and passing it here would count a sighting nobody made.
Options
:sightings- consecutive passes a candidate must survive before removal (required);:max_per_pass- most directories to remove in one pass (required);:max_tracked- most candidates to carry sightings for (required).
Whether name, an entry at the root of a log data directory, is a directory the storage layout could
have written for a segment: not one of the reserved names beside them (the format marker, a shard
directory) and with at least one reading. The one rule for what a segment directory is, shared by the
sweep and by the boot check that refuses to start over segments nobody knows
(Malachi.Storage.DataDirGuard).