Malachi. Storage. SegmentStore behaviour
(malachi v0.22.7)
View Source
Behaviour for a pluggable segment storage engine.
NorthGuard's segment storage is explicitly pluggable ("fps-store" is just the primary
implementation). Malachi mirrors that: the durable hot path lives behind this
behaviour so the pure-Elixir implementation (Malachi.Storage.ElixirStore) can later
be swapped for a native one (Direct I/O via a Rust NIF) only if profiling under real
concurrency shows the BEAM file layer to be the bottleneck. See
docs/ARCHITECTURE.md.
Durability contract
append/2 only buffers; records become durable and readable after sync/1, which
must fsync before returning. read/3 serves only committed (synced) records, never
buffered-but-unsynced data, matching NorthGuard's "ack only committed records".
Failure contract
Every callback whose type includes {:error, term()} returns it when a storage operation fails (a
full volume, a failing device, a missing permission), carrying the POSIX reason the file layer
reported ({:error, :enospc}), and never raises for one. verify/3 is the one that reports such a
failure inside its damage map instead, with the POSIX reason as :reason, because to a scrub a copy
that cannot be read is a copy that is damaged here.
A store reports a failure and decides nothing about what it means. Whether the segment is fenced,
retried or sealed belongs to the caller (Malachi.Cluster.ReplicationServer), for the same reason
integrity/1 exists: the caller owns the segment id and the deployment context.
A handle that answered an error is unusable. An operation can fail after changing the file (a write
that landed before its sync failed), and the handle does not describe that state, so it must not be
retried. close/1 stays safe to call on it.
Summary
Callbacks
Buffers records (assigning each a logical offset). Returns the updated handle and the
first/last offsets assigned. Buffered records are durable after sync/1, or sooner if
an implementation flushes automatically on a size threshold.
Closes the segment's file handle. Best-effort: it always answers :ok, including on a handle that
answered an error, because a close that fails has nothing left for its caller to do.
What the last scan of this segment concluded: :ok, or a map describing the first damage found,
with at least :reason, :position and :unreadable_bytes. Only recover/3 scans, so handles
from open/3 and open_read/3 answer :ok.
How many bytes of committed records the segment holds, which is not necessarily how large its file is.
The logical offset the next appended record will receive.
Creates and opens a new, empty segment. Fails if one already exists.
Opens an existing sealed segment read-only, cheaply (no full scan): the committed
:record_count and :base_offset are supplied by the caller (which knows them from
log metadata) and the sparse index is loaded from the persisted sidecar.
Whether the segment has buffered records not yet flushed (an explicit sync/1 is due).
Reads up to max_records committed records starting at logical offset.
Returns :eof if offset is at or beyond the committed end.
Rewrites the segment's index from the segment itself, replacing whatever was there.
Reopens an existing segment, recovering committed state and truncating any partial trailing write.
Flushes, fsyncs, and seals the segment (immutable). Subsequent append/2 must fail.
Whether the segment is sealed (immutable).
Whether an active segment has hit a seal threshold (size or age) at time now_ms.
Flushes buffered records and makes them durable. After this, appended records are committed and
readable. An implementation may skip the sync when nothing is buffered, so callers must not use
this as a barrier for writes made through anything but append/2.
Truncates the segment so that end_offset becomes the offset after its last record, dropping every
record at or above it, and makes the result durable before answering.
Verifies a stored segment's integrity read-only, without opening it for writing and without
ever repairing or truncating it: every frame's checksum is checked and the scan must consume the
file exactly. Used by the background scrub (Malachi.Cluster.Scrubber) to catch corruption at
rest, which is invisible to the byte-size probe when the damage keeps the file's length.
Types
@type handle() :: term()
Opaque handle to an open segment, threaded through calls.
Callbacks
@callback append(handle(), [Malachi.Log.Record.t()]) :: {:ok, handle(), first :: non_neg_integer(), last :: non_neg_integer()} | {:error, term()}
Buffers records (assigning each a logical offset). Returns the updated handle and the
first/last offsets assigned. Buffered records are durable after sync/1, or sooner if
an implementation flushes automatically on a size threshold.
@callback close(handle()) :: :ok
Closes the segment's file handle. Best-effort: it always answers :ok, including on a handle that
answered an error, because a close that fails has nothing left for its caller to do.
What the last scan of this segment concluded: :ok, or a map describing the first damage found,
with at least :reason, :position and :unreadable_bytes. Only recover/3 scans, so handles
from open/3 and open_read/3 answer :ok.
This exists because the recovery scan already knows a segment is damaged, and a store must not
decide what to do about it: the caller (Malachi.Cluster.ReplicationServer) owns the segment id
and the deployment context, so it is the one that logs and emits the telemetry.
@callback logical_bytes(handle()) :: non_neg_integer()
How many bytes of committed records the segment holds, which is not necessarily how large its file is.
An implementation may size a segment's file ahead of its contents (Malachi.Storage.Preallocation),
so an active segment's file can be far larger than the log it carries. Anything that needs the
segment's LENGTH must ask here; File.stat answers a different question and, for a segment being
written, answers it wrongly.
@callback next_offset(handle()) :: non_neg_integer()
The logical offset the next appended record will receive.
@callback open(directory :: Path.t(), segment_id :: term(), opts :: keyword()) :: {:ok, handle()} | {:error, term()}
Creates and opens a new, empty segment. Fails if one already exists.
@callback open_read(directory :: Path.t(), segment_id :: term(), opts :: keyword()) :: {:ok, handle()} | {:error, term()}
Opens an existing sealed segment read-only, cheaply (no full scan): the committed
:record_count and :base_offset are supplied by the caller (which knows them from
log metadata) and the sparse index is loaded from the persisted sidecar.
Whether the segment has buffered records not yet flushed (an explicit sync/1 is due).
@callback read(handle(), offset :: non_neg_integer(), max_records :: pos_integer()) :: {:ok, [Malachi.Log.Record.t()]} | :eof | {:error, term()}
Reads up to max_records committed records starting at logical offset.
Returns :eof if offset is at or beyond the committed end.
@callback rebuild_index(directory :: Path.t(), segment_id :: term(), opts :: keyword()) :: :ok | {:error, term()}
Rewrites the segment's index from the segment itself, replacing whatever was there.
This is the repair for a damaged index, and it needs no replica: the index is derived from the records, so a node can always rebuild its own from the segment it already holds, unlike a damaged segment, which has to be refetched from a peer. Only call it on a segment whose records verify, or the rebuilt index will faithfully describe the damage.
@callback recover(directory :: Path.t(), segment_id :: term(), opts :: keyword()) :: {:ok, handle()} | {:error, term()}
Reopens an existing segment, recovering committed state and truncating any partial trailing write.
Flushes, fsyncs, and seals the segment (immutable). Subsequent append/2 must fail.
Whether the segment is sealed (immutable).
@callback should_seal?(handle(), now_ms :: non_neg_integer()) :: boolean()
Whether an active segment has hit a seal threshold (size or age) at time now_ms.
Flushes buffered records and makes them durable. After this, appended records are committed and
readable. An implementation may skip the sync when nothing is buffered, so callers must not use
this as a barrier for writes made through anything but append/2.
@callback truncate_to(handle(), end_offset :: non_neg_integer()) :: {:ok, handle()} | {:error, term()}
Truncates the segment so that end_offset becomes the offset after its last record, dropping every
record at or above it, and makes the result durable before answering.
The only callback that destroys committed bytes on purpose, and the only one allowed on a SEALED
segment. It exists because a copy can recover holding records past the length the control plane
recorded for its segment, and a seal that does not reach the bytes is a seal in name only
(Malachi.Cluster.SealedOverrun). A sealed segment is immutable in the sense that matters, which is
that no new offset is ever issued inside it; bringing a copy DOWN to the recorded length issues
nothing and is what makes the record of the seal true.
Three refusals, none of them optional:
- an
end_offsetbelow the segment's base offset answers{:error, :below_base_offset}, since a segment cannot hold fewer than zero records and the caller has asked about the wrong segment; - a handle with records still buffered answers
{:error, :pending_records}, because they describe bytes the file does not have yet and truncating around them is meaningless; - a handle that refuses appends refuses this too, with the same reason, because the bytes it chose to preserve are exactly the ones a truncation would rewrite.
An end_offset at or above the segment's own end is a no-op that answers {:ok, handle}: a copy
that is short of the recorded length is a different defect, repaired by refetching from a peer.
@callback verify(directory :: Path.t(), segment_id :: term(), opts :: keyword()) :: {:ok, %{records: non_neg_integer(), bytes: non_neg_integer()}} | {:error, %{position: non_neg_integer(), reason: atom(), file: Path.t()} | :enoent}
Verifies a stored segment's integrity read-only, without opening it for writing and without
ever repairing or truncating it: every frame's checksum is checked and the scan must consume the
file exactly. Used by the background scrub (Malachi.Cluster.Scrubber) to catch corruption at
rest, which is invisible to the byte-size probe when the damage keeps the file's length.
Returns {:ok, %{records: n, bytes: b}} for an intact segment, {:error, %{position: byte, reason: reason, file: path}} for the first damaged frame, or {:error, :enoent} when the
segment is not stored here (a segment deleted by retention mid-scan is not a failure).