Malachi.Storage.SegmentStore behaviour (malachi v0.22.7)

View Source

Behaviour for a pluggable segment storage engine.

NorthGuard's segment storage is explicitly pluggable ("fps-store" is just the primary implementation). Malachi mirrors that: the durable hot path lives behind this behaviour so the pure-Elixir implementation (Malachi.Storage.ElixirStore) can later be swapped for a native one (Direct I/O via a Rust NIF) only if profiling under real concurrency shows the BEAM file layer to be the bottleneck. See docs/ARCHITECTURE.md.

Durability contract

append/2 only buffers; records become durable and readable after sync/1, which must fsync before returning. read/3 serves only committed (synced) records, never buffered-but-unsynced data, matching NorthGuard's "ack only committed records".

Failure contract

Every callback whose type includes {:error, term()} returns it when a storage operation fails (a full volume, a failing device, a missing permission), carrying the POSIX reason the file layer reported ({:error, :enospc}), and never raises for one. verify/3 is the one that reports such a failure inside its damage map instead, with the POSIX reason as :reason, because to a scrub a copy that cannot be read is a copy that is damaged here.

A store reports a failure and decides nothing about what it means. Whether the segment is fenced, retried or sealed belongs to the caller (Malachi.Cluster.ReplicationServer), for the same reason integrity/1 exists: the caller owns the segment id and the deployment context.

A handle that answered an error is unusable. An operation can fail after changing the file (a write that landed before its sync failed), and the handle does not describe that state, so it must not be retried. close/1 stays safe to call on it.

Summary

Types

Opaque handle to an open segment, threaded through calls.

Callbacks

Buffers records (assigning each a logical offset). Returns the updated handle and the first/last offsets assigned. Buffered records are durable after sync/1, or sooner if an implementation flushes automatically on a size threshold.

Closes the segment's file handle. Best-effort: it always answers :ok, including on a handle that answered an error, because a close that fails has nothing left for its caller to do.

What the last scan of this segment concluded: :ok, or a map describing the first damage found, with at least :reason, :position and :unreadable_bytes. Only recover/3 scans, so handles from open/3 and open_read/3 answer :ok.

How many bytes of committed records the segment holds, which is not necessarily how large its file is.

The logical offset the next appended record will receive.

Creates and opens a new, empty segment. Fails if one already exists.

Opens an existing sealed segment read-only, cheaply (no full scan): the committed :record_count and :base_offset are supplied by the caller (which knows them from log metadata) and the sparse index is loaded from the persisted sidecar.

Whether the segment has buffered records not yet flushed (an explicit sync/1 is due).

Reads up to max_records committed records starting at logical offset. Returns :eof if offset is at or beyond the committed end.

Rewrites the segment's index from the segment itself, replacing whatever was there.

Reopens an existing segment, recovering committed state and truncating any partial trailing write.

Flushes, fsyncs, and seals the segment (immutable). Subsequent append/2 must fail.

Whether the segment is sealed (immutable).

Whether an active segment has hit a seal threshold (size or age) at time now_ms.

Flushes buffered records and makes them durable. After this, appended records are committed and readable. An implementation may skip the sync when nothing is buffered, so callers must not use this as a barrier for writes made through anything but append/2.

Truncates the segment so that end_offset becomes the offset after its last record, dropping every record at or above it, and makes the result durable before answering.

Verifies a stored segment's integrity read-only, without opening it for writing and without ever repairing or truncating it: every frame's checksum is checked and the scan must consume the file exactly. Used by the background scrub (Malachi.Cluster.Scrubber) to catch corruption at rest, which is invisible to the byte-size probe when the damage keeps the file's length.

Types

handle()

@type handle() :: term()

Opaque handle to an open segment, threaded through calls.

Callbacks

append(handle, list)

@callback append(handle(), [Malachi.Log.Record.t()]) ::
  {:ok, handle(), first :: non_neg_integer(), last :: non_neg_integer()}
  | {:error, term()}

Buffers records (assigning each a logical offset). Returns the updated handle and the first/last offsets assigned. Buffered records are durable after sync/1, or sooner if an implementation flushes automatically on a size threshold.

close(handle)

@callback close(handle()) :: :ok

Closes the segment's file handle. Best-effort: it always answers :ok, including on a handle that answered an error, because a close that fails has nothing left for its caller to do.

integrity(handle)

@callback integrity(handle()) :: :ok | map()

What the last scan of this segment concluded: :ok, or a map describing the first damage found, with at least :reason, :position and :unreadable_bytes. Only recover/3 scans, so handles from open/3 and open_read/3 answer :ok.

This exists because the recovery scan already knows a segment is damaged, and a store must not decide what to do about it: the caller (Malachi.Cluster.ReplicationServer) owns the segment id and the deployment context, so it is the one that logs and emits the telemetry.

logical_bytes(handle)

@callback logical_bytes(handle()) :: non_neg_integer()

How many bytes of committed records the segment holds, which is not necessarily how large its file is.

An implementation may size a segment's file ahead of its contents (Malachi.Storage.Preallocation), so an active segment's file can be far larger than the log it carries. Anything that needs the segment's LENGTH must ask here; File.stat answers a different question and, for a segment being written, answers it wrongly.

next_offset(handle)

@callback next_offset(handle()) :: non_neg_integer()

The logical offset the next appended record will receive.

open(directory, segment_id, opts)

@callback open(directory :: Path.t(), segment_id :: term(), opts :: keyword()) ::
  {:ok, handle()} | {:error, term()}

Creates and opens a new, empty segment. Fails if one already exists.

open_read(directory, segment_id, opts)

@callback open_read(directory :: Path.t(), segment_id :: term(), opts :: keyword()) ::
  {:ok, handle()} | {:error, term()}

Opens an existing sealed segment read-only, cheaply (no full scan): the committed :record_count and :base_offset are supplied by the caller (which knows them from log metadata) and the sparse index is loaded from the persisted sidecar.

pending?(handle)

@callback pending?(handle()) :: boolean()

Whether the segment has buffered records not yet flushed (an explicit sync/1 is due).

read(handle, offset, max_records)

@callback read(handle(), offset :: non_neg_integer(), max_records :: pos_integer()) ::
  {:ok, [Malachi.Log.Record.t()]} | :eof | {:error, term()}

Reads up to max_records committed records starting at logical offset. Returns :eof if offset is at or beyond the committed end.

rebuild_index(directory, segment_id, opts)

@callback rebuild_index(directory :: Path.t(), segment_id :: term(), opts :: keyword()) ::
  :ok | {:error, term()}

Rewrites the segment's index from the segment itself, replacing whatever was there.

This is the repair for a damaged index, and it needs no replica: the index is derived from the records, so a node can always rebuild its own from the segment it already holds, unlike a damaged segment, which has to be refetched from a peer. Only call it on a segment whose records verify, or the rebuilt index will faithfully describe the damage.

recover(directory, segment_id, opts)

@callback recover(directory :: Path.t(), segment_id :: term(), opts :: keyword()) ::
  {:ok, handle()} | {:error, term()}

Reopens an existing segment, recovering committed state and truncating any partial trailing write.

seal(handle)

@callback seal(handle()) :: {:ok, handle()} | {:error, term()}

Flushes, fsyncs, and seals the segment (immutable). Subsequent append/2 must fail.

sealed?(handle)

@callback sealed?(handle()) :: boolean()

Whether the segment is sealed (immutable).

should_seal?(handle, now_ms)

@callback should_seal?(handle(), now_ms :: non_neg_integer()) :: boolean()

Whether an active segment has hit a seal threshold (size or age) at time now_ms.

sync(handle)

@callback sync(handle()) :: {:ok, handle()} | {:error, term()}

Flushes buffered records and makes them durable. After this, appended records are committed and readable. An implementation may skip the sync when nothing is buffered, so callers must not use this as a barrier for writes made through anything but append/2.

truncate_to(handle, end_offset)

@callback truncate_to(handle(), end_offset :: non_neg_integer()) ::
  {:ok, handle()} | {:error, term()}

Truncates the segment so that end_offset becomes the offset after its last record, dropping every record at or above it, and makes the result durable before answering.

The only callback that destroys committed bytes on purpose, and the only one allowed on a SEALED segment. It exists because a copy can recover holding records past the length the control plane recorded for its segment, and a seal that does not reach the bytes is a seal in name only (Malachi.Cluster.SealedOverrun). A sealed segment is immutable in the sense that matters, which is that no new offset is ever issued inside it; bringing a copy DOWN to the recorded length issues nothing and is what makes the record of the seal true.

Three refusals, none of them optional:

  • an end_offset below the segment's base offset answers {:error, :below_base_offset}, since a segment cannot hold fewer than zero records and the caller has asked about the wrong segment;
  • a handle with records still buffered answers {:error, :pending_records}, because they describe bytes the file does not have yet and truncating around them is meaningless;
  • a handle that refuses appends refuses this too, with the same reason, because the bytes it chose to preserve are exactly the ones a truncation would rewrite.

An end_offset at or above the segment's own end is a no-op that answers {:ok, handle}: a copy that is short of the recorded length is a different defect, repaired by refetching from a peer.

verify(directory, segment_id, opts)

@callback verify(directory :: Path.t(), segment_id :: term(), opts :: keyword()) ::
  {:ok, %{records: non_neg_integer(), bytes: non_neg_integer()}}
  | {:error,
     %{position: non_neg_integer(), reason: atom(), file: Path.t()} | :enoent}

Verifies a stored segment's integrity read-only, without opening it for writing and without ever repairing or truncating it: every frame's checksum is checked and the scan must consume the file exactly. Used by the background scrub (Malachi.Cluster.Scrubber) to catch corruption at rest, which is invisible to the byte-size probe when the damage keeps the file's length.

Returns {:ok, %{records: n, bytes: b}} for an intact segment, {:error, %{position: byte, reason: reason, file: path}} for the first damaged frame, or {:error, :enoent} when the segment is not stored here (a segment deleted by retention mid-scan is not a failure).