The log model

View Source

Malachi is a log, not a queue. This guide explains the four concepts a client touches, topic, key, cursor, consumer group, and the three the server manages underneath - range, segment, replica set.

Analogy. A log is an append-only ledger: you only ever write at the end, and you never edit or delete what is already there. A queue hands an item out and forgets it; a log keeps everything and lets many readers move through at their own pace.

What a client sees

A client deals in three things and nothing else:

  • topic: a named, ordered, replicated log.
  • key: on produce, routes each record to a range of the topic's keyspace. Ordering is guaranteed per key, not globally.
  • opaque cursor: on consume, a position token you echo back to continue.

That is the entire client-facing model. There is no partition count to choose, no offset arithmetic, no rebalance protocol to implement.

flowchart LR
  K["record key (e.g. user-42)"] --> H["hash(key)"]
  H --> P["a point in the keyspace [0, 2^bits)"]
  P --> R["exactly one range (one shelf)"]

Analogy. The key is like a name you file under: the same name always lands on the same shelf, so everything for one key stays in order. You never choose the shelf; the hash does.

Why the cursor is opaque

Internally a cursor encodes per-range positions. It is deliberately opaque so the broker can split, merge and restripe ranges while the cluster is running without breaking clients.

This is the core departure from Kafka, which exposes partitions and offsets to clients. Once a client knows "partition 7, offset 12345", the partition count is frozen into your clients' assumptions, resharding becomes a migration event. Here the equivalent change is a server-side operation the client never notices.

flowchart TD
  subgraph offset["Exposed offset (brittle)"]
    O1["client remembers 'partition 7, offset 12345'"] --> O2["a reshard changes the partition count"] --> O3["the client's assumption breaks"]
  end
  subgraph cursor["Opaque cursor (durable)"]
    C1["client holds a token (a coat-check ticket)"] --> C2["the broker rearranges ranges underneath"] --> C3["the same token still resolves"]
  end

Analogy. An opaque cursor is a coat-check ticket. You hold the ticket, not the coat's rack position, so the cloakroom can rearrange the racks and your ticket still finds your coat. An exposed offset is like memorizing "row 3, hook 5": the moment they rearrange, your note is wrong.

What the server manages

flowchart TD
  T["Topic (the whole library)"] --> R["Range (one shelf: a band of keys)"]
  R --> S["Segment (a book, sealed when full)"]
  S --> Rec["Record (a page: key, value, headers)"]
  R -. "splits as it grows" .-> R2["Range (a second shelf)"]

Analogy. A topic is a library, a range is one shelf holding a band of keys, a segment is a book on that shelf (sealed once full, never rewritten), and a record is a page. When a shelf fills up, the library adds a second shelf and moves half the books' labels over: no book is recopied, only the catalog changes.

  • Range. A topic's keyspace is divided into ranges. A record's key hashes to a position, and that position lands in exactly one range. Ranges split as they grow, which is how a topic scales without the client choosing a partition count up front.
  • Segment. Each range is a series of segments. The active segment takes appends until it crosses its size threshold, then it is sealed and a new one rolls. Sealing is a fence, not a label: the store is closed first and reports where it ended, and that answer becomes the segment's recorded length, so a sealed segment's extent can never disagree with what its replicas hold. Sealed segments are immutable, which is what makes re-replicating them safe.
  • Replica set. Each segment is replicated across nodes chosen by rendezvous (HRW) hashing. A write is acknowledged when a quorum has it durably (fsync before counting), so the system tolerates ⌊(N-1)/2⌋ slow or failed replicas.

Reads never expose any of this. Malachi.LogApi.fetch/5 returns records plus the next cursor.

Consumer groups

A cursor you carry yourself is fine for a single reader. For a shared, resumable position, use a consumer group: the server commits the group's position, so a restart resumes where the group left off.

node consumer.js orders --group workers    # position committed server-side

For parallel consumption, run several members of the same group with distinct member ids, the server assigns each member a share of the topic's ranges and scopes its reads to them. The client still never sees a range id.

Analogy. A consumer group is a team sharing one bookmark that the server holds. Add members and the team also splits the reading, each taking a section, like several people working through one filing cabinet, each assigned a set of drawers.

Durability and ordering, precisely

  • Ordering is per range, and therefore per key (a key always hashes to the same range), never global across a topic. Note the guarantee is the range's, not the key's: two different keys that land in the same range are also ordered relative to each other, but you must not rely on that, because a range split can separate them later.
  • Durability: a produce is acknowledged after a quorum of the segment's replicas has fsynced it.
  • Delivery: at-least-once. A consumer group commits positions, so a crash between processing and commit re-delivers; make your handlers idempotent.

Analogy. Order is the drawer's, not the surname's. Two letters filed in the same drawer today arrive in the order you filed them, but if that drawer is later split in two, they may end up in separate drawers. So rely on order only for the same key, never across keys. Durability is a majority of clerks signing the receipt before you are told it is stored.

Same-key ordering survives resharding (the departure from Kafka)

Order for a single key holds even when the topic is resharded, which is where Malachi differs from Kafka.

A key hashes to a fixed position, so every record for that key lands in the same range, and the range's single primary appends them in the order it receives them. That is the same per-key guarantee Kafka gives per partition, but Kafka qualifies it with "as long as the partition count does not change": its client-visible partition is hash(key) mod partitions, so raising the partition count moves the key to a different partition. Its new records go there while its old records stay behind, and the per-key order breaks.

Malachi never exposes a partition or an offset, only the opaque cursor, so it has no such caveat. A range split is a metadata operation with a seal-first fence: the parent range is sealed before either child takes a write. A reader of that key then drains the parent's sealed records for its slice of the keyspace first (oldest first), then the child's new records. The key stays in exactly one child, and its history reads old-then-new, in order, straight through the split. See Architecture for the split and the cross-epoch read.

flowchart TB
  subgraph kafka["Kafka: raising the partition count scatters key K"]
    KA["r1, r2 in the old partition"]
    KB["r3 in the new partition"]
    KA -. "two partitions, per-key order lost" .-> KB
  end
  subgraph malachi["Malachi: a seal-first split keeps key K one ordered stream"]
    MA["r1, r2 in the parent (sealed)"] --> MB["r3 in the child"] --> MC["cross-epoch read: r1, r2, r3 in order"]
  end

Analogy. Kafka resharding is like renumbering everyone's mailbox: your old and new mail land in different boxes. Here you hold a coat-check ticket, not a box number, so the cloakroom can split a rack in two and the ticket still walks you through your coats in the order you left them.

What the guarantee does not cover

The order above is the order the range's primary appends in. Four things follow from that, and none of them are covered by the guarantee:

  • It is order, not adjacency. A range holds many keys, so between two records of the same key there are records of other keys. "The next record for this key is written after the previous one" is guaranteed; "at the next offset" is not.

  • Concurrent writers are not ordered relative to each other. Two produces in flight at the same time, on different connections or different nodes, land in whatever order they reach the primary's mailbox. There are no producer sequence numbers and no idempotence. Ordering is guaranteed for writes a client has ordered itself, that is, waiting for the acknowledgment before sending the next one. Kafka's guarantee has the same shape: it is per producer, per partition.

  • A retried produce lands at its retry position. The primary appends a batch to its local log before waiting for the quorum, so a produce that came back no_quorum and was retried appears twice, with the second copy at the later position. Delivery is at-least-once, and the failed attempt does not hold its place in the order.

  • A primary that dies seals its segment rather than handing it over. Writing rolls to a fresh segment, which is what NorthGuard does, and it is what keeps an offset from being issued twice: a batch is acknowledged once a majority holds it, so a replica outside that majority can be behind, and promoting that one would let it append at offsets the dead primary had already acknowledged. Sealing removes the possibility instead of detecting it. Finding the seal point and making it binding are two steps, in this order: the pass first measures every live replica, leaving it writable, and only once enough answers are in does it fence them, sealing each answering copy before the control plane records anything. An old primary that comes back then appends to its own log and finds no quorum, because every follower it reaches refuses the push. The order matters because a fence has no inverse: closing replicas of a segment the pass then declines to seal would leave them refusing writes forever, shrinking every later write quorum for nothing. The seal goes at the highest durable end reported, once the seal quorum has answered: n - q + 1 of n replicas, where q is the acknowledgement quorum. Why that covers everything acknowledged: take any acknowledged record, at offset o. It lives on at least q replicas, the answering replicas are at least n - q + 1, and two such subsets of n replicas always share one, so some answering replica holds it. A replica's log is contiguous, so that replica's durable end is above o, and the highest end among the answers is at least that. The replica doing the covering can be a different one for each record, which is why the seal takes the highest end rather than trusting any single replica to hold the whole segment. Note also that it is the highest end among the answers, not the end a majority of them agree on; that lower point can sit below a record the dead primary acknowledged with a single survivor. At three replicas the seal quorum is two; at two it is one, since an acknowledgement needed both, so a segment placed on two brokers is sealed on its survivor. Below the seal quorum there is no intersection to argue from, so the segment is left alone and its range stops accepting writes until enough answer again.

  • A primary that is alive does not keep a segment writable on its own. Every append needs the acknowledgement quorum, so a segment whose live replicas fall below it (the follower of a two-replica segment gone) is sealed on those that answer under the same rule. A broker the membership view calls gone is asked directly first, read-only, and one that answers counts as alive: a suspicion is not a failure, and it seals nothing.

  • The write half of the split gap is closed; the read half needs no closing. Before a split (or a merge) the parent's write head is fenced in the data plane: its primary seals the segment's log and refuses every later append with {:error, {:sealed, end_offset}}, durably, so it refuses again after a restart. A frontend that has not yet seen the split is refused, seats itself at the fenced edge, and retries against the successor. The head is found in the control plane, not in the splitting node's own cache, so the fence does not depend on that node having produced to the range and an operator can split from any node (#41).

    What the fence does not claim. It closes the segment's primary, which is enough because an append is only accepted by the head of the replica set, and once the segment is sealed no failover can promote a new head (failover seals rather than promotes). It does not stop reads of the parent, which keep working and are correct cross-epoch behavior rather than a gap. And it can only fence a head the control plane knows about, which is why a split of a range that still shows a write head is refused rather than performed unfenced.

Where this is going

Ranges splitting is one axis of scale; the metadata itself is the other. The control plane is sharded across virtual nodes (each its own Raft group) and supports online vnode split and grow re-sharding. See Architecture for how that works.

Analogy. There are two ways to grow a library: add more shelves (split ranges to hold more records), and hire more librarians each owning a section of the catalog (shard the metadata). Malachi does both, while the library stays open.