malachi
A node just failed. How many orders were lost? None.
A topic split in two. How many records were copied? None.
A node restarted. How many cursors broke? None.
Clients speak topics, keys and opaque cursors, never partitions or offsets, so Malachi can split, merge and restripe its storage underneath without breaking them.
Malachi acknowledges a write only after a majority of the segment's copies have fsynced it. With three copies, one node can fail and every acknowledged record is still on the other two.
A split seals the range's active segment and gives each half of its keyspace to a new child. It is a metadata change: no segment is copied or rewritten, and merges happen only between buddy halves.
Clients hold topics, keys and an opaque cursor. Splits and merges happen underneath; a read drains the parent's sealed data before the child's, so records with the same key stay in order.
Topic, range and segment metadata lives in Raft groups on ra, inside each node. Records stay out of Raft: the primary ships them to its followers in batches.
The model
Each dot is a record landing on a key; the keyspace runs top to bottom, time runs left to right. A range takes appends into its active segment. When its load passes a threshold it splits into two buddy halves of the keyspace, and the parent's segment is sealed first, behind a fence, before either child takes a write. When load falls, buddies merge back. The cursor a client holds never changes shape.
Inside a topic
A topic is a keyspace divided into ranges. Its active ranges cover the keyspace exactly once, and they split and merge at runtime. Here, range 1 split into ranges 2 and 3.
A range owns one block of the keyspace. Every record whose key lands in that block is appended to it, in a sequence of segments. After a split the sealed parent stays as history its children read through.
The unit of replication. The active segment takes appends at its write position; a sealed segment is immutable, and retention deletes the oldest first.
A record frame: a short header, then the payload its CRC32 covers: key, value and headers. Malachi does not interpret the value.
What you get
A record's key hashes to a position in the topic's keyspace, and that position falls in exactly one range. Split and merge are metadata operations: segments are never physically combined or copied, and a merge happens only between buddy ranges. Same-key ordering survives resharding.
A segment is the unit of replication. It takes appends while active and becomes immutable once sealed, by size, by age, or on failure. Every record frame carries a checksum, and a scrubber re-verifies sealed segments and rebuilds a damaged copy from a replica that still verifies.
Topic, range and segment metadata lives in vnodes, each a Raft group on ra, sharded on a hash ring by topic. Records do not go through Raft: the primary ships batches to its followers and acknowledges a write once a quorum has fsynced it.
Every producer and consumer authenticates. Users and per-topic ACLs are replicated across the cluster in their own Raft group. Passwords are hashed with Argon2, TLS is required in production, nodes can talk over mutual TLS, and failed logins lock an account out progressively.
How it works
01
The key routes each record to the one range that owns its slice of the keyspace. When a range gets busy it splits at the midpoint of its keyspace: its segment is sealed, and each half goes to a new child range.
02
The primary ships each batch to its followers and acknowledges it once a majority has fsynced it. A full, old or failed segment is sealed on every copy and never edited again.
active ─ 1 GB · 1 h · failure ─▶ sealed03
A fetch returns records and an opaque cursor; pass it back to resume. After a split, the read drains the parent's sealed records first, then follows its key into the child.
Compared
Sources: Kafka docs, Basic operations · Kafka docs, Design · Confluent Platform docs
Source: Redpanda docs, Manage Topics
Sources: Pulsar docs, Architecture · Pulsar docs, Manage topics · Pulsar 5.0 docs, Scalable topics · Pulsar 5.0 release announcement
Apache Kafka, Apache Pulsar and Redpanda are trademarks of their owners. Malachi is not affiliated with or endorsed by them. Every claim above was checked against the linked documentation.
Malachi listens on 4040 and the dashboard on 4041. It is in active development and not production-ready.
# Set your own admin password (12 characters or more) in the shell first:
# export MALACHI_ADMIN_PASS='your own password'
docker run \
-p 127.0.0.1:4040:4040 \
-p 127.0.0.1:4041:4041 \
-e MALACHI_ADMIN_PASS \
-e MALACHI_REQUIRE_TLS=false \
hectorcardoso/malachi:latest