Operations

Running Malachi: the ports it opens, how to scrape it, what to watch, and the settings that must not be left at their development values.

Ports

Only 4040 needs to be reachable by clients. Treat 4041 and 4042 as internal ports: they expose operational detail and user management.

The console endpoint

The operator console is served from its own port, beside the dashboard, until it reaches parity with it (docs/design/operator-interfaces.md, sections 10.6 and 13). It serves the console bundle from priv/static/console/, which is never committed. The bundle is to be built and injected by CI once the web console itself lands (phase 3 of that document); until then no build produces one, and the endpoint answers 503 as described below.

• No bundle, no failure. With no index.html in that directory, the node boots, logs a warning, and every console request answers 503 saying the console was not built into the release.
• Switch it off with MALACHI_CONSOLE_ENABLED=false, which starts no listener at all, for a node run headless against a console served elsewhere.
• A port it cannot open does not stop the broker. The error is logged and the node runs without a console until the port is freed and the node restarts.
• Plain HTTP only, for now. TLS for the console is #70. Put it behind a TLS-terminating proxy, like the dashboard, when it is reached from anywhere but the host.
• Caching. Every file is hashed at startup and served with that hash as its ETag. Files under assets/ whose last name segment looks like a build hash (eight or more letters, digits or underscores with a digit, as Vite writes them) are cached for a year as immutable; index.html and everything else are revalidated on every use, which costs a 304. A stable file that must change in place belongs outside assets/, or under a name that cannot pass for a hash. A new bundle is picked up at restart. Until then a file changed or removed while the node runs, or swapped for a symbolic link (the file or a directory above it), answers 404 rather than new bytes under the old ETag. The check looks for links on every level below the root and compares each file's device, inode, size and modification time before sending it, so it does not see a rewrite that keeps the size and the modification time, nor a swap made between the check and the send. Replace a bundle by restarting the node, not by writing into it.
• Limits. The header limits and read deadline are the dashboard's (MALACHI_DASHBOARD_MAX_HEADER_*), and an idle keep alive connection is closed after that deadline.
• Sessions. The application shell needs no login; the /api/v1 routes do, with Authorization: Bearer or the malachi_token cookie a login on the dashboard sets, which the browser also sends to the console's port on the same host. Each route requires a console role (console roles), and GET /api/v1/me says who the session is.

Health checks

Two endpoints, and the difference matters for orchestrators:

GET /health   the process is up
GET /ready    it is ready to serve

Use /health for a liveness probe and /ready for readiness. Wiring readiness to /health means traffic arrives before the node can serve it.

Metrics

/metrics does content negotiation. Ask for text/plain and you get the Prometheus text exposition format, every series namespaced malachi_:

curl -H 'Accept: text/plain' http://localhost:4041/metrics

Without that header the same endpoint returns the JSON payload the dashboard uses. Either way it needs a session whose account holds the console viewer role or above (or the admin permission): a scraper logging in with a produce or consume account is refused with a 403.

Worth alerting on:

• malachi_domain_violations: segments that cannot meet their placement spread requirement. Non-zero means replicas are concentrating where they should not be, which is a real availability risk that is otherwise silent.
• malachi_storage_integrity_failures_total{reason}: a stored segment failed checksum verification. Non-zero means data at rest is damaged on that node, and the condition is otherwise invisible: a damaged copy serves short reads with no error, so a consumer either stalls at the damaged offset or silently skips the rest of that segment. reason="incomplete" on an active segment is ordinary crash recovery (a partial tail that was never acked); on a sealed segment any reason means corruption at rest, and that copy needs to be rebuilt from an intact replica. The matching log line names the segment and the byte position.
• malachi_storage_flush_duration_seconds: the group-commit flush latency, as a histogram (_bucket{le} four per octave from 8us to about 16.8s, plus _sum and _count). This is the write plus sync every acknowledged produce waits behind, so it is the first place to look when produce throughput drops without CPU rising: an fsync-bound disk shows up here as a rising p99 before anything else moves. Read it windowed, histogram_quantile(0.99, rate(malachi_storage_flush_duration_seconds_bucket[5m])), and sum the buckets across nodes before taking the quantile for a cluster-wide view. Only flushes that wrote records and succeeded are counted; a failed one is in malachi_storage_failures_total. malachi_storage_flushed_records_total divided by malachi_storage_flush_duration_seconds_count is records per sync, which is how well group commit is coalescing (near 1 means it is not), and malachi_storage_flushed_bytes_total over that same count is the average flush size. malachi_storage_flush_duration_seconds_created is the Unix time the histogram began: it changes only when the node restarts, so a tool subtracting two scrapes can tell a restart from counters that simply kept growing.
• Session and auth counters. Note that :session_expired and :session_hijack_attempt are not disjoint: one validation can emit both, so summing them does not count failed validations. The hijack counter means "a token arrived from an unexpected IP", which ordinary NAT rotation can also trigger, so set thresholds against that broader meaning. See Malachi.Auth.SessionManager.

Durability tuning (group commit)

Every produce is fsynced before its ack; group commit coalesces those fsyncs. Two independent knobs, one per path (the clustering guide explains the decision rule with examples):

# rf=1, broker-level. Recommended on for throughput workloads. It writes a segment's primary alone, so it
# applies only with a configured replication factor of 1: a single node runs a one-member cluster with the
# default factor of 3, and sets MALACHI_LOG_REPLICATION_FACTOR=1 to use it (the node warns at boot
# otherwise). The replication-level knob below batches the fsync on every replica instead.
MALACHI_LOG_REPLICATION_FACTOR=1
MALACHI_GROUP_COMMIT=true
MALACHI_GROUP_COMMIT_INTERVAL_MS=5           # flush period; ~the latency each produce pays
MALACHI_GROUP_COMMIT_FLUSH_MAX_RECORDS=8000  # eager flush: bound each fsync even on slow disks
MALACHI_GROUP_COMMIT_MAX_INFLIGHT=200000     # backpressure valve: shed with :overloaded past this

# The produce path has TWO refusals, and a client is meant to tell them apart. :overloaded is the valve
# above, the broker saying it is saturated right now; :rate_limited is the OPT-IN publish quota below,
# this user saying it is over its own allowance. Off by default (0 = no limit); see docs/RATE_LIMITING.md.
MALACHI_PUBLISH_RATE_LIMIT=0                 # produce requests per window, per user, PER NODE
MALACHI_PUBLISH_RATE_WINDOW_MS=1000

# rf>1 (replicated), replication-level. Default OFF: enable only for hot-range, fsync-bound
# workloads (many producers per range); on thin-spread loads it lowers throughput.
MALACHI_REPLICATION_GROUP_COMMIT=false
MALACHI_REPLICATION_GROUP_COMMIT_INTERVAL_MS=10  # its own flush period, decoupled from the rf=1 one

# The replication server's minimum heap (words). Every produce batch is encoded and written in that one
# process, and at the VM's default heap it garbage-collects several times per batch; 256K words (2MB per
# node) halves that. Raise it only if a profile shows the server collecting per batch again, e.g. with much
# larger batches; 0 restores the VM default.
MALACHI_REPLICATION_MIN_HEAP_WORDS=256000

# Active-segment roll size (bytes); unset keeps the 64MB default. Smaller segments seal (and become
# independently replicable/repairable units) sooner, at the cost of more metadata churn.
MALACHI_SEGMENT_MAX_BYTES=67108864

Integrity scrub

Every node continuously re-verifies the data it stores. A sealed segment is immutable and nothing re-reads it, so a checksum is otherwise only confirmed when a consumer happens to read that exact record: bit rot there is silent, and silent in the worst way, because a damaged copy answers reads with the records before the damage and nothing after, with no error. A consumer then stalls at that offset or skips the rest of the range.

The scrub walks the node's own sealed segments, checks every record's checksum, and repairs a damaged copy from a replica that still verifies:

MALACHI_SCRUB_ENABLED=true           # default; set to false to turn the scrub off entirely
MALACHI_SCRUB_INTERVAL_MS=60000      # time between passes
MALACHI_SCRUB_SEGMENTS_PER_TICK=1    # segments verified per pass

A full cycle takes sealed segments on the node x interval / segments per tick. With the defaults and 64MB segments a node verifies about 90GB a day, so 10k sealed segments are revisited roughly weekly, the usual period for disk scrubbing. Raise the interval on a slow or busy disk; lower it (or raise the per-tick count) to cover a large dataset more often.

What it costs. The scan itself was measured directly at 740 MB/s (64-byte records) to 2.7 GB/s (1KB records) on one core, so a 64MB segment costs 24 to 86ms of a core, and the shipped cadence works out to under 0.15% of one core and about 1 MB/s of reads. End to end the effect is smaller than a benchmark can resolve: on the 3-node Docker cluster (benchmark/docker-scrub.sh, which interleaves the cases so ordering cannot bias them) both the default cadence and one three thousand times faster landed inside the machine's run-to-run spread of roughly 15%. Re-run that sweep on real hardware before raising the rate a lot, and note the honest caveat: those runs had the whole dataset in page cache, so the scrub was reading RAM. On a node whose data dwarfs its memory the scan is real disk I/O and competes with the write path.

On detection, the node asks the segment's other replicas to verify their own copies. Only when one of them confirms a copy that is both intact and complete does the repair proceed: if this node is the segment's primary it first moves itself to the end of the replica set, so reads go to an intact replica immediately, and only then is the local copy deleted and refetched, then verified again. Complete matters as much as intact, because a replica whose checksums all pass can still be missing whole records at the end, and repairing from one would delete this copy to refetch less than it held. So a replica qualifies only when its scan matches the record and byte counts the control plane recorded when the segment was sealed. If no replica qualifies, nothing is deleted and the failure is logged loudly: a partially readable copy is worth more than no copy. A repair is traced as malachi.scrub.repair.

Two series to watch, and they answer different questions. malachi_storage_integrity_failures_total{reason} says whether anything is damaged, and malachi_storage_scrub_segments_total{result} says whether the scrub is even running: a verified total that stops advancing means the checking stopped, which the failure counter alone can never tell you, since it reads zero both when all is well and when nothing is looking. Within that second series, result="unrepairable" is the one worth paging on: it counts damage the cluster could not heal by itself, so unlike result="repaired" it does not go away on its own.

Note what the failure counter deliberately does not include: a torn frame at the end of an active segment after a crash. Those bytes were never acknowledged, so recovering past them is routine, and counting it would make an ordinary restart look like corruption at rest. It is still logged. Rot in an active segment, and anything at all in a sealed one, is counted.

A single-node deployment scrubs too, and there the distinction matters more: with no replica there is nothing to repair from, so every finding lands in the unrepairable path with a loud log. That is still the difference between knowing and not knowing that data at rest went bad, and because recovery never truncates a copy damaged by rot (a bad checksum or a mangled frame header), the frames after the damage are still on disk for a manual salvage.

Be precise about which damage that covers, because the two shapes are not treated alike. Rot is left alone whatever the file's state. A torn tail is dropped unless the file carries its own seal marker, and a replica usually does not: that marker is written when the segment's log rolls locally, not when the control plane seals the segment. That asymmetry is deliberate rather than an oversight, since a torn tail has nothing valid after it by construction, so there is nothing to salvage. Rot is the case where the bytes past the damage are still worth keeping.

That holds for an active segment too. A node that restarts onto rot in the segment it is still writing keeps the damaged frame and everything behind it, and never appends over them: the copy is taken out of service as a storage failure (malachi_storage_failures_total{reason="other"}, and a failed in storage log line naming the segment and damaged_tail). What happens next depends on the replication factor:

• RF 2 or more. The heal pass seals the segment on the intact copies at the furthest end they hold, and writing rolls to a new segment, which is what NorthGuard does when a replica of a segment fails. Producers see nothing. The damaged copy is replaced by self-healing. At RF 2 one intact copy is enough: an acknowledgement needed both copies, so the intact one holds every acknowledged record.
• RF 1. There is no other copy to seal on, so the range stops taking writes until an operator acts. That is deliberate: the alternative was writing new records over ones that may have been acknowledged. Rolling a single-copy segment on without that risk is tracked in #210.

To recover a blocked range by hand, on the node that reported the damage:

1. Stop the node and copy the segment's .log file somewhere safe. The frames after the damage are still in it.
2. Either truncate the file at the position the integrity log line reported, accepting the loss of everything from there on, or, with a replica elsewhere, remove this node's copy of the segment directory and let the cluster re-replicate it.
3. Start the node again.

Chaos certification

scripts/docker-chaos-test.sh runs the certification drill on a local 3-node RF=3 Docker cluster: synthetic traffic flows while a node is power-pulled (SIGKILL), partitioned off the network, stalled (SIGSTOP, sockets open but mute), and finally every node is rolling-restarted. Three invariants must hold or the script exits nonzero:

1. No acknowledged write is ever lost. A checker produces sequential values through the whole window, retrying through the faults, and records only the confirmed ones; at the end every one of them must read back (rf=3 quorum durability).
2. The cluster reconverges to 3/3 healthy after every event.
3. Availability recovers: errors during an event are expected, and a clean produce+fetch must pass once the chaos ends.

Run it before releases or after touching replication, failover, or membership code.

scripts/docker-storage-chaos.sh extends the drill to storage faults, injected with the target node stopped: a follower's segment copy suffers a torn write (cut short with a garbage partial frame appended: recovery clamps at the last CRC-valid frame and catch-up or the healing pass repairs the tail), a gross truncation to half, and a sealed-segment directory deleted outright. The deletion exercises the self-healing integrity probe: metadata still says the segment has all its replicas, so only a physical check (on-disk bytes vs the sealed byte size, run each healing pass) can spot the silent under-replication and re-backfill the copy. On top of the three invariants above, the storage run requires physical reconvergence: every node's copy of every chaos-topic segment must be readable and hold the same records, byte for byte over the valid part of its files, and a sealed segment exactly its sealed length. The files themselves need not match: only a fenced copy has its preallocated tail trimmed, and each node rolls its internal files at its own sync points. Corruption always targets follower copies; corruption of a primary copy is seal-on-failure territory (roadmap). A storage FAILURE is not: when a write or read fails on a node (a full volume, a failing device), that node stops using the segment's copy, answers {:error, {:storage, reason}} for it, counts it in malachi_storage_failures_total, and the healing pass seals the segment on its other replicas so producers move to a new one. From then on the failed copy counts as a lost replica: the healing pass backfills a replacement on another broker and deletes the broken copy (which lets that node use the segment id again), and the scrub skips it meanwhile instead of refetching it onto the disk that failed. With no spare broker the copy stays, behind the healthy replicas, and the pass reports the segment as {:no_spare_broker, copies}. The drill's last event fills a node's volume to certify it. In-place corruption that keeps the byte size (bit rot) and a rotted sparse index are covered too, by the integrity scrub described above. The run sets MALACHI_SEGMENT_MAX_BYTES and MALACHI_LOG_ROLL_MAX_BYTES low so segments seal and roll within the window; both knobs are available to any deployment that wants smaller roll sizes.

scripts/docker-config-chaos.sh certifies config deployments, the way this repo deploys them (one image, config via env): a harmless setting is rolled across the nodes one at a time, requiring the checker's acks to keep flowing between every step and the new value to be effective on all three nodes at the end; then a config that fails fast at boot is pushed to a single node, which must crash-loop and never go healthy while the other two keep serving quorum writes, and rolling the env back must bring it home to 3/3. The same closing invariants apply: no acknowledged write lost, full reconvergence, clean produce+fetch after the chaos.

Every drill's closing reconvergence check also compares the control plane across members: every Raft group must hold the same state on every node at equal applied indexes, since members that diverged still answer health checks.

scripts/docker-upgrade-chaos.sh certifies the procedure in Upgrades and the rollback floor: a cluster rolled from the release before this code to this code under the machine version pin, back again, then forward, finalized, with a canary feature switched on and a rollback that must be refused with exit 78.

Retention

Segments are reclaimed by age or total size:

MALACHI_RETENTION_MAX_AGE_MS=604800000     # 7 days
MALACHI_RETENTION_MAX_BYTES=10737418240    # 10 GiB per range
MALACHI_RETENTION_INTERVAL_MS=60000

Leave a limit unset to disable it. With both unset, segments are kept forever unless a topic's own storage policy says otherwise: the sweep runs either way, and with no bound anywhere it expires nothing, though it still rolls active segments older than MALACHI_SEGMENT_MAX_AGE_MS (below). Do not write 0 meaning "unlimited": 0 is a valid budget of zero bytes, and it expires every sealed segment it can. A bound must be a whole number from 0 through 2^64 - 1: a negative one, or one past that, stops the node at boot.

Only sealed segments are eligible, so the active segment is never deleted. The byte budget is per range, not per topic or per node. With both limits set a segment goes if either says so.

A segment seals by size (MALACHI_SEGMENT_MAX_BYTES) or by age: the sweep also asks for every active segment older than MALACHI_SEGMENT_MAX_AGE_MS (7 days unless set, at least 60000) to be sealed, so a topic too quiet to fill a segment still expires on time. A record lives at most that plus MALACHI_RETENTION_MAX_AGE_MS, plus up to two MALACHI_RETENTION_INTERVAL_MS (the sweep that asks for the roll and the one that expires it) and the time the fence takes to answer. See Per-topic retention for the per-topic override and what each roll costs.

These are the cluster's defaults. A topic bound to a storage policy takes the bounds the policy sets and inherits the rest; see Per-topic retention for mix malachi.policy, the wire keys and the dashboard routes, and mix malachi.policy get <topic> to see which value applies and why.

Retention metrics

Retention is observable from both sides: what the sweep deleted, and which readers were moved past data that was no longer there.

The sweep (only the node that leads a vnode sweeps it, so read these summed across nodes):

• malachi_retention_sweep_duration_seconds: a histogram of sweep durations, with the same buckets as the flush histogram. Its _count is the number of sweeps this node ran. A _count that stops advancing on every node means no sweep is running, the question an operator cannot answer otherwise.
• malachi_retention_segments_expired_total{topic} and malachi_retention_bytes_expired_total{topic}: what the sweep deleted.
• malachi_retention_expire_failures_total{reply}: deletes the control plane refused, by reply: migrating (the topic was moving between vnodes), segment_active (still the write head) or other (anything else, a Raft timeout included). A segment the sweep found already gone is neither expired again nor a failure.

The readers:

• malachi_retention_skips_total{topic,reader,group,origin,span}: how many times a reader was moved past data no longer stored (expired by retention, or deleted by an operator). Each distinct skip is counted once, however often a group re-reads it before committing. reader says which kind of reader it was: group names it in group, none is a fetch outside a group and other a group folded by the label cap, both with an empty group. Nothing reserves a group name, so reader is what keeps a group that calls itself __other__ or "" apart from those two buckets. origin="cursor" is the one to alert on: a group that held a position and fell behind retention. origin="start" is a reader that had no position, a new group or a range's children after a split, which start over the range's history; retention reaching them is expected.
• malachi_retention_offsets_skipped_total{topic,reader,group,origin,span}: how many offsets those skips stepped over. span="exact" is exact. span="upper_bound" is a skip over the ancestor of a split range: the count covers the ancestor's whole range, of which this reader would only have received its own key slice. span="unknown" is an ancestor with nothing left stored whose end this node could not recover after a restart, so only the fact of the skip is known and the offsets are 0. It counts offsets, not records lost.

A suggested alert: increase(malachi_retention_skips_total{origin="cursor"}[15m]) > 0, which names the group that lost data. The first skip of a reader is also logged, then at most once per reader per MALACHI_RETENTION_SKIP_LOG_WINDOW_MS (10 minutes), with the count of skips held back since the last line.

group is a label a client chooses, so it is capped: past MALACHI_RETENTION_METRICS_MAX_GROUPS (1000) topic and group pairs on a node, new groups are folded into reader="other" with no name, and the log line still names them. The sum over readers per topic stays exact. The skip reporter's memory is bounded the same way by MALACHI_RETENTION_SKIP_LEDGER_MAX (10000 skips remembered); a forgotten skip read again is counted again.

The orphan sweep

A replica that does not answer its delete keeps the segment's directory, and no later sweep can ask for it again: a segment gone from the control plane never comes back. A separate worker per node reclaims those directories, and other leaks of the same shape (a catch-up that failed after creating the directory, a copy healing moved elsewhere).

MALACHI_RETENTION_ORPHAN_SWEEP=delete         # delete | report | off
MALACHI_RETENTION_ORPHAN_SWEEP_INTERVAL_MS=300000
MALACHI_RETENTION_ORPHAN_MIN_AGE_MS=600000
MALACHI_RETENTION_ORPHAN_SIGHTINGS=2
MALACHI_RETENTION_ORPHAN_MAX_PER_PASS=50
MALACHI_RETENTION_ORPHAN_MAX_TRACKED=10000

It runs on every node, not only the one that sweeps retention: only a node can read its own disk. A single node is no exception: it runs its control plane as a one-member ra cluster and asks it the same way. Before a directory can go, the sweep asks the metadata vnode that owns its segment, routed by the segment's topic like any other command for it, and read linearizably. It does not use this node's cached copy of the metadata: that copy is refreshed by a reconcile that can fall behind without failing (a control plane slow enough that every refresh outlives its tick), and a view that aged in silence is missing exactly the segments whose replicas have just arrived here. A registration commits on its owner before any replica creates the directory, so the owner's answer has no lag.

The pass is skipped whole when an owner does not answer, when the topology changed while it asked, when there is no ring yet, or when the whole question took longer than its deadline (15 seconds by default; ra follows a leader redirect with a fresh timeout, so the deadline is on the question, not each read). On a sharded cluster the topology comes from the durable ring store, which a vnode split writes its pending intent to before it moves anything, so a node whose gossip lags cannot ask only an owner the topic already left.

A directory whose owner does not list its segment is asked of every vnode before it can go, because a broker routing by a stale ring can write a topic to a vnode that does not own it (see docs/ARCHITECTURE.md, "a vnode accepts metadata outside its arc"). So while any vnode is silent, a pass with a real orphan to decide is skipped, even when the silent vnode owns none of this node's directories.

Some directories are held without counting a sighting: one whose segment id carries no topic (no owner to ask, held for good), one whose topic a vnode split is moving and that neither the old nor the new owner lists at the moment each is read, and one a vnode that does not own it still lists. The log names them when that set changes.

On top of that answer, a directory is removed only after it has been unexplained on SIGHTINGS consecutive passes and is older than MIN_AGE_MS. Only the MAX_TRACKED oldest candidates are asked about in one pass, which bounds the work on this node and how many owners are asked; each owner answers with its whole segment map whatever the count. Candidates left out for a later pass are logged on every pass that leaves some out; a held pass is logged once, when holding starts; directories kept as undecided are named when that set changes.

The log data directory belongs to Malachi alone. Do not nest MALACHI_RA_DATA_DIR inside it, and do not keep backups or hand-made copies there. The sweep only ever considers a directory whose name the storage layout could have written, so anything else is out of its reach by construction, but that is a check on the name and not a promise about what may share the directory.

report does everything except the removal, which is how to see the list before trusting it on a cluster for the first time. off does not even list. A value that is none of the three stops the node at boot rather than falling back, because the fallback is the mode that deletes and a typo would have chosen it in silence.

• malachi_retention_orphan_directories_left_total{topic}: directories an expire left behind because the replica did not answer. It moves whether or not the sweep is on.
• malachi_retention_orphan_directories_removed_total: directories the sweep reclaimed. Read against the one above: a gap that keeps growing means the sweep is off, is being held back by a guard, or is not keeping up with MAX_PER_PASS.

Recovering a single node

A single node keeps its metadata in a one-member ra cluster, named malachi_log unless MALACHI_LOG_CLUSTER says otherwise. ra keys that cluster by the node's name: its log lives under MALACHI_RA_DATA_DIR/<node name>, and the member is recorded under that name. So a single node keeps what it acknowledged across a restart as long as three things survive it: MALACHI_LOG_DATA_DIR, MALACHI_RA_DATA_DIR, and the node name. A release names the node after its host, so a container needs a fixed --hostname (or RELEASE_NODE); iex -S mix without --name always runs as nonode@nohost.

Before the broker starts, every node checks that its log directory and its control plane describe the same data, and refuses to start (exit status 78, one REFUSING TO START line) in these cases:

• A log directory of another control plane. A start records the control plane name in the log directory (malachi.cluster) once it passes every check, or just before it writes a seed, and MALACHI_LOG_CLUSTER now names another one. A start refused before its seed is written records nothing. One that fails after the seed check passed (the seed write failed, or the membership check refused) keeps the name: the seed's vnodes carry it, and a write that timed out may have landed. Removing malachi.cluster is safe only when that seed is known not to have landed (on a cluster, mix malachi.ring --show against a running peer says whether a ring is recorded). The ring store and the ra members are keyed by node name, not by cluster name, so the two control planes would mix over the same ra directory and the sweep could delete the first one's segments, even on a member that holds none of them. Adoption does not apply. Set MALACHI_LOG_CLUSTER back, or form the new cluster with another MALACHI_LOG_DATA_DIR and another MALACHI_RA_DATA_DIR. A directory written before the marker existed gets one on its next start that passes, so a rename made at the same time as that upgrade is not caught by this check (the other checks still apply).
• Segments nobody knows. The log directory already holds segment directories, and the control plane that would describe them is about to be formed instead of resumed: the ring store was never started on this node (sharded or not), or the unsharded metadata member of MALACHI_LOG_CLUSTER was never started here, or a sharded ring is about to be seeded (every vnode formed now, knowing none of them), or, on a sharded node configured alone, one of its vnodes was never started here. A cluster is not held to the vnode check: a vnode with other replicas keeps its history on them, and a rebalance moves vnodes off a node without republishing the ring. A vnode whose only replica is this node (MALACHI_LOG_VNODE_REPLICATION_FACTOR=1 on a cluster) has no such protection. The ring store is checked before the boot starts it and the seed before it is written, so those refusals leave nothing behind; the vnode and member checks run after the ring store is resumed or seeded. The ring store is one per ra directory and node name, whatever the cluster is called, so a new MALACHI_LOG_CLUSTER needs a new MALACHI_RA_DATA_DIR as well. A control plane formed now would be empty, the orphan sweep would delete every one of those directories, and a topic created again under an old name would read the old bytes as its own. The causes are a changed node name, a lost ra directory, a renamed MALACHI_LOG_CLUSTER over a directory with no marker yet, or segments written by a release that kept a single node's metadata in memory (those are unreachable: the metadata that described them is gone). Bring back the node name, the ra directory or the cluster name. If the directories are not wanted, set MALACHI_ADOPT_ORPHANED_LOG_DIR=true: the node starts, logs the names it adopted, and the orphan sweep removes them on its usual schedule. A cluster member is refused too, since it would come back as a voter with no history; when its peers still hold the control plane (only its own ra directory was lost), adopting is the normal way back: the sweep asks the owners and removes only what none of them lists. A node that joins for the first time has nothing on disk to refuse.
• A single node told it has peers. The control plane was started on this node as a one-member cluster (sharded or not), and MALACHI_LOG_NODES now lists other nodes. Growing a single node into a cluster in place is not supported yet: this node would come back alone while the others formed a second cluster under the same name. Remove the other nodes, or form the new cluster under another MALACHI_LOG_CLUSTER with another MALACHI_LOG_DATA_DIR and another MALACHI_RA_DATA_DIR, and move the data with a client. Pointing a new cluster name at the old log directory is the first case above, and refuses.
• A switch to sharding. The control plane already ran unsharded on this node, and MALACHI_LOG_VNODES now asks for vnodes. Converting it in place is not supported: the sharded ring would outrank the environment from then on and the topics' current metadata would never be read again, so the orphan sweep would delete their segments. The node refuses before it writes the sharded ring, so removing MALACHI_LOG_VNODES starts it unsharded again. To shard, form a new cluster under another MALACHI_LOG_CLUSTER with another MALACHI_LOG_DATA_DIR and another MALACHI_RA_DATA_DIR (there is one ring store per ra directory and node name, whatever the cluster is called), and move the data with a client.
• A sharded ring over an unsharded control plane. The control plane ran unsharded on this node, and a sharded ring is already recorded for it: written by a peer that never ran unsharded (one listed but never started), by an earlier release, which converted a control plane in place when MALACHI_LOG_VNODES was set, or by an earlier release on this node under another MALACHI_LOG_CLUSTER over the same ra directory. That ring outranks any setting, so the node refuses on every start and there is no way back in place. A cluster converted that way by an earlier release has to be formed again under another MALACHI_LOG_CLUSTER, MALACHI_LOG_DATA_DIR and MALACHI_RA_DATA_DIR; its unsharded topics were most likely swept already, and keeping the old directories until the data is no longer wanted costs nothing.
• A membership that cannot be read. The same check needs the ring store member's recorded membership, and the member did not report it within MALACHI_LOG_RING_BOOT_TIMEOUT_MS (a long log replay, or a member that cannot come back). A check that cannot be made does not read as passed. Look at the node's ra log, or raise the timeout.

MALACHI_DATA_SHARDS above 1 with no MALACHI_LOG_CLUSTER is a throughput measurement mode, not a deployment: each shard keeps its metadata in memory, nothing written survives a restart, and it runs no orphan sweep. The node logs a warning saying so at boot.

When a topic's policy cannot be read

Retention resolves a topic's policy against the cluster's policy store on every sweep. Two cases stop it expiring, both on purpose, because expiring under the cluster-wide limits instead would delete exactly the data a more permissive policy exists to keep, on every replica, with no way back.

• The store did not answer. The whole sweep is skipped and one line says so. Nothing else is needed to notice: malachi_retention_sweep_duration_seconds's _count stops advancing, which is the signal this guide already tells you to alert on.
• The topic points at a policy name the store does not define, because it was deleted with --force or a bind raced a delete (both refused without it). That topic expires nothing, and malachi_retention_unresolved_policy_sweeps_total{topic} names it on every sweep.

The second case holds disk until someone fixes the binding. Bound how much with:

MALACHI_RETENTION_UNRESOLVED_POLICY_MAX_AGE_MS=604800000   # 7 days; unset means keep everything

Unset by default on purpose: that bound is a retention decision, and the only person who can make it is the one who wrote the policy the binding points at.

These names are reserved for later retention work and not emitted yet: malachi_retention_segments_pinned{topic,group} (consumer-aware retention) and malachi_segment_rolls_total{reason} (time-based rolls).

TLS

MALACHI_ENABLE_TLS=true
MALACHI_REQUIRE_TLS=true
MALACHI_TLS_CERTFILE=/etc/malachi/server.pem
MALACHI_TLS_KEYFILE=/etc/malachi/server-key.pem
MALACHI_TLS_CACERTFILE=/etc/malachi/ca.pem
MALACHI_TLS_VERIFY=verify_peer
MALACHI_TLS_VERSIONS=tlsv1.3,tlsv1.2

MALACHI_ENABLE_TLS offers TLS; MALACHI_REQUIRE_TLS refuses plaintext.

In production both are on unless you turn them off. REQUIRE_TLS defaults to true under MIX_ENV=prod, and ENABLE_TLS simply follows it. So the risk here is not forgetting to enable TLS, it is the opposite: setting MALACHI_REQUIRE_TLS=false to get past a certificate problem and leaving it that way, which disables both at once. Outside production both default to off.

Invalid TLS configuration raises at boot in production rather than starting insecurely; in dev and test it only warns.

Upgrades and the rollback floor

An upgrade has two floors, and a rollback has to clear both: the on-disk format below, and the control-plane machine version further down. They move independently, so check them separately before rolling a release back.

The on-disk format

The root of the log data directory (MALACHI_LOG_DATA_DIR) holds a small text file, malachi.format:

format=1
written_by=0.12.0
requires=0.12.0

format is the on-disk format the directory holds, written_by the release that wrote the file, and requires the oldest release that can read that format. With several data-plane shards, the one file at the root covers every shard_<n> subdirectory.

The rule: a release starts on a data directory only when format is at or below the highest format it understands. Otherwise it refuses to start before opening anything, because an older release does not know a newer frame, would read it as damage, and could write over records that were acknowledged.

What that means for a rollback:

• A directory without the file (a fresh node, or one last written by a release older than 0.12.0) gets one at baseline format 1 on the first start. Starting a newer release does not raise it, whatever format that release can read.
• The format rises only when a format-changing feature is switched on for the cluster. Until then, a rollback to any release from 0.12.0 on is free.
• After a format-changing feature is switched on, the floor is the release named in requires. Rolling back below it is refused.
• Never roll back below 0.12.0 once any format above 1 exists. Releases before 0.12.0 do not read the file at all, so nothing stops them from opening newer data, and that is exactly the overwrite this file exists to prevent.
• Do not delete or edit the file to get a node to start. It is the only thing between an older release and data it cannot read.

A refused start exits with status 78 and logs one line that begins with REFUSING TO START (exit 78): and names the directory, both format levels and the release to start instead. The same line is printed on stderr, so it reaches container logs even when the logger has not flushed. The same status covers a marker that cannot be parsed (restore it from a backup or another node) and one that cannot be read or written (fix the volume or its permissions), and a node whose build cannot honour a cluster flag that is already on (see Cluster flags).

A service manager that restarts on failure will restart a refused node again and again. Tell it not to:

• systemd: RestartPreventExitStatus=78.
• Docker Compose: restart: on-failure restarts on any non-zero status, so bound it (restart: on-failure:5) and alert on exit code 78 (docker inspect -f '{{.State.ExitCode}}').
• Kubernetes: the pod goes to CrashLoopBackOff; the last terminated state shows exit code 78 and the log line above.

The control-plane machine version

The control plane (topic metadata, the lease, the ring, users, lockouts, ACLs, cluster flags and storage policies) lives in Raft groups whose state machines carry a version. A group moves to a new version only once every member runs code that supports it, and a command a release introduces is refused, the same way on every member, until then. So a cluster in the middle of a rolling upgrade cannot end up with members that disagree about its state.

The version switch happens on its own when the last node has been upgraded. From that moment, a node started on the previous build stops applying the control-plane log instead of diverging from it: it stays up, but it falls behind, and it logs stopped applying entries and emits the [:malachi, :ra, :machine_version] telemetry event with stuck: true until it is upgraded again. That is this floor: it moves with the machine version, not with the on-disk format above.

To keep rolling back possible until you are satisfied with a release, hold the version with a pin:

1. Set MALACHI_RA_MACHINE_VERSION on every node to the version the cluster runs now, and roll the new build out node by node. The group stays at the pinned version, and any node can go back to the previous build.
2. When the release is proven, finalize: remove MALACHI_RA_MACHINE_VERSION and restart the nodes one at a time. The groups switch to the new version once the last node is back, and the rollback floor moves up.

The first release that versions these machines is version 1, and every earlier build counts as version 0.

Version 2 adds the cluster flag store's enable_flag command. Version 3 adds the storage policy store's define_policy and delete_policy. Version 4 adds the metadata command that binds a topic to a storage policy (bind_topic_policy), so mix malachi.policy bind is refused, with a message saying to finish the rolling upgrade, until every node runs a release with version 4 and the pin is lifted. Version 5 adds the console roles to the user store (set_role, creating a user with a role, and importing users with roles): until the cluster reaches 5, assigning a role is refused on every surface, as HTTP 409 errors.cluster.upgrade_pending on the dashboard and the console and as the same "finish the rolling upgrade" message over the wire and in mix malachi.user. A default user that carries a role is not created, while creating a user without one keeps working. node scripts/user.js create --role creates the user and then sets the role in a second request, so on such a cluster it leaves the user created without a role. Finalizing it gives up rolling back below the release that introduced the roles. Each time, the machines that gained nothing move with it at no cost, and a member on the older build refuses the new command until the whole group has moved.

The same rule applies at every step: to keep a rollback to the previous build possible while you roll out, upgrade with MALACHI_RA_MACHINE_VERSION set to the version the cluster runs now, and finalize afterwards.

A malformed value (anything but a non-negative integer) stops the node at boot. Silently dropping the pin would finalize the upgrade, so it is treated as an error. A pin above the version a build implements has no effect.

Cluster flags

Finishing a rolling upgrade and committing to a new behaviour are two different moments. A cluster flag is the second one: an operator switches a feature on, once, for the whole cluster, after every node runs a build that can handle it. Until a flag is on, every node keeps to the old behaviour and a rollback stays free.

mix malachi.flag --list
mix malachi.flag enable <flag>

Both run against a running node over Erlang distribution, the same way mix malachi.ring and mix malachi.reshard do (--node, --cookie).

Each node advertises the set of flags its build can honour, through the same SWIM gossip that carries liveness. Switching a flag on is refused unless every node listed in MALACHI_LOG_NODES is alive and advertises that flag, and the refusal names the ones that are not:

these nodes are not alive or do not support that flag: malachi@node3. Upgrade them, or take them
out of MALACHI_LOG_NODES, then run this again

A node that is down counts as not supporting it. Upgrade it, or take it out of MALACHI_LOG_NODES on every node, before switching the flag on.

Two things follow from a flag being permanent:

• There is no command to switch one off. Once a feature is on, peers and stored data may already be in the new shape, and going back is a migration, not a toggle. Decide before you enable, not after.
• A node that cannot honour an enabled flag refuses to start, with exit status 78 and a line naming the flag and what the build advertises. In practice that means a node brought back on a build from before the feature: upgrade it rather than trying to start it again.

Why a node remembers its own incarnation

Each node keeps one number in malachi.incarnation, at the root of the log data directory beside malachi.format. It is how a restarted node outranks what its peers still remember about it.

Membership gossip resolves conflicting news about a member by that number, and a member is the only one who raises its own. A node that came back counting from scratch would announce itself below what its peers hold, every peer would ignore the announcement, and they would keep the attributes they had, including the capability list. A node brought back on an older build would then still be counted as supporting a feature it no longer does.

Nothing is asked of you for this, including on the upgrade that introduces the file. A node coming from a build that never wrote one has no record to read, while its peers may remember it well above zero, because the old build raised its incarnation in memory on every refutation and every attribute change. So the first reservation starts above the current second rather than at 1. An incarnation counts refutations, which reaches tens or hundreds; seconds since 1970 do not, so the upgraded node outranks anything the old build could have reached. There is no procedure here, and that is deliberate: the number it would have to be given is whatever the peers happen to remember, which is not visible from the node being started.

The clock is only a floor. A recorded ceiling higher than it still wins, so a clock moving backwards cannot lower a node.

It matters only when it goes wrong: a node whose incarnation cannot be reserved does not start, and one that later cannot record a new ceiling stops itself so it can reserve a fresh one on the way back up. Both say so on one line. Starting anyway would be worse than not starting, because the node would be ignored by its peers with no way back: they would keep the record they have, and since the node answers their pings they never suspect it, so nothing ever provokes the refutation that would lift it.

Do not delete the file while the node is stopped, and copy it along with the data directory if you ever move one. The clock floor makes a lost file survivable rather than fatal, which is not the same as a licence to remove one: a node restarted within minutes of losing it can still come back below where it already was.

What a node does with the flags at boot

A node always starts, and it always serves. Nothing about the flags holds it back or takes it out of rotation. What the flags decide is when it stops:

The third row is the one worth understanding. Waiting there, either by refusing to start or by reporting itself unready, sounds safer and is not: the flag store cannot reach a quorum until enough nodes run a build that has it, so the first node of a rolling upgrade would be waiting for a cluster that is waiting for it, and the upgrade would stall on its first step. Serving the previous behaviour is safe in the meantime because a flag is only ever switched on once every node already supports it, so the previous behaviour is one every node in the cluster still understands.

Nothing is given up by that. A node that cannot honour an enabled flag still exits 78 the moment it reads one, and a node whose data directory was written in a format this binary cannot read is stopped earlier and synchronously by the format marker above, which needs no quorum and no peers.

A flag flip is also what raises the on-disk format above the baseline, so the rollback floor described under The on-disk format moves at that moment and not at any other.

Before you go to production

The checks that catch the common mistakes:

• [ ] Passwords set explicitly. Production requires MALACHI_ADMIN_PASS and friends via environment;

  no credentials ship in the base config. The dev defaults (`admin123`) exist only in `dev.exs` and
  `test.exs`.

• [ ] MALACHI_REQUIRE_TLS not set to false. It defaults to true in production, so the check is

  that nobody disabled it while debugging certificates.

• [ ] Dashboard not publicly reachable, and every operator account holding the least console role it

  needs (`viewer` to read). See [console roles](authentication.md#console-roles).

• [ ] MALACHI_LOG_REPLICATION_FACTOR at least 3 if you want to survive a node loss. With 2, quorum

  is 2, so losing either replica stalls writes.

• [ ] Retention configured. The default keeps everything, and the disk fills quietly.
• [ ] Readiness probe on /ready, not /health.
• [ ] You know your rollback floor. Read Upgrades and the rollback floor

  before the first upgrade, and make your service manager stop restarting on exit status 78.

• [ ] The upgrade you are about to make was certified. Releases are tagged on every merge and are not

  held back for it, so check it yourself: the
  [Rolling upgrade certification results](../generated/chaos-upgrade-results.md) page shows the last
  nightly run, which rolls a cluster from the newest release whose code differs from main's to main's code and back under
  load, following the procedure in [Upgrades and the rollback floor](#upgrades-and-the-rollback-floor).
  Its record names both releases and the commit it measured. For another pair, run the workflow by hand
  with `old_ref` (see [Running the chaos drills](running-chaos-drills.md#rolling-upgrade-and-rollback)).

• [ ] MALACHI_LOG_NODES lists exactly the nodes you run. A node left in the list that is not running

  blocks every cluster flag, and one missing from it is not counted when a flag is switched on.

• [ ] Your service manager treats exit 78 differently from an ordinary failure. Exit 78 means the

  node cannot run with the binary it was given, so restarting it only hides that; an upgrade is what
  helps. systemd can stop outright (`RestartPreventExitStatus=78`); Compose cannot tell the two
  apart, so bound the retries (`restart: on-failure:5`) and alert on exit code 78.

• [ ] malachi_domain_violations alerted on.
• [ ] If you use ACLs, MALACHI_ACL_STRICT=true. Without it grants are inert and global permissions

  still allow everything. See [Per-topic ACLs](per-topic-acls.md).

Docker

Images are published multi-arch. See Running with Docker for compose files and the environment matrix.

Next

For growing the cluster, see Clustering and re-sharding.