Malachi.DataPlaneRouter (malachi v0.22.7)

View Source

Single source of truth for data-plane sharding: which BrokerServer owns a topic.

By default there is one shard (Malachi.LogBroker) and every topic maps to it, so routing is a pure no-op and a deployment is byte-identical to one that never knew about sharding. Setting MALACHI_DATA_SHARDS to N > 1 (with no control plane cluster, see Malachi.Config.log_cluster/2) runs N independent in-memory BrokerServer shards; a topic is pinned to one shard by :erlang.phash2/2, so every operation for that topic (create, produce, fetch, consume) lands on the same shard and same-key ordering is preserved.

This is an experimental measurement mode for quantifying how far parallel brokers lift the networked throughput ceiling (the single serial BrokerServer mailbox), not a production feature: consumer-group coordination, retention, and the dashboard still target shard 0 only, and the shards' metadata does not survive a restart.

Summary

Functions

shard_count()

The shard count (always >= 1): the configured one, or 1 whenever a control plane cluster is configured. A clustered node starts one broker however many shards were asked for (Malachi.Application), so hashing a topic across the configured count would route it to a Malachi.LogBroker<i> that does not exist.

shard_for(topic)

The BrokerServer name that owns topic. With one shard this is always Malachi.LogBroker, so callers can route through it unconditionally on the hot path.

shard_name(index)

The registered name of shard index (0-based). Shard 0 is Malachi.LogBroker.

shards(base_dir)

The {shard_name, data_dir} pairs to supervise, given the base log directory. One shard keeps the base directory as-is; multiple shards each get an isolated shard_<i> subdirectory (siblings, so no shard recovers or enumerates another's segments).

Functions

shard_count()

@spec shard_count() :: pos_integer()

The shard count (always >= 1): the configured one, or 1 whenever a control plane cluster is configured. A clustered node starts one broker however many shards were asked for (Malachi.Application), so hashing a topic across the configured count would route it to a Malachi.LogBroker<i> that does not exist.

shard_for(topic)

@spec shard_for(String.t()) :: module()

The BrokerServer name that owns topic. With one shard this is always Malachi.LogBroker, so callers can route through it unconditionally on the hot path.

shard_name(index)

@spec shard_name(non_neg_integer()) :: module()

The registered name of shard index (0-based). Shard 0 is Malachi.LogBroker.

shards(base_dir)

@spec shards(String.t()) :: [{module(), String.t()}]

The {shard_name, data_dir} pairs to supervise, given the base log directory. One shard keeps the base directory as-is; multiple shards each get an isolated shard_<i> subdirectory (siblings, so no shard recovers or enumerates another's segments).