Malachi.Config (malachi v0.22.7)

View Source

Helpers for normalizing operator-supplied runtime configuration.

Extracted from config/runtime.exs so the rules can be tested directly. That file is skipped entirely under config_env() == :test, so anything defined inline in it is unreachable from the suite; a pure function here is not.

checked/4 is the other half of the same idea, applied where the value is finally used rather than where it is parsed: the config layer turns an environment variable into a term without judging it, and the process that needs it decides whether the term can do the job.

Summary

Functions

value when valid? accepts it, otherwise default, saying out loud which setting was refused.

The cluster's identity as GET /api/v1/me reports it, from MALACHI_CLUSTER_DISPLAY_NAME, MALACHI_CLUSTER_COLOR and MALACHI_CLUSTER_ICON. Each is nil when absent or blank. The color is a #RRGGBB hex value and the icon a short slug the console maps to an icon (^[a-z0-9-]{1,32}$); anything else stops the node, since a console that cannot render the identity defeats its purpose, which is telling clusters apart at a glance. Every node should carry the same values.

Normalizes an on-disk data directory taken from an environment variable.

The default users in MALACHI_DEFAULT_USERS: entries separated by ;, each user:password:permissions[:role], permissions separated by , and possibly empty (an operator with a console role and no wire permission, ops:secret::viewer). Returns {username, password, permissions, role} tuples, role nil when absent. An unknown permission or role, an empty username or password, or an entry of the wrong shape stops the node: a typo here would otherwise seed an account that cannot do what the operator meant, or one that silently can do more, such as an admin with an empty password. A password cannot contain : or ;, the separators. Every error names the entry by its position alone, its place among the ; separated segments with empty ones counted, and prints no field, the username included, since a mistyped separator can move part of a password into any of them.

The number in raw as a float, or default when the variable is absent or blank. Anything else raises, for the reason given in integer/3.

The whole number in raw, or default when the variable is absent or blank. Anything else raises, so the node refuses to boot.

The control plane cluster this node runs, from MALACHI_LOG_CLUSTER and the data-plane shard count.

Normalizes the ra machine version pin taken from MALACHI_RA_MACHINE_VERSION.

A retention bound from raw: nil when the variable is absent or blank (the rule is off), otherwise a whole number from 0 through 2^64 - 1. Anything else stops the node, for the reason integer/3 gives and two more. A negative age or byte budget expires everything the rule can see. A bound above 2^64 - 1 is stored whole but travels in 64 bits (Malachi.Wire), so get_topic_policy would report it truncated, 2^64 as a zero budget, while the sweep applies the real value.

Normalizes the orphan sweep mode taken from MALACHI_RETENTION_ORPHAN_SWEEP.

Normalizes the OpenTelemetry sampling ratio taken from MALACHI_TRACING_SAMPLE_RATIO.

How old an active segment may get before the retention sweep rolls it, from MALACHI_SEGMENT_MAX_AGE_MS: 7 days when the variable is absent or blank, otherwise a whole number of milliseconds from the field's floor (retention.segment_max_age_ms in Malachi.Cluster.Policy, one minute) through 2^64 - 1. Anything else stops the node, for the reasons retention_bound/2 gives and one more: below the floor, the sweep that asks for the roll runs too rarely to honor the value, and every roll is one more segment in the metadata.

Functions

checked(value, setting, default, valid?)

@spec checked(value, atom(), value, (value -> boolean())) :: value when value: term()

value when valid? accepts it, otherwise default, saying out loud which setting was refused.

Environment variables reach a process already parsed but not judged: MALACHI_SCRUB_INTERVAL_MS=0 is a valid integer and a busy loop, and MALACHI_RETENTION_SKIP_LEDGER_MAX=0 is a valid integer and a FunctionClauseError inside a server the application supervisor starts. Refusing to boot over an operator's typo turns one lost knob into a lost node, so the documented default is used and the line names the setting, what arrived and what is being used instead.

setting appears in the log, so it should be the name the operator can act on.

Examples

iex> Malachi.Config.checked(5_000, :scrubber_interval, 60_000, &(is_integer(&1) and &1 > 0))
5_000

cluster_identity(name, color, icon)

@spec cluster_identity(String.t() | nil, String.t() | nil, String.t() | nil) :: %{
  name: String.t() | nil,
  color: String.t() | nil,
  icon: String.t() | nil
}

The cluster's identity as GET /api/v1/me reports it, from MALACHI_CLUSTER_DISPLAY_NAME, MALACHI_CLUSTER_COLOR and MALACHI_CLUSTER_ICON. Each is nil when absent or blank. The color is a #RRGGBB hex value and the icon a short slug the console maps to an icon (^[a-z0-9-]{1,32}$); anything else stops the node, since a console that cannot render the identity defeats its purpose, which is telling clusters apart at a glance. Every node should carry the same values.

Examples

iex> Malachi.Config.cluster_identity(" prod-eu ", "#A1B2C3", "globe")
%{name: "prod-eu", color: "#A1B2C3", icon: "globe"}

iex> Malachi.Config.cluster_identity(nil, "", nil)
%{name: nil, color: nil, icon: nil}

data_dir(var, value, env)

Normalizes an on-disk data directory taken from an environment variable.

Returns the trimmed path, or nil when the value is absent or blank (including all-whitespace), in which case the caller falls back to its own default. In :prod a relative path is rejected rather than accepted: it resolves against the process working directory, so the node would write its durable segments to ephemeral container storage instead of the mounted volume, and the loss would only surface on the next restart. Other environments stay permissive, so a relative path is fine for local work.

var names the source variable and appears in the error, so the operator sees which one to fix.

Examples

iex> Malachi.Config.data_dir("MALACHI_LOG_DATA_DIR", "/mnt/vol/log", :prod)
"/mnt/vol/log"

iex> Malachi.Config.data_dir("MALACHI_LOG_DATA_DIR", "  ", :prod)
nil

iex> Malachi.Config.data_dir("MALACHI_LOG_DATA_DIR", "data/log", :dev)
"data/log"

default_users(raw)

@spec default_users(String.t()) :: [
  {String.t(), String.t(), [atom()], Malachi.Auth.ConsoleRole.t() | nil}
]

The default users in MALACHI_DEFAULT_USERS: entries separated by ;, each user:password:permissions[:role], permissions separated by , and possibly empty (an operator with a console role and no wire permission, ops:secret::viewer). Returns {username, password, permissions, role} tuples, role nil when absent. An unknown permission or role, an empty username or password, or an entry of the wrong shape stops the node: a typo here would otherwise seed an account that cannot do what the operator meant, or one that silently can do more, such as an admin with an empty password. A password cannot contain : or ;, the separators. Every error names the entry by its position alone, its place among the ; separated segments with empty ones counted, and prints no field, the username included, since a mistyped separator can move part of a password into any of them.

Examples

iex> Malachi.Config.default_users("admin:s3cret:admin;ops:pw::viewer")
[{"admin", "s3cret", [:admin], nil}, {"ops", "pw", [], :viewer}]

float(var, raw, default)

@spec float(String.t(), String.t() | nil, value) :: float() | value when value: term()

The number in raw as a float, or default when the variable is absent or blank. Anything else raises, for the reason given in integer/3.

Float.parse/1 truncates the same way, and a decimal comma is the everyday case: 0,5 arrives as 0.0, which as a memory threshold means the alarm fires immediately and forever.

Examples

iex> Malachi.Config.float("MALACHI_X", nil, 0.7)
0.7

iex> Malachi.Config.float("MALACHI_X", "0.5", 0.7)
0.5

integer(var, raw, default)

@spec integer(String.t(), String.t() | nil, value) :: integer() | value
when value: term()

The whole number in raw, or default when the variable is absent or blank. Anything else raises, so the node refuses to boot.

The counterpart of checked/4, and what separates them is what being wrong costs. checked/4 judges a value that parsed: MALACHI_SCRUB_INTERVAL_MS=0 is a number, the operator's intent is legible, and the documented default is a reasonable stand-in, so refusing to boot there would turn one lost knob into a lost node. This one judges whether there is a number at all.

Integer.parse/1 keeps the leading digits and discards the rest, which is worse than refusing and worse than defaulting: 600_000 arrives as 600 and 10m as 10, values the operator never wrote and that no default can stand in for, because nothing here knows what was meant. The ones that hurt are the ones that come out smaller. MALACHI_RETENTION_MAX_AGE_MS=7_776_000_000 would expire everything sealed more than seven milliseconds ago, on every replica, with no way back.

var is the environment variable's own name, because that is what the operator has to go and fix.

Examples

iex> Malachi.Config.integer("MALACHI_X", nil, 5)
5

iex> Malachi.Config.integer("MALACHI_X", " ", 5)
5

iex> Malachi.Config.integer("MALACHI_X", " 12 ", 5)
12

log_cluster(raw, data_shards)

@spec log_cluster(String.t() | nil, integer()) :: atom() | nil

The control plane cluster this node runs, from MALACHI_LOG_CLUSTER and the data-plane shard count.

  • A name given: that cluster, whatever the shard count (a clustered node runs one data-plane shard, Malachi.DataPlaneRouter.shard_count/0).
  • No name and one shard (the default): :malachi_log, so a single node runs its metadata as a one-member ra cluster and keeps it across a restart. Before #273 this was nil, in-memory metadata that a restart forgot, after which the orphan sweep deleted every segment written before the boot.
  • No name and more than one shard: nil, the in-memory data-plane sharding measurement mode, whose shards hold their metadata in memory only and run no orphan sweep.

The name comes from a trusted operator (deploy config), so creating the atom is fine.

Examples

iex> Malachi.Config.log_cluster(nil, 1)
:malachi_log

iex> Malachi.Config.log_cluster(" ", 4)
nil

iex> Malachi.Config.log_cluster("orders_meta", 4)
:orders_meta

ra_machine_version_pin(raw)

@spec ra_machine_version_pin(String.t() | nil) :: non_neg_integer() | nil

Normalizes the ra machine version pin taken from MALACHI_RA_MACHINE_VERSION.

Returns nil when the value is absent or blank (no pin: the node advertises the newest version its code implements) and the integer when it is a whole non-negative number. Anything else raises, so the node refuses to boot. Silently ignoring a typo would be the worst outcome here: an operator who pinned the version to keep a rollback possible would find the pin gone and the upgrade finalized, which moves the rollback floor. See Malachi.Cluster.MachineVersion for what the pin holds back.

Examples

iex> Malachi.Config.ra_machine_version_pin(nil)
nil

iex> Malachi.Config.ra_machine_version_pin(" ")
nil

iex> Malachi.Config.ra_machine_version_pin(" 1 ")
1

retention_bound(var, raw)

@spec retention_bound(String.t(), String.t() | nil) :: non_neg_integer() | nil

A retention bound from raw: nil when the variable is absent or blank (the rule is off), otherwise a whole number from 0 through 2^64 - 1. Anything else stops the node, for the reason integer/3 gives and two more. A negative age or byte budget expires everything the rule can see. A bound above 2^64 - 1 is stored whole but travels in 64 bits (Malachi.Wire), so get_topic_policy would report it truncated, 2^64 as a zero budget, while the sweep applies the real value.

Examples

iex> Malachi.Config.retention_bound("MALACHI_RETENTION_MAX_BYTES", nil)
nil

iex> Malachi.Config.retention_bound("MALACHI_RETENTION_MAX_BYTES", "10737418240")
10737418240

retention_orphan_sweep(raw)

@spec retention_orphan_sweep(String.t() | nil) :: :delete | :report | :off

Normalizes the orphan sweep mode taken from MALACHI_RETENTION_ORPHAN_SWEEP.

nil or a blank value is :delete, the documented default. delete, report and off are accepted whatever their case and surrounding whitespace. Anything else raises.

Raising rather than falling back is the same rule as ra_machine_version_pin/1, and for the same reason: this is a knob whose wrong value is not a lost knob. checked/4 exists for an interval, where the cost of refusing the value is one cadence; here MALACHI_RETENTION_ORPHAN_SWEEP=Report from an operator who meant NOT to delete would have selected the mode that deletes, and said nothing. A node that refuses to start is a problem an operator sees; directories that are gone are not.

Examples

iex> Malachi.Config.retention_orphan_sweep("  Report ")
:report

iex> Malachi.Config.retention_orphan_sweep(nil)
:delete

sampling_ratio(raw)

@spec sampling_ratio(String.t() | nil) :: {:ok, float()} | :invalid

Normalizes the OpenTelemetry sampling ratio taken from MALACHI_TRACING_SAMPLE_RATIO.

Returns {:ok, ratio} for a number in 0.0..1.0, and :invalid for anything else, leaving the caller to warn and fall back. Absent is {:ok, 1.0}: tracing is opt-in, so a deployment that turned it on without naming a ratio asked to see everything.

Strict on purpose, unlike the lenient float parsing used for most settings. Float.parse/1 returns the leading number and discards the rest, so "0,1" would become 0.0 and trace nothing, while "ten" would fail to parse and take the default, tracing everything. Both are silent, and the second is silent in the direction that puts real work on a production node. Requiring the parse to consume the whole string is what separates a value from a typo.

Examples

iex> Malachi.Config.sampling_ratio(nil)
{:ok, 1.0}

iex> Malachi.Config.sampling_ratio("0.25")
{:ok, 0.25}

iex> Malachi.Config.sampling_ratio("0")
{:ok, 0.0}

iex> Malachi.Config.sampling_ratio("0,1")
:invalid

iex> Malachi.Config.sampling_ratio("ten")
:invalid

iex> Malachi.Config.sampling_ratio("1.5")
:invalid

segment_max_age_ms(raw)

@spec segment_max_age_ms(String.t() | nil) :: pos_integer()

How old an active segment may get before the retention sweep rolls it, from MALACHI_SEGMENT_MAX_AGE_MS: 7 days when the variable is absent or blank, otherwise a whole number of milliseconds from the field's floor (retention.segment_max_age_ms in Malachi.Cluster.Policy, one minute) through 2^64 - 1. Anything else stops the node, for the reasons retention_bound/2 gives and one more: below the floor, the sweep that asks for the roll runs too rarely to honor the value, and every roll is one more segment in the metadata.

Examples

iex> Malachi.Config.segment_max_age_ms(nil)
604_800_000

iex> Malachi.Config.segment_max_age_ms("3600000")
3_600_000