Malachi. Config
(malachi v0.22.7)
View Source
Helpers for normalizing operator-supplied runtime configuration.
Extracted from config/runtime.exs so the rules can be tested directly. That file is skipped
entirely under config_env() == :test, so anything defined inline in it is unreachable from the
suite; a pure function here is not.
checked/4 is the other half of the same idea, applied where the value is finally used rather than
where it is parsed: the config layer turns an environment variable into a term without judging it,
and the process that needs it decides whether the term can do the job.
Summary
Functions
value when valid? accepts it, otherwise default, saying out loud which setting was refused.
The cluster's identity as GET /api/v1/me reports it, from MALACHI_CLUSTER_DISPLAY_NAME,
MALACHI_CLUSTER_COLOR and MALACHI_CLUSTER_ICON. Each is nil when absent or blank. The color is a
#RRGGBB hex value and the icon a short slug the console maps to an icon (^[a-z0-9-]{1,32}$);
anything else stops the node, since a console that cannot render the identity defeats its purpose,
which is telling clusters apart at a glance. Every node should carry the same values.
Normalizes an on-disk data directory taken from an environment variable.
The default users in MALACHI_DEFAULT_USERS: entries separated by ;, each
user:password:permissions[:role], permissions separated by , and possibly empty (an operator with a
console role and no wire permission, ops:secret::viewer). Returns {username, password, permissions, role} tuples, role nil when absent. An unknown permission or role, an empty username or password, or
an entry of the wrong shape stops the node: a typo here would otherwise seed an account that cannot do
what the operator meant, or one that silently can do more, such as an admin with an empty password. A
password cannot contain : or ;, the separators. Every error names the entry by its position alone, its
place among the ; separated segments with empty ones counted, and prints no field, the username
included, since a mistyped separator can move part of a password into any of them.
The number in raw as a float, or default when the variable is absent or blank. Anything else
raises, for the reason given in integer/3.
The whole number in raw, or default when the variable is absent or blank. Anything else raises,
so the node refuses to boot.
The control plane cluster this node runs, from MALACHI_LOG_CLUSTER and the data-plane shard count.
Normalizes the ra machine version pin taken from MALACHI_RA_MACHINE_VERSION.
A retention bound from raw: nil when the variable is absent or blank (the rule is off), otherwise a
whole number from 0 through 2^64 - 1. Anything else stops the node, for the reason integer/3 gives
and two more. A negative age or byte budget expires everything the rule can see. A bound above
2^64 - 1 is stored whole but travels in 64 bits (Malachi.Wire), so get_topic_policy would report
it truncated, 2^64 as a zero budget, while the sweep applies the real value.
Normalizes the orphan sweep mode taken from MALACHI_RETENTION_ORPHAN_SWEEP.
Normalizes the OpenTelemetry sampling ratio taken from MALACHI_TRACING_SAMPLE_RATIO.
How old an active segment may get before the retention sweep rolls it, from
MALACHI_SEGMENT_MAX_AGE_MS: 7 days when the variable is absent or blank, otherwise a whole number of
milliseconds from the field's floor (retention.segment_max_age_ms in Malachi.Cluster.Policy, one
minute) through 2^64 - 1. Anything else stops the node, for the reasons retention_bound/2 gives and
one more: below the floor, the sweep that asks for the roll runs too rarely to honor the value, and
every roll is one more segment in the metadata.
Functions
value when valid? accepts it, otherwise default, saying out loud which setting was refused.
Environment variables reach a process already parsed but not judged: MALACHI_SCRUB_INTERVAL_MS=0
is a valid integer and a busy loop, and MALACHI_RETENTION_SKIP_LEDGER_MAX=0 is a valid integer and
a FunctionClauseError inside a server the application supervisor starts. Refusing to boot over an
operator's typo turns one lost knob into a lost node, so the documented default is used and the line
names the setting, what arrived and what is being used instead.
setting appears in the log, so it should be the name the operator can act on.
Examples
iex> Malachi.Config.checked(5_000, :scrubber_interval, 60_000, &(is_integer(&1) and &1 > 0))
5_000
@spec cluster_identity(String.t() | nil, String.t() | nil, String.t() | nil) :: %{ name: String.t() | nil, color: String.t() | nil, icon: String.t() | nil }
The cluster's identity as GET /api/v1/me reports it, from MALACHI_CLUSTER_DISPLAY_NAME,
MALACHI_CLUSTER_COLOR and MALACHI_CLUSTER_ICON. Each is nil when absent or blank. The color is a
#RRGGBB hex value and the icon a short slug the console maps to an icon (^[a-z0-9-]{1,32}$);
anything else stops the node, since a console that cannot render the identity defeats its purpose,
which is telling clusters apart at a glance. Every node should carry the same values.
Examples
iex> Malachi.Config.cluster_identity(" prod-eu ", "#A1B2C3", "globe")
%{name: "prod-eu", color: "#A1B2C3", icon: "globe"}
iex> Malachi.Config.cluster_identity(nil, "", nil)
%{name: nil, color: nil, icon: nil}
Normalizes an on-disk data directory taken from an environment variable.
Returns the trimmed path, or nil when the value is absent or blank (including all-whitespace), in
which case the caller falls back to its own default. In :prod a relative path is rejected rather
than accepted: it resolves against the process working directory, so the node would write its durable
segments to ephemeral container storage instead of the mounted volume, and the loss would only surface
on the next restart. Other environments stay permissive, so a relative path is fine for local work.
var names the source variable and appears in the error, so the operator sees which one to fix.
Examples
iex> Malachi.Config.data_dir("MALACHI_LOG_DATA_DIR", "/mnt/vol/log", :prod)
"/mnt/vol/log"
iex> Malachi.Config.data_dir("MALACHI_LOG_DATA_DIR", " ", :prod)
nil
iex> Malachi.Config.data_dir("MALACHI_LOG_DATA_DIR", "data/log", :dev)
"data/log"
@spec default_users(String.t()) :: [ {String.t(), String.t(), [atom()], Malachi.Auth.ConsoleRole.t() | nil} ]
The default users in MALACHI_DEFAULT_USERS: entries separated by ;, each
user:password:permissions[:role], permissions separated by , and possibly empty (an operator with a
console role and no wire permission, ops:secret::viewer). Returns {username, password, permissions, role} tuples, role nil when absent. An unknown permission or role, an empty username or password, or
an entry of the wrong shape stops the node: a typo here would otherwise seed an account that cannot do
what the operator meant, or one that silently can do more, such as an admin with an empty password. A
password cannot contain : or ;, the separators. Every error names the entry by its position alone, its
place among the ; separated segments with empty ones counted, and prints no field, the username
included, since a mistyped separator can move part of a password into any of them.
Examples
iex> Malachi.Config.default_users("admin:s3cret:admin;ops:pw::viewer")
[{"admin", "s3cret", [:admin], nil}, {"ops", "pw", [], :viewer}]
The number in raw as a float, or default when the variable is absent or blank. Anything else
raises, for the reason given in integer/3.
Float.parse/1 truncates the same way, and a decimal comma is the everyday case: 0,5 arrives as
0.0, which as a memory threshold means the alarm fires immediately and forever.
Examples
iex> Malachi.Config.float("MALACHI_X", nil, 0.7)
0.7
iex> Malachi.Config.float("MALACHI_X", "0.5", 0.7)
0.5
The whole number in raw, or default when the variable is absent or blank. Anything else raises,
so the node refuses to boot.
The counterpart of checked/4, and what separates them is what being wrong costs. checked/4 judges
a value that parsed: MALACHI_SCRUB_INTERVAL_MS=0 is a number, the operator's intent is legible, and
the documented default is a reasonable stand-in, so refusing to boot there would turn one lost knob
into a lost node. This one judges whether there is a number at all.
Integer.parse/1 keeps the leading digits and discards the rest, which is worse than refusing and
worse than defaulting: 600_000 arrives as 600 and 10m as 10, values the operator never wrote
and that no default can stand in for, because nothing here knows what was meant. The ones that hurt
are the ones that come out smaller. MALACHI_RETENTION_MAX_AGE_MS=7_776_000_000 would expire
everything sealed more than seven milliseconds ago, on every replica, with no way back.
var is the environment variable's own name, because that is what the operator has to go and fix.
Examples
iex> Malachi.Config.integer("MALACHI_X", nil, 5)
5
iex> Malachi.Config.integer("MALACHI_X", " ", 5)
5
iex> Malachi.Config.integer("MALACHI_X", " 12 ", 5)
12
The control plane cluster this node runs, from MALACHI_LOG_CLUSTER and the data-plane shard count.
- A name given: that cluster, whatever the shard count (a clustered node runs one data-plane
shard,
Malachi.DataPlaneRouter.shard_count/0). - No name and one shard (the default):
:malachi_log, so a single node runs its metadata as a one-memberracluster and keeps it across a restart. Before #273 this wasnil, in-memory metadata that a restart forgot, after which the orphan sweep deleted every segment written before the boot. - No name and more than one shard:
nil, the in-memory data-plane sharding measurement mode, whose shards hold their metadata in memory only and run no orphan sweep.
The name comes from a trusted operator (deploy config), so creating the atom is fine.
Examples
iex> Malachi.Config.log_cluster(nil, 1)
:malachi_log
iex> Malachi.Config.log_cluster(" ", 4)
nil
iex> Malachi.Config.log_cluster("orders_meta", 4)
:orders_meta
@spec ra_machine_version_pin(String.t() | nil) :: non_neg_integer() | nil
Normalizes the ra machine version pin taken from MALACHI_RA_MACHINE_VERSION.
Returns nil when the value is absent or blank (no pin: the node advertises the newest version its
code implements) and the integer when it is a whole non-negative number. Anything else raises, so the
node refuses to boot. Silently ignoring a typo would be the worst outcome here: an operator who pinned
the version to keep a rollback possible would find the pin gone and the upgrade finalized, which
moves the rollback floor. See Malachi.Cluster.MachineVersion for what the pin holds back.
Examples
iex> Malachi.Config.ra_machine_version_pin(nil)
nil
iex> Malachi.Config.ra_machine_version_pin(" ")
nil
iex> Malachi.Config.ra_machine_version_pin(" 1 ")
1
@spec retention_bound(String.t(), String.t() | nil) :: non_neg_integer() | nil
A retention bound from raw: nil when the variable is absent or blank (the rule is off), otherwise a
whole number from 0 through 2^64 - 1. Anything else stops the node, for the reason integer/3 gives
and two more. A negative age or byte budget expires everything the rule can see. A bound above
2^64 - 1 is stored whole but travels in 64 bits (Malachi.Wire), so get_topic_policy would report
it truncated, 2^64 as a zero budget, while the sweep applies the real value.
Examples
iex> Malachi.Config.retention_bound("MALACHI_RETENTION_MAX_BYTES", nil)
nil
iex> Malachi.Config.retention_bound("MALACHI_RETENTION_MAX_BYTES", "10737418240")
10737418240
@spec retention_orphan_sweep(String.t() | nil) :: :delete | :report | :off
Normalizes the orphan sweep mode taken from MALACHI_RETENTION_ORPHAN_SWEEP.
nil or a blank value is :delete, the documented default. delete, report and off are
accepted whatever their case and surrounding whitespace. Anything else raises.
Raising rather than falling back is the same rule as ra_machine_version_pin/1, and for the same
reason: this is a knob whose wrong value is not a lost knob. checked/4 exists for an interval,
where the cost of refusing the value is one cadence; here MALACHI_RETENTION_ORPHAN_SWEEP=Report
from an operator who meant NOT to delete would have selected the mode that deletes, and said
nothing. A node that refuses to start is a problem an operator sees; directories that are gone are
not.
Examples
iex> Malachi.Config.retention_orphan_sweep(" Report ")
:report
iex> Malachi.Config.retention_orphan_sweep(nil)
:delete
Normalizes the OpenTelemetry sampling ratio taken from MALACHI_TRACING_SAMPLE_RATIO.
Returns {:ok, ratio} for a number in 0.0..1.0, and :invalid for anything else, leaving the
caller to warn and fall back. Absent is {:ok, 1.0}: tracing is opt-in, so a deployment that turned
it on without naming a ratio asked to see everything.
Strict on purpose, unlike the lenient float parsing used for most settings. Float.parse/1 returns
the leading number and discards the rest, so "0,1" would become 0.0 and trace nothing, while
"ten" would fail to parse and take the default, tracing everything. Both are silent, and the
second is silent in the direction that puts real work on a production node. Requiring the parse to
consume the whole string is what separates a value from a typo.
Examples
iex> Malachi.Config.sampling_ratio(nil)
{:ok, 1.0}
iex> Malachi.Config.sampling_ratio("0.25")
{:ok, 0.25}
iex> Malachi.Config.sampling_ratio("0")
{:ok, 0.0}
iex> Malachi.Config.sampling_ratio("0,1")
:invalid
iex> Malachi.Config.sampling_ratio("ten")
:invalid
iex> Malachi.Config.sampling_ratio("1.5")
:invalid
@spec segment_max_age_ms(String.t() | nil) :: pos_integer()
How old an active segment may get before the retention sweep rolls it, from
MALACHI_SEGMENT_MAX_AGE_MS: 7 days when the variable is absent or blank, otherwise a whole number of
milliseconds from the field's floor (retention.segment_max_age_ms in Malachi.Cluster.Policy, one
minute) through 2^64 - 1. Anything else stops the node, for the reasons retention_bound/2 gives and
one more: below the floor, the sweep that asks for the roll runs too rarely to honor the value, and
every roll is one more segment in the metadata.
Examples
iex> Malachi.Config.segment_max_age_ms(nil)
604_800_000
iex> Malachi.Config.segment_max_age_ms("3600000")
3_600_000