Malachi.Cluster.SegmentRoll (malachi v0.22.7)

View Source

Pure decision for the age roll: which active segments are old enough that the retention sweep should ask for them to be sealed (the segment.ms of other logs).

Age retention (Malachi.Cluster.Retention) only ever sees sealed segments, and a segment seals by size on its own. A topic too quiet to fill :segment_max_bytes would never seal one, so it would never expire anything, however old its records got. Rolling the active segment once it is older than :segment_max_age_ms closes that gap: a record lives at most the roll interval plus the age bound, plus up to two sweep intervals (the sweep that asks for the roll and the one that expires the segment) and the fence's answer.

The limit for each range is its topic's effective :segment_max_age_ms, resolved exactly as the retention bounds are (Malachi.Cluster.Retention.effective/4): the topic's policy over the global value, nil (the policy turned it off) rolls nothing, and a topic bound to a policy name nothing defines rolls nothing either, as it expires nothing.

A segment is due when now_ms - opened_at >= limit. One with no opened_at (an apply that had no clock, or a topic export from before segments carried one, see Malachi.Metadata.apply/3) has no known age and is due at once: rolling it costs one seal, and the segment that follows it carries a real opening time. A segment map with no opened_at key at all comes from a Raft leader still on older code, during a rolling upgrade, and is left alone until the upgraded leader's replay stamps it.

Emptiness is not judged here. Only the fence knows where a segment ends, and a frontend's own offset counter is not the control plane's for a range other nodes produce to. An active segment exists only once a produce registered it, so an empty one is rare, and the zero-length seal it gets is one the read path already serves (Malachi.Broker); no successor is opened until the next produce, so it happens once.

Summary

Functions

The active segments that are due for an age roll at now_ms (epoch ms), at most one per range (a range has one write head). global_policy, policies and unresolved_max_age_ms are what Malachi.Cluster.Retention.expired/5 takes, resolved once per sweep by the caller.

Functions

due(metadata, now_ms, global_policy, policies, unresolved_max_age_ms \\ nil)

The active segments that are due for an age roll at now_ms (epoch ms), at most one per range (a range has one write head). global_policy, policies and unresolved_max_age_ms are what Malachi.Cluster.Retention.expired/5 takes, resolved once per sweep by the caller.