Running the Node.js load test

View Source

scripts/loadtest.js drives load over the binary wire protocol using the reference Node client in scripts/lib/. It is the external view of the server: real TCP, real serialization, real authentication, from a runtime that is not the BEAM. That last part is the point of keeping it alongside the Elixir generator: a number produced by a client that shares nothing with the server is harder to fool yourself with.

It is single-threaded (one Node event loop), so it saturates a server with concurrency rather than with cores. Unpinned on your own machine, the multi-core Elixir generator pushes a server harder; the published ceiling instead pins both generators to a single core (see How the published ceiling is measured) so the Node-vs-Elixir comparison is fair rather than a contest of generator runtimes.

Before you start

It needs Node 22.15 or later, the first release whose zlib has zstd, which the payload checks in --self-test use. CI runs Node 22. The generator connects to an already running server; it does not start one.

MIX_ENV=dev mix run --no-halt

The default credentials are app / app123, which the dev config seeds with produce and consume permission. Host, port and credentials come from the environment, not from flags:

MALACHI_HOST=localhost MALACHI_PORT=4040 MALACHI_USER=app MALACHI_PASS=app123

The four scenarios

cd scripts

# produce: append batches as fast as the server accepts them
node loadtest.js --scenario produce --connections 20 --batch 10 --record-size 256

# fetch: read a backlog it seeds first
node loadtest.js --scenario fetch --prepopulate 50000 --max 200

# stream: server-pushed delivery with a credit window
node loadtest.js --scenario stream --connections 4 --window 500

# mixed: produce and fetch interleaved on the same connections
node loadtest.js --scenario mixed --connections 20 --record-size 512 --keys 1000

--keys sets key cardinality, which matters for anything that partitions by key. --prepopulate defaults to 10000 for the scenarios that need a backlog and to nothing for produce.

Payloads

By default every record value is one repeated byte, which is what every published series so far was measured with. Constant bytes compress to almost nothing, so no compression figure taken with them means anything. --payload picks what the values are, with the same names in both generators:

  • constant (default): the repeated byte.
  • json: a seeded, event-like JSON document (an id, a timestamp, an event type, a user, a region, a status, an amount and a short text) of exactly --record-size bytes. It needs at least 149 bytes; a smaller size is refused before any connection opens.
  • random: seeded noise, the incompressible control.

--payload-seed (0 to 4294967295, default 1) picks the values, and only json and random accept it. The same mode, seed and size give byte-identical values in the Elixir and the Node generator. The values are generated once, before the run starts, into a pool every connection shares: about 8MiB of values (capped at 65536 values, so less below 128-byte records), and never less than one batch per connection. Each connection starts on a batch of its own and walks forward from it, so no produce repeats a value inside its batch and a connection only sends a value again after it has sent the whole pool. Large batches at many connections make the pool larger: batch 4096 of 256-byte values at 64 connections holds 64MiB. Only the starts are kept apart: a connection that runs ahead of its neighbour sends a few of the same batches as the neighbour close together. How much a block compresses then depends on how many records it holds, which is --batch: with zstd at level 1 and 256-byte values, about 1.3x at one record per block and 3.8x at a thousand. The document, the generator and the pool are specified in Malachi.Loadtest.Payload.

A json or random run names its payload in the regime label (..., payload json)) and records payload and payload_seed beside batch and record_size in its JSON; a constant run's label reads exactly as it always did.

node loadtest.js --scenario produce --connections 20 --batch 100 --payload json --payload-seed 7

Closed loop and open loop

By default every connection runs operation -> await -> operation in a tight loop. That finds the ceiling: the server is always saturated, and the latency you get is latency at saturation.

--rate switches to a fixed arrival rate instead, and this is the mode to use when you care about latency rather than about the ceiling:

node loadtest.js --scenario produce --rate 20000 --connections 50 --duration 30

Requests are then fired on schedule regardless of whether earlier ones finished, and each one's latency is measured from the time it was scheduled, not from the time it was sent. That is what corrects coordinated omission: in a closed loop, a server stall stops new requests from being issued, so the stall hides itself and the percentiles come out flattering. Under --rate the requests that queued behind a stall carry its cost, which is what a client would actually experience. --max-inflight caps how far behind the generator will let itself fall, and hitting it flags the run as saturated rather than pretending the rate was met.

Recording a result

--json prints the report as a single JSON document with a meta block that records the command, the git ref, the version and the hardware, so the number stays interpretable months later:

node loadtest.js --scenario produce --connections 20 --duration 10 --batch 10 \
  --record-size 256 --json > /tmp/result.json

The Node.js load test results page renders exactly this document, from benchmark/published/loadtest-node.json, and so does the headline of the benchmark dashboard.

How the published ceiling is measured

You do not update that file by hand, and a hand-edit would not survive: the Publish results workflow measures it on a CI runner on every push to main and commits what it measured, so the site always shows the merged code rather than whichever laptop last captured a sample. It does not run the bare command above. It runs scripts/loadtest-ceiling.sh with GENERATOR=node, which boots a dedicated server pinned to three cores for every point, pins this generator to the fourth, and sweeps two axes: the batch size, then for each batch size the connection count. Each batch size's peak is its ceiling:

GENERATOR=node SRV_CPUSET=1,2,3 LT_CPUSET=0 OUT=/tmp/loadtest-node.json scripts/loadtest-ceiling.sh

The batch size is swept because the cost of a flush is a curve over its size, and it changes sign: segment preallocation makes a 2.5KB flush 69.7% faster and a 1MB flush 16.2% slower, crossing near 170KB. A ceiling at one batch size describes one slice of that. The default BATCH_LADDER is 10 100 512 1024 4096 at 256B records, which is 2.5KB, 25KB, 128KB, 256KB and 1MB of values per request, and the script forces group commit off so every request is its own flush. Larger requests saturate at fewer connections, so each batch size has its own connection ladder: CONNS_LADDER_<batch> when set, else a default in the script (32 to 512 for batch 10, down to 4 to 64 for batch 4096). The published result keeps the flat fields readers already use for HEADLINE_BATCH (10, the regime every earlier number describes), puts that regime into every headline, and carries the whole curve beside it.

Each point runs once (REPS=1, stated in the result), in an order that interleaves the batch sizes so a slow stretch on the runner does not read as the effect of one of them, and the headline peak is run a second time at the end. How far that A-A repeat moves is the noise floor the page prints next to the curve: compare batch sizes within one run, since the published ceiling has moved by more than 30% between CI runs of unchanged code. The whole sweep takes about 16 minutes per generator on a GitHub runner (14m54s and 15m11s on the two legs of the run that measured it), against the job's timeout-minutes: 30, which is there for a run that is stuck rather than slow. An invalid knob exits with status 2 before anything boots; a headline batch size with no clean point still writes the result and exits with status 1, which keeps it from being published.

The pinning is the whole point: on one runner an unpinned generator steals server CPU and flatters the number. Because the generator gets a single core, a low ceiling can be this client capping rather than the server, so every point samples both sides' CPU across its measured window and the published page reports them: a server near three of three cores saturated (its ceiling was found), a generator near one of one capped first (the point is a lower bound, and says so). The window starts when the generator creates the file named by --measure-marker, after every connection authenticated and the warmup ended; sampling from the spawn instead measured the server verifying credentials. The load drives a single topic, which in Malachi means a single range and a serialized append on its primary, so the published figure is the one-topic ceiling. If a batch size peaks at the top of its connection ladder the script warns to widen CONNS_LADDER_<batch>, since the knee may lie beyond it.

How connections are opened is part of the methodology. Every connection pays a server-side credential verification (Argon2, expensive by design), so opening hundreds simultaneously is an auth storm: on the 3-core CI server it exhausted the high rungs' client timeouts before a single record flowed. Both generators therefore take the same --connect-strategy flag: bounded (default; at most --connect-concurrency connects in flight, 32), stagger (connection i starts after i * --connect-stagger-ms), or all-at-once (the storm, kept for reproducing it on purpose). The ceiling harness pins bounded at 32 explicitly, and each pacing knob is only accepted with the strategy that reads it.

To see what a change does before merging, open a pull request: the workflow runs there too without committing, because runner variance would otherwise put noise in every diff. Node and Elixir run on separate runners so neither load test influences the other. It posts the numbers as a comment when the branch is in this repository; from a fork that step is skipped and the numbers come back as the run's artifacts. Run the script the same way locally when you want that ceiling; run the bare command above when you want a quick single-point measurement instead.

Write to a path outside the repository and move the file into place afterward, as above. Piping straight onto the tracked file truncates it first, which makes the working tree dirty while the run is in flight, and the recorded git ref then comes out as <sha>-dirty instead of naming a commit anyone can check out.

Reading the output

errors is the first number to look at: a throughput figure taken from a run with errors is a measurement of the failure, not of the server. A stream run also reports dropped and reconnects: a stream connection that is lost is counted there rather than in errors, and reconnected and resubscribed within the window as the Elixir generator does, so a stream run with no errors can still have been losing connections. Outside the stream scenario those two fields are null, since a lost connection there is counted in errors instead, once for every request that fails on it. --warmup excludes the opening seconds from the statistics, which matters because the first connections pay for topic creation and JIT warmup.

--self-test validates the latency histogram against a brute-force reference without a server, and the payload generator against the golden vectors the Elixir one is tested with, if you ever doubt the percentiles or the values themselves:

node loadtest.js --self-test