Seeding Randomness in Distributed Test Runners

The flaky failure mode

A test suite that runs on a distributed runner often crashes on a single commit and passes on the next. The culprit is usually a source of non‑determinism that leaks into every worker. Clocks, network ordering and, most insidiously, independent random streams all produce divergent behaviour. When a test creates data with Math.random() or a language‑specific RNG, each process seeds the generator from the current time. Two workers that start a few milliseconds apart will see different sequences. The test may pass on one worker and fail on another, giving the illusion of a flaky test.

Why a single seed matters

If every worker draws from the same pseudo‑random sequence, the test becomes repeatable. The seed is the only source of entropy; all downstream randomness is a deterministic function of that seed. This does not remove randomness from the test – it merely makes the randomness reproducible. The failure mode is not “randomness is bad”, it is “randomness is uncontrolled”. Controlling the seed eliminates the uncontrolled part while preserving the ability to explore edge cases.

Consider a property‑based test that generates a list of integers and asserts that sorting never throws. In a single‑process run the test passes. In a distributed run the list generated on worker 3 differs from the list on worker 7 because each worker seeded its RNG from Date.now(). Worker 7 occasionally produces a list that triggers a bug in the sort implementation, causing a flaky failure that appears only when the test is sharded.

Propagating the seed across workers

The simplest posture is to generate a seed once, before any worker starts, and pass it to every process. In a Node‑based CI pipeline the pattern looks like this:

// seed.ts – executed by the CI driver
import { randomBytes } from 'crypto';
const seed = randomBytes(4).readUInt32BE(0);
process.env.TEST_SEED = seed.toString();
console.log(`TEST_SEED=${seed}`);

The CI driver runs node seed.ts and then launches the test workers with the same environment. Each worker reads the seed and initialises its RNG in a reproducible way:

// rng.ts – imported by every test file
const seed = Number(process.env.TEST_SEED);
let state = (seed + Number(process.env.WORKER_ID || 0)) >>> 0;

export function random(): number {
  // xorshift32 – small, fast, repeatable
  state ^= state << 13;
  state ^= state >>> 17;
  state ^= state << 5;
  return (state >>> 0) / 0xffffffff;
}

WORKER_ID is supplied by the test runner (Jest, Mocha, etc.) and ensures that each worker steps through a distinct but deterministic subsequence. All test code that previously called Math.random() now imports random from rng.ts. The only change to the test suite is the import; the logical behaviour stays the same.

Because the seed is generated once, the entire run is repeatable. Running the same commit twice, with the same CI configuration, will produce identical random streams on every worker. If a failure appears, you can rerun the same seed to debug the exact scenario that caused the flake.

Verifying repeatability

Repeatability can be demonstrated without mandatory tooling. The SHPBL certification harness is an optional component that can execute a test artefact and record a verdict. When the harness observes identical outputs on two successive executions with the same seed, it may return CERTIFIED. If the harness cannot exercise the input contract— for example, because the test depends on external timing— it returns INCONCLUSIVE, indicating a harness limitation rather than a defect in the test itself. Using the harness is useful for formal evidence, but the seed‑propagation technique works independently of any verdict.

A concrete step for Monday

Add a tiny seed‑generation script to your CI pipeline. Export the seed as an environment variable, and modify your test harness to initialise a repeatable RNG from that variable. Replace direct calls to the language’s default random function with the seeded implementation. Commit the change, run the suite on a distributed runner, and verify that the same seed produces identical results on two consecutive runs. If the optional harness reports CERTIFIED, you have removed a major source of flaky tests without sacrificing the ability to explore random inputs.

The effort is a few lines of code and a single CI configuration tweak. The payoff is a test suite that behaves the same on every commit, no matter how many workers you throw at it. This small discipline removes a hidden source of non‑determinism and brings your builds closer to the repeatable, model‑independent posture that underpins reliable software delivery.