Seeding Randomness Across Distributed Test Workers

Flaky tests betray unsynchronised randomness

A test suite that runs in parallel on a CI orchestrator often fails sporadically. The failure is not a code defect; it is a timing artefact. Each worker process creates its own random number generator. The generators start from different seeds, usually derived from the system clock. The same test case can see a different sequence of values on each run. When the test logic depends on those values – for example, shuffling a list, generating IDs, or picking a timeout – the outcome varies. The result is a flaky test that passes on a local machine but fails intermittently in the CI pipeline.

The root cause is a lack of seed synchronisation. The problem is not unique to any language or framework. It is a fundamental property of distributed execution: every process inherits its own entropy source unless told otherwise. Without a shared seed, the test run is nondeterministic by design.

Propagating a single seed restores determinism

The remedy is simple: generate one seed in the CI orchestrator and pass it to every test worker. All workers must initialise their random number generators from that seed before any test code runs. When the seed is fixed, the sequence of pseudo‑random values is identical across workers and across CI runs. The test suite becomes repeatable; flaky failures disappear.

The approach works for any source of randomness that respects seeding – Math.random in JavaScript, random in Python, java.util.Random in Java, etc. It also works for libraries that expose a seed‑able generator, such as seedrandom for Node or numpy.random.Generator for Python. The key is to enforce the seed at the earliest possible point, before any module imports that might create their own generator.

A minimal implementation in a Node CI pipeline

Assume the CI system can set environment variables for each job. The orchestrator generates a 64‑bit integer seed and stores it in TEST_SEED. Each worker runs a small bootstrap script before the test runner starts.

## orchestrator step – generate a seed once
export TEST_SEED=$(node -e "console.log(Math.floor(Math.random()*2**53))")

// bootstrap.js – run before any test files are loaded
const seed = parseInt(process.env.TEST_SEED, 10);
if (Number.isNaN(seed)) {
  throw new Error('TEST_SEED is missing or invalid');
}
const seedrandom = require('seedrandom');
seedrandom(seed, { global: true }); // overrides Math.random globally

The CI configuration then invokes the test runner with the bootstrap script:

script:
  - node bootstrap.js && npm test

All subsequent calls to Math.random draw from the same deterministic stream. If a library creates its own RNG, it can be patched in the same bootstrap step, for example by setting crypto.randomBytes to a deterministic wrapper.

The same pattern applies to other runtimes. In Python, the orchestrator writes the seed to a file or environment variable, and the test runner imports a small module that calls random.seed(os.getenv('TEST_SEED')). In Java, a system property can be read by a JUnit @BeforeAll hook that initialises new Random(seed) and injects it where needed.

Integrating the seed into deterministic CI

The seed must be reproducible across the whole CI job. Record the seed in the job’s artefacts or logs so that a failed run can be replayed with the identical seed. If a test fails, rerun the job with the recorded seed to confirm that the failure is reproducible. If it is not, the flakiness has been eliminated.

When the CI orchestrator spawns multiple containers or pods, propagate the environment variable through the container runtime or pod spec. In Kubernetes, add env: [{ name: TEST_SEED, value: "1234567890" }] to each pod template. In GitHub Actions, use jobs.<id>.steps.env.TEST_SEED.

The technique complements other deterministic‑CI measures such as injecting deterministic time providers or pinning socket timeouts. The earlier you seed, the fewer hidden sources of entropy remain. For a broader view of why eliminating runtime variance matters, see the pillar post on model independence: /blog/why-model-independence-is-an-engineering-posture.

Monday’s concrete step

1. Add a step to your CI definition that generates a numeric seed and exports it as TEST_SEED.
2. Create a tiny bootstrap script that reads TEST_SEED and forces the global random generator to use it.
3. Wire the bootstrap script into the test command.
4. Log the seed value for every CI run.

Run the pipeline on a branch with flaky tests. The failures should stop. If they persist, the cause lies elsewhere – perhaps in network ordering or external services – and can be investigated independently.

By treating randomness as a configurable input rather than an implicit system property, you turn a source of nondeterminism into a controllable parameter. The test suite becomes repeatable, the CI pipeline becomes trustworthy, and the team regains confidence in automated verification.