Eliminating Clock-Based Flakiness in Parallel Tests

A test suite runs green on a 16-core workstation in four seconds. In CI, across a 64-worker matrix running in parallel, two assertions fail intermittently: a token expiration test fires two milliseconds too early, and a rate-limiter test accepts a burst it should have rejected. Re-running the job passes.

The failure is not a bug in your business logic. It is an unmanaged call to the system clock.

When tests read the machine's wall clock directly, execution time ceases to be an isolated variable. Under CPU contention, garbage collection pauses, or parallel worker scheduling, elapsed real time varies unpredictably between statements. The result is flaky tests that erode trust in CI and block releases.

Eliminating this failure mode requires treating time like any other dependency: passed explicitly and controlled by the test runner.

The Cost of the Ambient Clock

Ambient time leaks into code through default APIs: Date.now(), time.Now(), System.currentTimeMillis(), or Instant.now(). These calls read the operating system's real-time clock.

Consider a simple token bucket rate limiter:

// Flaky: bound to the host clock
class RateLimiter {
  private lastRefill = Date.now();

  allow(tokens: number): boolean {
    const now = Date.now();
    this.refill(now - this.lastRefill);
    this.lastRefill = now;
    // ...
  }
}

In a test, verifying that the bucket refills after 100 milliseconds usually leads to a real-time sleep:

rateLimiter.allow(10);
await sleep(105);
assert.equal(rateLimiter.allow(1), true);

This pattern breaks in parallel execution for three reasons:

1. Jitter: If the CI worker stalls for 10 milliseconds during thread scheduling, the test passes. If it stalls for 94 milliseconds, the test might fail.
2. Speed: Real-time sleeps artificially slow the test suite. A suite with a thousand 50ms sleeps takes nearly a minute in serial execution.
3. Reproducibility: A failure captured at 23:59:59 UTC on the 31st of the month will not reproduce at 00:00:01 UTC on the 1st.

Just as unmanaged concurrency creates execution drift—which we explored regarding async execution jitter in concurrency tests—unmanaged time prevents deterministic testing.

Introducing a Virtual Clock

Deterministic time requires separating the measurement of duration from the passage of real time. This is done by passing a clock interface to any component that measures intervals, expires state, or schedules delayed work.

interface Clock {
  now(): number;
}

class VirtualClock implements Clock {
  private currentTime: number;

  constructor(initialTime = 0) {
    this.currentTime = initialTime;
  }

  now(): number {
    return this.currentTime;
  }

  advance(ms: number): void {
    this.currentTime += ms;
  }
}

The component accepts the clock via its constructor:

class RateLimiter {
  private lastRefill: number;

  constructor(private clock: Clock) {
    this.lastRefill = this.clock.now();
  }

  allow(tokens: number): boolean {
    const now = this.clock.now();
    this.refill(now - this.lastRefill);
    this.lastRefill = now;
    // ...
  }
}

The test advances the virtual clock explicitly. No real time elapses, no sleep is scheduled, and the test runs synchronously in microseconds:

const clock = new VirtualClock(1_000_000);
const limiter = new RateLimiter(clock);

limiter.allow(10);
clock.advance(100);
assert.equal(limiter.allow(1), true);

Whether the CI runner is under zero load or saturated across 64 cores, clock.advance(100) advances internal state by exactly 100 units. The test executes identically on every machine, every time.

Why Time Control Feeds Repeatable Builds

Eliminating ambient variance is central to repeatable builds. If a build or verification step reads ambient system state—whether that is the system clock, unpinned iteration order, or an external API—the output ceases to be repeatable.

This connects directly to our broader posture: why model independence is an engineering posture. When software depends on unmanaged runtime variance, correctness becomes statistical rather than guaranteed. Controlling time is the baseline for isolating runtime behavior so it can be verified conclusively.

How SHPBL Enforces Time Isolation

In our certification harness, every artifact submitted to the gauntlet is executed under strict isolation before receiving a verdict: CERTIFIED, PROVISIONAL, INCONCLUSIVE, or FAILED.

Components that handle scheduling, leases, or caching cannot reach out to host-level temporal APIs. They must accept explicit inputs or standard virtualised time structures. This discipline ensures that our verification results are reproducible across runs, underpinning the byte-for-byte integrity sealed in our root of trust.

What to Do on Monday

1. Audit clock calls: Search your repository for direct invocations of system time (Date.now(), time.Now(), System.currentTimeMillis()).
2. Add a linter rule: Use static analysis (such as ESLint's no-restricted-syntax or Go's forbidigo) to ban direct wall-clock reads outside of your system boundary or entrypoints.
3. Inject an explicit clock: Introduce a Clock interface to stateful modules that handle expiry, backoff, or rate limiting.
4. Remove test sleeps: Replace every sleep() or delay() call in your test suite with explicit virtual clock advancement.