Grading Test Harness Flakiness with Explicit Verdicts

A CI run fails on test 318 with a 500-millisecond timeout. A developer clicks "Re-run failed jobs." The second run passes. The pull request turns green and merges.

Two weeks later, the service drops requests under peak load with the same timeout signature. The post-mortem logs the incident, noting that the test had historically been "flaky."

Calling a test flaky explains nothing. It describes an observation—intermittent results—while obscuring the cause. When verification tooling only allows binary pass/fail output, teams facing intermittent failures have two bad options: halt all deployments for an unrepeatable failure, or add automatic retries until the test passes. Most teams choose retries. In doing so, they convert harness noise into unverified code in production.

Why Retries Hide Harness Limits

When a test produces inconsistent outcomes, the failure originates in one of two places: the artifact under test or the test harness executing it.

If the artifact contains an unmanaged race condition, the intermittent failure is a genuine code defect. If the test harness relies on shared state, wall-clock sleeps, or unisolated network sockets, the failure is a harness limit.

Binary test runners cannot represent this distinction. A test that fails due to environmental jitter is reported as broken code. A test that passes on its third retry is reported as verified code.

## The standard retry pattern: green-washing harness debt
npm test -- --retries=3

This pattern green-washes verification debt. A retry loop does not make a test harness more accurate; it simply defines correctness as "succeeded at least once before giving up." If a race condition exists in the artifact, running it three times until it hits the happy path guarantees that defects bypass the gate.

When verification infrastructure collapses every run into 0 (pass) or 1 (fail), teams are forced to ignore real signals to keep pull requests moving.

Isolating Flakiness with Explicit Verdicts

The alternative to binary green-washing is grading the verification state honestly.

Instead of treating a flaky test as a mystery to be retried, record what actually happened. A certification harness should separate harness limits from artifact execution and invariant verification across four explicit states:

1. CERTIFIED: The harness exercised the input contract, all invariant assertions passed, and the execution is repeatable.
2. PROVISIONAL: The execution is reproducible, but correctness is not yet asserted.
3. INCONCLUSIVE: The harness could not exercise the input contract. This is a harness limit, not an artifact defect.
4. FAILED: An assertion was explicitly violated or an unhandled exception occurred under controlled execution.

This taxonomy—detailed further in the framework for the four verdicts between green tests and correct code—removes the incentive to lie about test results.

When a harness cannot bind a port or times out before running its assertions, the result is INCONCLUSIVE. The pipeline does not mark it green, nor does it log an artifact defect. When an artifact executes cleanly and repeatedly but lacks rigorous invariant assertions, it receives a PROVISIONAL verdict.

Treating unasserted runs as provisional and harness failures as inconclusive stops harness ambiguity from polluting certified release gates. You can inspect an execution in the live harness to see how isolating harness limits protects the validity of passing runs.

The Cost of Conflating Harness Limits with Code Defects

When teams do not separate harness failures from code failures, two failure modes occur:

First, developers stop trusting CI. Once a test suite develops a reputation for noise, engineers assume every failed run is an infrastructure glitch. Genuine verification failures get retried until they slip through.

Second, teams spend engineering sprints attempting to debug "bugs" in application code that are actually artifacts of poor harness isolation. As explored in when the harness fails instead of the code, spending developer hours rewriting clean domain logic because a mock server timed out is pure waste.

Explicit grading allows you to quarantine harness debt. A build with provisional components may proceed through local integration lanes, but it cannot pass a release gate that demands verified certification.

What to Do on Monday

You do not need to rewrite your entire test infrastructure overnight to stop flaky tests from degrading your release gates. Start with three tactical changes:

1. Remove blind retries. Stop setting global retry flags on entire test suites. If a test fails because of environmental setup, record it as an inconclusive harness limit rather than retrying it until it turns green.
2. Distinguish harness failures from assertion failures. When a mock times out or a shared port conflicts, log the failure against the harness infrastructure. Do not assign application engineers to debug domain code when the runner failed to execute the contract.
3. Require certification for production release gates. Allow provisional runs during early local assembly, but require fully asserted, certified verdicts before code merges to main. A green build produced by automated retries is not certified code.