Defining Incremental Correctness Levels with Harness Verdicts

The illusion of a green test suite

Teams often equate a passing test run with “the code works”. A green suite only proves the test harness could execute the artifact. It says nothing about whether the artifact satisfies its functional contract. When a change breaks an edge case that no test covers, the suite stays green and the defect slips into production. The failure mode is clear: confidence is built on a single verdict – “the code runs” – while the actual guarantee is unknown.

A ladder of guarantees

The harness can return one of four outcomes: PROVISIONAL, CERTIFIED, INCONCLUSIVE, and FAILED. The first two form a ladder of guarantees.

• PROVISIONAL – the harness exercised the artifact against a defined input contract and observed reproducible behaviour. The result is repeatable, but correctness has not been asserted. The verdict signals that the artifact is stable enough for integration testing, but it still requires additional validation.

• CERTIFIED – the harness exercised the artifact and verified that its output matches the contract’s specification. The verdict is a formal guarantee that the artifact behaves as intended for the covered cases.

• INCONCLUSIVE – the harness could not exercise the contract. This is a limitation of the test harness, not a defect in the artifact.

• FAILED – the harness exercised the contract and observed a breach or a runtime error.

Mapping these outcomes to a ladder makes the gap between “it runs” and “it is correct” explicit. Teams can see at a glance whether an artifact is merely runnable, stable, or formally verified.

Surface the ladder in CI

CI dashboards traditionally display a single “green” or “red” status. Replace that binary indicator with the harness verdict. When a pull request triggers the harness, the CI job should publish one of the four verdict labels alongside the usual pass/fail flag. A typical flow looks like this:

1. The build stage produces a compile artifact. Success is a prerequisite but does not generate a harness verdict.
2. The harness stage runs the artifact against the contract. If it finishes without contract violations, the job records PROVISIONAL.
3. A separate certification stage runs a stricter contract suite. Success upgrades the label to CERTIFIED.

If the harness cannot exercise the contract, the job records INCONCLUSIVE and links to the live run page /harness. If a contract violation occurs, the job records FAILED. Because the verdicts are independent of the test suite, a green test run no longer masks missing functional guarantees. Engineers can set branch‑protection rules that require at least PROVISIONAL before merging, or reserve CERTIFIED for release candidates. The approach is described in detail in the pillar post /blog/the-four-verdicts-between-green-tests-and-correct-code.

Take the first step on Monday

On Monday, add a CI job that runs the harness and records its verdict as a custom status label. Use the existing harness endpoint; no new tooling is required. Then update your branch‑protection rule to reject merges that do not reach PROVISIONAL. This small change makes the correctness ladder visible to the whole team and forces the next iteration of testing to move beyond “just runs”. The result is a dashboard that tells you exactly what guarantee each artifact carries, and a development rhythm that treats “green” as a necessary, not sufficient, condition for correctness.