Layered Verdicts: From Provisional to Certified in CI Pipelines

Provisional verdicts mask missing guarantees

A green test suite is often taken as proof that a change is safe. In practice the suite only checks that the code compiles and that the existing tests run without error. It does not confirm that the artifact respects its input contract or that edge‑case behaviour is correct. Teams that rely on that single rung of the certification ladder end up with a false sense of safety. When a regression slips through, the cost is re‑work, hot‑fixes and lost confidence.

The problem is not the tests themselves. It is the missing step that would move the artifact from a provisional verdict to a certified one. Without that step, CI pipelines stop at “the code builds”. The next sections describe how to extend the pipeline with a staged harness that upgrades the verdict only after additional checks pass.

Add a second rung: a CI harness that enforces code verification

A CI harness is a small, repeatable program that executes the artifact against a curated set of verification inputs. Unlike unit tests, the harness is designed to exercise the contract boundaries that matter to downstream consumers. The harness returns one of the four explicit verdicts defined in the SHPBL model: CERTIFIED, PROVISIONAL, INCONCLUSIVE, or FAILED.

When the harness reports PROVISIONAL, the artifact has passed the build intent gate and the basic test suite, but the verification inputs have not yet been exercised. That is the default state for most CI runs today. The key is to treat PROVISIONAL as a temporary status, not as a final sign‑off.

The next step is to add a verification stage that runs only when the provisional verdict is produced. The stage supplies a richer input matrix – the same 40‑primitive matrix used by the discovery‑engine vault – and records the outcome in the same harness. If the artifact satisfies all verification cases, the harness upgrades the verdict to CERTIFIED. If any case fails, the verdict becomes FAILED and the pipeline blocks promotion.

This approach mirrors the certification ladder described in the pillar post /blog/the-four-verdicts-between-green-tests-and-correct-code. It makes the ladder visible in CI: each commit climbs one rung, and only when the top rung is reached does the artifact become eligible for release.

Staged harness design – concrete example

Consider a library that parses JSON into a domain model. The provisional test suite checks that valid JSON strings produce objects without throwing. The verification harness adds three stages:

1. Boundary inputs – malformed JSON, missing fields, and unexpected types. The harness expects the library to reject these inputs with a specific error code.
2. Property‑based samples – automatically generated JSON structures that respect the schema but vary in depth and size. The harness checks that the output model satisfies invariants (e.g., non‑negative IDs).
3. Real‑world fixtures – a curated set of payloads taken from production logs. The harness validates that the library’s output matches a previously certified snapshot.

Each stage runs in isolation. If a stage cannot exercise the contract (for example, a fixture triggers an unimplemented code path), the harness records INCONCLUSIVE rather than FAILED. That signals a gap in the verification suite, not a defect in the artifact.

The harness itself is a repeatable artifact. Its source lives in version control, its checksum is published, and the same binary can be executed locally or via the SHPBL offline edition. Because the harness is model‑independent, its verdicts are computed rather than generated, guaranteeing repeatability across environments.

What to do on Monday

1. Identify one existing CI job that currently stops at “tests pass”. Add a step that invokes a verification harness for the artifact produced by that job.
2. Configure the harness to return the four explicit verdicts. Treat any PROVISIONAL result as a pipeline failure that requires a follow‑up verification run.
3. Record the verdict in the build metadata. Use that metadata to gate downstream promotion steps – only CERTIFIED builds may be deployed to staging.

By the end of the week you will have a CI pipeline that climbs the certification ladder instead of flattening it at green tests. The extra verification cost is bounded: the harness runs only when the provisional verdict is produced, and it can be parallelised across the same resources used for the test suite.

Adopting this staged approach removes the illusion of safety that a green suite provides. It gives teams a concrete path from provisional to certified, and it makes code verification an integral part of the delivery workflow.