Use Provisional Harness Verdicts to Safely Promote Artifacts

Green test suites give a false sense of correctness

A recent merge caused a runtime crash in production. The CI pipeline reported all tests passed. The team assumed the artifact was safe to ship. The crash happened because the test suite never exercised a particular input shape. The failure was not a bug in the code; it was a gap in the verification process.

When a harness reports CERTIFIED, the artifact has been exercised end‑to‑end and the verdict is recorded. When the harness can only run the artifact but cannot assert correctness, it returns PROVISIONAL. Treating a provisional result as “green” blurs the line between “it compiles” and “it is correct”. The ladder of verdicts described in the pillar post [/blog/the-four-verdicts-between-green-tests-and-correct-code] makes that distinction explicit. Ignoring it leaves teams vulnerable to silent regressions.

Provisional verdict as a safety net

A provisional verdict means the harness executed the artifact, but the correctness checks are incomplete. The harness could not verify the output against a known oracle, or the test data set was insufficient. The verdict is still useful: it proves the artifact can be built and run without immediate failure. It also records the exact inputs that were exercised, providing a trace for later analysis.

Consider a library that parses CSV files. The harness runs the parser on a synthetic dataset of 10 rows. The parser does not crash, so the harness returns PROVISIONAL. Later, a real‑world file with 10 000 rows and a malformed header triggers an exception that the provisional run never saw. Because the provisional verdict is recorded, the team can see that the artifact passed the build intent gate but lacked full coverage. The verdict itself becomes a guardrail: it signals “run‑able but not fully vetted”.

Limited promotion while full certification runs

The key is to allow a provisional artifact to move through certain stages of the release pipeline, but to block it from reaching production until a CERTIFIED verdict arrives. This approach balances speed and confidence.

A typical CI flow can be adjusted as follows:

1. The build intent gate registers the artifact. Licensing and invariants are resolved. The artifact is stored in the catalog.
2. The harness runs immediately with a lightweight test matrix. It returns PROVISIONAL.
3. The CI system tags the artifact with the provisional stamp and permits it to be merged into a “staging‑only” branch. Automated integration tests that do not depend on full correctness can still run.
4. In parallel, a more exhaustive harness run is scheduled. It uses the full 40‑primitive matrix defined by the discovery‑engine vault. When this run finishes with CERTIFIED, a promotion job lifts the artifact to the release channel.
5. If the exhaustive run returns FAILED or INCONCLUSIVE, the promotion job aborts and alerts the team.

Because the provisional verdict is explicit, downstream jobs can query the catalog and decide whether to accept the artifact. The same method is used by the live run page [/harness], which shows the current verdict for each artifact. Teams that block release promotion on provisional verdicts (see [/blog/block-release-promotion-on-provisional-verdicts]) already benefit from this discipline.

What you can do on Monday

1. Open your CI pipeline definition. Locate the step that invokes the certification harness.
2. Add a conditional gate that checks the verdict field. If the verdict is PROVISIONAL, route the artifact to a “pre‑prod” branch instead of the main release branch.
3. Enable a scheduled job that triggers a full harness run nightly. Use the same API endpoint that the TypeScript client (@shpbl/sdk) provides, documented at [/api-access].

By the end of the week you will have a pipeline that promotes only after a CERTIFIED verdict, while still allowing work to continue on provisional builds. The result is faster feedback without the illusion of full correctness.