Treat Provisional Verdicts as Actionable Signals in CI

Green suites give a false sense of safety

Most CI pipelines stop at “all tests pass”. The green bar tells you the code compiles and the unit suite runs, but it says nothing about whether the artifact actually does what its contract requires. When a change slips through a green suite, the next failure is often discovered in production, not in the build. The missing piece is an explicit check that the artifact behaves correctly under the conditions the contract defines.

A provisional verdict is an actionable indicator

The certification harness runs the artifact against a curated set of input contracts. A PROVISIONAL result means the harness could execute the artifact reproducibly, but it has not yet proved full correctness. The verdict is not a failure; it is a measured statement of coverage. It tells you that the artifact is reachable and stable enough to be exercised, yet the contract validation is incomplete. This distinction is critical: a failed verdict signals a defect, while a provisional verdict signals a gap in verification.

Treating the provisional verdict as a first‑class signal aligns with the ladder of verdicts described in the pillar post /blog/the-four-verdicts-between-green-tests-and-correct-code. That ladder exists to prevent teams from conflating “compiles” with “correct”. By surfacing the provisional state, you make the gap visible rather than assuming it does not exist.

Surface the provisional verdict in CI

The CI job that runs the certification harness should publish the verdict as a separate status check. In GitHub Actions, for example, the step can emit a JSON payload that the workflow interprets:

jobs:
  certify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run certification harness
        id: harness
        run: |
          result=$(shpbl-harness run ./artifact)
          echo "verdict=$result" >> $GITHUB_OUTPUT
      - name: Annotate PR
        if: steps.harness.outputs.verdict == 'PROVISIONAL'
        run: |
          echo "::warning file=artifact.ts,line=1::Provisional verdict – contract coverage incomplete"

The ::warning annotation makes the provisional verdict appear alongside the green test check, without breaking the merge. A dashboard that aggregates these annotations can colour‑code the status: CERTIFIED (green), PROVISIONAL (amber), INCONCLUSIVE (grey), FAILED (red). The harness documentation /harness explains the exact output format, making it straightforward to parse.

Direct test investment using the provisional signal

Once the CI pipeline highlights provisional verdicts, the next step is to decide where to invest testing effort. Because a provisional verdict guarantees reproducibility, you know the artifact can be exercised reliably. The missing piece is the contract validation. Use the following workflow:

1. Identify the input contracts that the harness did not exercise. The harness log lists the untested matrix entries; these are the immediate gaps.
2. Write property‑based or integration tests that target those specific contracts. Because the harness already proved the artifact runs without crashing, the new tests can focus on behavioural assertions.
3. Re‑run the harness after the new tests are added. If the verdict upgrades to CERTIFIED, the investment paid off; if it remains provisional, repeat the gap analysis.

This loop turns a vague “we need more tests” into a concrete, data‑driven plan. It also prevents over‑testing unrelated code paths, keeping test suites lean while improving confidence.

Monday’s concrete step

Add a new CI job that runs the certification harness and publishes the verdict as a warning when it is provisional. Commit the snippet above, adjust the harness command to your artifact, and watch the PR checks show a provisional annotation alongside the existing green test check. That single change makes the missing verification visible and gives you a clear target for the next round of test investment.