CI Gate on Provisional Harness Verdicts with Optional Override

When a green test suite hides a provisional verdict

Your CI pipeline reports “all tests passed”. The build artefact is published, downstream services start using it, and a subtle bug later surfaces because the certification harness only marked the artefact PROVISIONAL. The failure mode is clear: a provisional verdict is treated as a pass, so the risk of an incomplete or incorrect implementation leaks into production.

The four‑verdict model (see the pillar post /blog/the-four-verdicts-between-green-tests-and-correct-code) makes the distinction explicit. A PROVISIONAL result means the harness could execute the artefact and reproduce its output, but correctness has not been asserted. Treating that as a green signal defeats the purpose of the harness.

Align CI gating with the harness outcome

The simplest way to keep the ladder honest is to make the harness the gatekeeper, not the unit‑test suite. In a typical CI job you can invoke the certification harness after the unit tests and fail the job if the verdict is anything other than CERTIFIED.

## .github/workflows/ci.yml
name: CI
on: [push, pull_request]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install dependencies
        run: npm ci
      - name: Run unit tests
        run: npm test
      - name: Run certification harness
        env:
          SHPBL_TOKEN: ${{ secrets.SHPBL_TOKEN }}
        run: |
          npx @shpbl/sdk run-harness ./dist/artifact.js \
            --output json > harness.json
      - name: Enforce verdict
        run: |
          VERDICT=$(jq -r .verdict harness.json)
          if [ "$VERDICT" != "CERTIFIED" ]; then
            echo "Build stopped: harness returned $VERDICT"
            exit 1
          fi

The run-harness command is a thin wrapper around the same certification harness you would invoke locally or via the HTTP API. By checking the verdict field, the pipeline halts promotion whenever the harness returns PROVISIONAL, INCONCLUSIVE, or FAILED.

Handling provisional artefacts when release is unavoidable

There are cases where a team must ship an artefact despite a provisional verdict—e.g. a time‑critical feature with low‑risk impact. SHPBL does not provide a built‑in “override” flag. Instead, the decision must be recorded outside the harness, typically in your own change‑control system or release notes.

A common pattern is:

1. Run the harness and capture the verdict.
2. If the verdict is PROVISIONAL, require a manual approval step before the artefact is promoted.
3. Record the approval decision in a separate audit log (e.g. a ticket, a commit comment, or a deployment manifest) that references the provisional status.

This keeps the technical gate (the harness) separate from the policy gate (human approval) without relying on undocumented harness behaviour.

Integrating the policy into your CI

1. Define a policy – decide which artefacts may be released with a provisional verdict and under what conditions (e.g., internal services only, after a documented sign‑off).
2. Add a manual approval step – in GitHub Actions you can use environment protection rules or workflow_dispatch to require a reviewer before proceeding past the “Enforce verdict” step.
3. Record the decision – write the approval outcome to your own audit store (a changelog file, a ticket reference, or a metadata artifact) so auditors can trace the risk acceptance.

By separating the technical gate (the harness) from the policy gate (manual approval), you retain a clear audit trail and avoid silently drifting the ladder.

Monday’s concrete step

1. Open your CI definition file.
2. Insert the “Run certification harness” and “Enforce verdict” steps as shown above.
3. Add a manual‑approval checkpoint (e.g. an environment protection rule) that blocks promotion when the verdict is not CERTIFIED.
4. Document in your release process how provisional approvals are recorded.

Commit the change, push to a feature branch, and watch the CI job fail on a provisional verdict. You now have a CI gate that respects the certification harness, and a documented path for intentional risk acceptance. The next sprint can focus on turning those provisional artefacts into fully CERTIFIED ones.