Distinguishing Provisional from Certified Artifacts in CI

A test suite that exits zero confirms only one thing: the process did not panic or throw an unhandled error. It does not prove that the code does what you think it does, or that anyone checked its boundary conditions.

When your pipeline collapses all test results into a binary pass/fail, you lose the distinction between an artifact that runs cleanly and an artifact whose correctness has actually been demonstrated. Unverified code gets promoted to production simply because nothing crashed during execution.

To make artifact verification meaningful, you need an explicit ladder between compilation and proof. In our four-verdict model, the critical boundary sits between PROVISIONAL and CERTIFIED.

The Boundary Between Provisional and Certified

The four verdicts emitted by the certification harness represent discrete operational states:

1. CERTIFIED: The artifact satisfies its declared input contracts and invariants across the full target execution matrix.
2. PROVISIONAL: The artifact builds cleanly and runs repeatably, but its behavioral correctness has not yet been asserted by contract tests.
3. INCONCLUSIVE: The harness could not exercise the input contract due to an environment, fixture, or harness limitation.
4. FAILED: The artifact violated an invariant, threw an unexpected error, or broke an asserted contract.

A provisional verdict is not a failure. It is an honest statement of verification depth: the build is reproducible and the smoke execution passes, but nobody has written or run the invariant checks required for certification.

Treating provisional code as certified creates silent technical debt. Treating it as a hard failure blocks progress on early-stage components. Emitting the explicit verdict lets CI pipelines treat them differently.

Configuring the Certification Harness

The harness requires two inputs to evaluate an artifact: an execution target and a set of explicit contract assertions. If the artifact executes cleanly but no contract assertions are supplied (or if only basic execution smoke tests run), the harness emits PROVISIONAL.

Here is a minimal configuration for a module evaluated with @shpbl/sdk or via the local CLI:

{
  "artifact": {
    "name": "rate-limiter-token-bucket",
    "entry": "./dist/index.js",
    "checksum": "sha256:7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069"
  },
  "verification": {
    "smoke": {
      "command": "node ./dist/index.js --healthcheck",
      "expectedExitCode": 0
    },
    "contracts": [
      {
        "id": "refill-rate-invariance",
        "spec": "./contracts/refill-rate.contract.js",
        "requiredFor": "CERTIFIED"
      }
    ]
  }
}

When run against this configuration, the certification harness operates through deterministic stages:

1. Validate the artifact checksum matches the published build.
2. Run the smoke execution. If this fails, emit FAILED.
3. Locate and execute the contract assertions listed in verification.contracts.
4. If no contracts are defined, or if the contract suite is omitted from the current run, emit PROVISIONAL.
5. If all required contracts execute and pass, emit CERTIFIED.

CI Reporting Without Binary Flattening

Standard CI runners expect a binary exit code. If you map PROVISIONAL directly to non-zero, you break non-blocking discovery and utility builds. If you map it to 0, downstream jobs cannot distinguish baseline builds from verified packages.

The correct approach in CI reporting is to emit machine-readable metadata alongside the status, allowing downstream release gates to enforce the required verification rung:

- name: Run Certification Harness
  id: harness
  run: |
    shpbl-harness evaluate --config ./harness.config.json --output ./harness-report.json
    echo "verdict=$(jq -r .verdict ./harness-report.json)" >> $GITHUB_OUTPUT

- name: Gate Production Release
  run: |
    VERDICT="${{ steps.harness.outputs.verdict }}"
    echo "Harness Verdict: $VERDICT"
    
    if [ "$VERDICT" = "PROVISIONAL" ]; then
      echo "::warning ::Artifact is reproducible but uncertified. Internal/canary deploy only."
      exit 0
    elif [ "$VERDICT" != "CERTIFIED" ]; then
      echo "::error ::Artifact verification failed with verdict: $VERDICT"
      exit 1
    fi

This structure lets your pipeline ingest provisional artifacts for integration testing or canary environments while enforcing that only CERTIFIED artifacts reach critical production gates. You can inspect how these runs execute in real time on the live harness view. For multi-tier environments, see how to handle layered verdicts in CI.

What to Do on Monday

Do not rewrite your entire test suite to build formal contracts. Start by categorizing what you actually have.

1. Pick one shared internal library that currently has "passing" tests.
2. Separate your tests into smoke tests (did it run?) and contract tests (did it uphold an invariant under edge inputs?).
3. If all you have are smoke tests, update your pipeline metadata to label the output PROVISIONAL instead of treating it as fully verified.
4. Add a single contract assertion that checks your most critical boundary condition, and use it as the threshold for CERTIFIED.

Naming your verification level honestly prevents unasserted code from passing as correct software.