Handling Inconclusive Harness Runs in CI Pipelines

The problem that shows up in CI

A CI job reports a red build because the test harness could not construct a valid input for a library function. The code under test has never been executed with that input. The failure is recorded as a test failure, yet the artifact itself may be perfectly sound. Teams spend hours chasing a bug that does not exist, only to discover that the harness hit a contract limit it cannot satisfy.

Why “fail” or “pass” misleads

The certification harness defines four verdicts: CERTIFIED, PROVISIONAL, INCONCLUSIVE, and FAILED. When a run lands in INCONCLUSIVE, the harness signals that it could not exercise the input contract. Treating that verdict as FAILED conflates a harness limitation with a defect in the artifact. Treating it as PASS hides a gap in verification, giving a false sense of correctness.

Both approaches corrupt the signal that CI is supposed to provide. A red build that actually reflects a missing test case is as unhelpful as a green build that skips a dangerous edge case. The ladder of verdicts described in the pillar post [/blog/the-four-verdicts-between-green-tests-and-correct-code] exists precisely to keep these states distinct.

Distinguish harness limits from artifact defects

The first step is to make the harness emit INCONCLUSIVE explicitly. The harness already records this state when it cannot satisfy the input contract; the CI configuration must surface it instead of mapping it to a generic failure code.

## Example snippet for a GitHub Actions step
- name: Run certification harness
  run: ./harness run --artifact ${{ matrix.artifact }}
  continue-on-error: true
- name: Record verdict
  run: |
    verdict=$(cat verdict.txt)
    echo "Verdict=$verdict" >> $GITHUB_ENV
    if [ "$verdict" = "INCONCLUSIVE" ]; then
      echo "::warning::Harness could not construct input – review contract"
    elif [ "$verdict" = "FAILED" ]; then
      echo "::error::Artifact defect detected"
      exit 1
    fi

In this pattern, INCONCLUSIVE produces a warning, not a failure. The CI run remains green, but the pipeline logs contain a clear marker that the harness hit a contract limit. Engineers can filter on that marker to separate infrastructure gaps from real defects.

Next, audit the contract definitions that the harness uses. Contracts are expressed as a matrix of primitive constraints. When a particular combination proves impossible to generate, it usually indicates an overly strict or under‑specified contract. Adjust the contract to either relax the impossible combination or provide a custom generator for that edge case. This step is a one‑off effort per contract, after which future runs will move from INCONCLUSIVE to PROVISIONAL or CERTIFIED.

Finally, document the distinction in the team’s testing policy. State that INCONCLUSIVE is a signal to improve the harness, not a defect report. Align the policy with the certification harness’s own definition of the verdicts, and make the policy visible in the repository’s README.

Making the signal actionable

1. Configure CI to surface INCONCLUSIVE – add a step that reads the harness verdict and emits a warning instead of failing the job.
2. Log the contract that caused the inconclusive run – the harness already records the offending input matrix; ensure that log is searchable.
3. Assign a backlog item to the contract – create a ticket titled “Resolve INCONCLUSIVE on X artifact” and link the log.
4. Iterate on the contract – add a generator or relax the constraint, then re‑run the harness locally until the verdict upgrades.

By treating the harness’s limits as a separate class of outcome, the CI pipeline regains its purpose: to tell you when the code fails, not when the test infrastructure fails.

Monday action: Open the CI configuration for the next failing harness run, add the continue-on-error pattern shown above, and create a ticket for the first INCONCLUSIVE verdict you see. This small change will immediately stop mis‑classifying harness limits as code defects and give you a concrete backlog item to improve verification.