Handling INCONCLUSIVE Harness Runs in Strict CI Pipelines

A CI runner fails because an isolated environment cannot generate an ephemeral TLS certificate with a specific curve. The service under test never launched. The database migrations never ran. The pull request turns red anyway.

The engineer assigned to triage spends an hour debugging business logic that was never executed.

The standard alternative is worse: a developer wraps the setup block in a broad try/catch, logs a warning, and lets the process exit zero. The build turns green. The team assumes the pull request passed verification, but the code path was completely skipped.

Binary CI models force test runners into this false dilemma. Either an infrastructure error halts deployment by pretending the code is broken, or the harness hides its own failure to keep the pipeline moving.

Isolating Harness Input Contracts

In rigorous code certification, a test execution is an attempt to construct an executable proof. To prove that a component satisfies its specification, the runner must supply inputs that conform to the component's preconditions.

When the runner cannot meet those preconditions, the failure belongs to the harness, not the artifact.

+-----------------------------------------------------------+
| Runner sets up fixture                                    |
+-----------------------------------------------------------+
       |
       +--> Preconditions met? 
               |
               +-- NO  --> [ INCONCLUSIVE ] (Harness limit)
               |
               +-- YES --> Execute component logic
                              |
                              +-- Invariant holds?
                                     |
                                     +-- YES --> [ CERTIFIED ]
                                     |
                                     +-- NO  --> [ FAILED ]

These boundaries form the basis of formal test harness verdicts. As detailed in our foundation on the four verdicts between green tests and correct code, an evaluation must distinguish between four distinct states: CERTIFIED, PROVISIONAL, INCONCLUSIVE, and FAILED.

An INCONCLUSIVE verdict occurs strictly when harness input contracts cannot be fulfilled. Common triggers include:

• A test fixture generator cannot satisfy a complex relational constraint within its timeout budget.
• An operating system dependency (such as a native kernel primitive or an isolated network namespace) is unavailable in the runner environment.
• Fuzzing or property generation rules produce zero valid inputs that match the precondition filters.

Marking these runs as FAILED creates noise and degrades developer trust in the pipeline. Marking them as passing corrupts the integrity of your certified software.

Routing INCONCLUSIVE in CI

Handling INCONCLUSIVE as an independent state allows pipelines to react proportionally without lying about what was verified.

Strict binary pipelines only have one axis: block or pass. A four-verdict pipeline decouples execution success from artifact safety.

## Example harness runner wrapper
run_certification_suite() {
  local target="$1"
  
  ./harness run "$target" --format=json > result.json
  local verdict
  verdict=$(jq -r '.verdict' result.json)

  case "$verdict" in
    CERTIFIED)
      echo "Artifact certified against full specification."
      exit 0
      ;;
    PROVISIONAL)
      echo "Artifact executed reproducibly; full specification not asserted."
      exit 0
      ;;
    INCONCLUSIVE)
      echo "Harness input contract unsatisfied: $(jq -r '.reason' result.json)"
      # Route to telemetry / non-blocking warning, or exit with a distinct code
      exit 2
      ;;
    FAILED)
      echo "Assertion failure in artifact logic: $(jq -r '.error' result.json)"
      exit 1
      ;;
  esac
}

By assigning distinct exit codes, your workflow engine can handle each outcome according to policy:

1. For non-critical branches or preview builds: An exit code of 2 (INCONCLUSIVE) records an alert in test telemetry and posts a neutral check run. It does not block the PR, but it explicitly denies CERTIFIED status to the build output.
2. For core releases and sensitive deployments: An INCONCLUSIVE verdict blocks promotion to production, but routes the incident to the platform or test infrastructure queue rather than the feature developer. The triage ticket states plainly: The harness failed to construct valid inputs for module X.
3. For manifest generation: Downstream systems inspecting the build artifact can read the harness metadata directly. See CI integration of harness verdicts to block unsafe deploys for patterns on recording these states in release manifests.

You can observe how these outcomes are separated in live execution on our public harness.

What to Change on Monday

Stop collapsing test setup errors into test assertion failures.

Audit your test base for fixtures that return early or silently suppress environment errors. Replace them with explicit harness checks:

• If an input precondition cannot be generated, abort the test run with an explicit INCONCLUSIVE status and the missing constraint details.
• Reserve test assertions (assert, expect) exclusively for verifying invariants on the artifact itself after all preconditions are verified.
• Update your CI configuration to accept exit code 2 as an unresolved environment state rather than a code defect.

When your test harness is honest about what it could not run, a green build means what it says.