Handling Inconclusive Test Harness Runs in Merge Queues

A merge queue running an automated verification gauntlet eventually hits a scenario it cannot set up. A mock fails to initialise because of an unhandled parameter type. A property-based generator cannot satisfy a cross-field constraint. The fixture builder throws an exception before the code under test executes a single instruction.

In most pipelines, this emits an exit code of 1. The queue evicts the pull request, marks the branch red, and notifies the author that their change is broken.

The author pulls the branch, inspects the log, and discovers the implementation is sound; the test harness choked on generating a valid input. The author now spends three hours debugging fixture code rather than feature code, or worse, disables the test to get past the queue.

The alternative response is just as damaging. Teams tired of harness crashes catch the setup exception, log a warning, and exit 0. The queue merges the branch. A green build enters the repository history, claiming the change passed verification when the code was never executed at all.

Harness Limits Are Not Code Defects

A binary pass/fail status forces harness infrastructure to pretend it knows things it does not. Real verification operates on a spectrum. As detailed in our breakdown of the four verdicts between green tests and correct code, a system must distinguish between code failing an assertion and the harness failing to construct the input contract.

When an input contract cannot be satisfied, the verdict is INCONCLUSIVE.

{
  "target": "pkg/auth/token.Validate",
  "verdict": "INCONCLUSIVE",
  "reason": "CONTRACT_GENERATION_FAILED",
  "detail": "Generator could not resolve recursive claims schema within depth limit 4",
  "executed_assertions": 0
}

An inconclusive verdict carries specific semantic weight:

1. The target artifact is not proven broken (FAILED).
2. The target artifact is not proven correct (CERTIFIED).
3. The harness could not exercise the path.

Conflating INCONCLUSIVE with FAILED punishes the developer for a gap in the testing apparatus. Conflating it with a pass silently degrades your code certification process.

Routing Inconclusive Runs in the Merge Queue

Merge queues exist to ensure that the main branch remains deployable. How the queue acts on an inconclusive verdict depends on the blast radius and the nature of the diff:

• The diff modifies harness code or contract definitions: The queue must block. If the developer intended to improve coverage and the harness fails to construct the input, the harness change is incomplete.
• The diff modifies standard business logic with existing regression coverage: If unit and integration suites pass, the pull request can merge under a provisional verdict. The system records that while the change did not regress baseline tests, extended contract verification was skipped due to a harness limit.
• The diff touches a high-criticality invariant: For cryptographic, authentication, or financial boundaries, an inconclusive run must pause promotion until the harness contract is repaired.

Capturing the verdict as structured metadata prevents the queue from halting on fixture bugs while keeping unverified paths visible in release records.

{
  "$schema": "https://shpbl.com/schemas/queue-verdict.v1.json",
  "run_id": "run_01HX9J4Z8V2B5M8T1E",
  "target_ref": "sha256:4f8a3c9e7b2d1a0f",
  "verdict": "INCONCLUSIVE",
  "disposition": "ROUTE_PROVISIONAL",
  "gate_policy": {
    "block_on_critical_invariants": true,
    "critical_paths_touched": []
  },
  "harness_telemetry": {
    "phase": "FIXTURE_SETUP",
    "failure_class": "UNSATISFIED_PRECONDITION",
    "exit_code": 3
  }
}

Queue controllers inspect verdict and disposition rather than raw process exit codes. A harness boundary failure routes through policy instead of polluting test telemetry with false negatives.