Fuzzing Input Contracts to Resolve Inconclusive Harness Verdicts

The problem with INCONCLUSIVE verdicts

A test harness can return INCONCLUSIVE when it cannot drive the artifact through the parts of its input contract that matter. The run finishes without error, but the verdict tells you nothing about the behaviour under the missing conditions. In a strict CI pipeline this is a dead end: the green suite passes, yet you have no executable proof that the component does what it claims.

Why static fixtures miss boundary conditions

Most harnesses start from a hand‑written fixture set. Those fixtures are convenient, but they are inevitably narrow. They cover the happy path and a few obvious edge cases. Anything outside that narrow band is never exercised. The harness therefore cannot decide whether the artifact would reject an out‑of‑range value, overflow a buffer, or silently ignore a required field.

The failure mode is unexercised contract. The harness reports INCONCLUSIVE, not because the artifact is broken, but because the input space was under‑sampled. The result is a false sense of safety; the green suite proves only that the fixture set is compatible, not that the component is correct.

Contract fuzzing as a deterministic resolver

Fuzzing the input contract means generating a systematic set of valid permutations that respect the contract’s type and structural rules, then feeding each permutation to the harness. The key is validity: every generated payload must satisfy the schema, otherwise the harness will reject it before reaching the component logic, and the verdict will remain INCONCLUSIVE.

A simple generator can be built from the contract’s JSON Schema (or TypeScript interface). For each property:

function fuzz(schema: JSONSchema): any {
  if (schema.type === 'integer') return [0, 1, -1, Number.MAX_SAFE_INTEGER];
  if (schema.type === 'string') return ['', 'a', 'longer‑string', '🚀'];
  if (schema.type === 'array') return [[], [fuzz(schema.items)], [fuzz(schema.items), fuzz(schema.items)]];
  // recurse for objects
}

The generator produces a modest cartesian product. Because the contract is finite, the set is repeatable. Running the harness against each generated input yields one of three outcomes:

• CERTIFIED – the component behaves as expected for that input.
• FAILED – the component crashes or returns an unexpected result.
• PROVISIONAL – the component runs but the harness cannot assert correctness (e.g., missing oracle).

When the harness returns FAILED or CERTIFIED, the INCONCLUSIVE verdict disappears. The harness now has exercised the previously hidden branch, and the verdict reflects the true state of the artifact.

This approach is described in more detail in the pillar post /blog/the-four-verdicts-between-green-tests-and-correct-code, which explains why moving from a green suite to an executable proof matters.

Putting it into your pipeline

1. Export the component’s input contract from your source (JSON Schema, OpenAPI, or TypeScript types).
2. Run the fuzz generator as a pre‑step to the harness. Keep the generated corpus under version control; it is repeatable and can be audited.
3. Feed each payload to the live run endpoint (see /harness for an example of a harness invocation). Record the verdicts.
4. Fail the CI job if any FAILED verdict appears, or if the ratio of INCONCLUSIVE stays above a threshold you define.

The extra step adds a few minutes to the CI cycle, but it replaces a silent green run with a concrete executable proof that the component respects its contract across a realistic input space.

Monday’s action

Clone the contract file for the component you are about to ship. Write a tiny script that produces the fuzzed inputs as shown above, and run the harness against them locally. If you see any FAILED verdicts, fix the component before the next commit. This single afternoon experiment will turn an INCONCLUSIVE result into a meaningful certification signal.