Turning Inconclusive Harness Results into Targeted Test Suites

The audit stops at a wall

You run a repository audit. The harness executes the artifact. It returns INCONCLUSIVE. Nothing else happens. The audit report points to a missing input, but the workflow does not continue. You are left with a ticket that never moves. The cost of the audit is sunk, and the repair never materialises.

What makes a harness verdict INCONCLUSIVE

The harness is a certification runner. It can only judge an artifact when it can exercise the input contract fully. If the contract is underspecified – a missing environment variable, an undefined file path, an ambiguous API shape – the runner cannot reach a pass or fail decision. It records INCONCLUSIVE and stops. The verdict is not a defect in the artifact; it is a gap in the test harness configuration.

The gap is explicit in the harness log. It lists the required input name, the expected type, and the point where execution halted. For example:

INCONCLUSIVE: missing input “CONFIG_PATH” required by step “loadConfig”

That line tells you exactly what the harness could not provide. It is the missing piece that prevents the audit from becoming a repair.

Extracting the missing input into a focused test

The first step is to turn the log entry into a concrete test case. The test must supply the missing input and assert the behaviour that the harness could not observe. The process can be scripted:

1. Parse the harness log for missing input lines.
2. For each line, generate a minimal test file that sets the input and calls the entry point that previously failed.
3. Add an assertion that the call returns without error, or that a known side‑effect occurs.

A short snippet in TypeScript illustrates the idea:

import { runArtifact } from '@shpbl/sdk';

test('loadConfig with CONFIG_PATH', async () => {
  const result = await runArtifact({
    inputs: { CONFIG_PATH: '/tmp/config.json' },
  });
  expect(result.error).toBeUndefined();
});

The generated test is narrow. It does not try to cover the whole artifact, only the path that the harness could not traverse. Because the test is derived directly from the INCONCLUSIVE log, it is guaranteed to target the exact missing input.

Running this test locally validates that the input is sufficient. If the test fails, the log now shows a different error – perhaps a downstream validation problem – which can be addressed in the next iteration.

Feeding the generated test back into the repair loop

Once the focused test passes, the audit can move forward. The repair pipeline expects a PROVISIONAL verdict before it will create a fix branch. By adding the new test to the repository’s test suite, the harness will now have a complete input contract. Re‑run the harness; it should emit PROVISIONAL or CERTIFIED.

At this point the standard audit‑repair flow resumes. The artifact is re‑evaluated, the harness confirms reproducibility, and an automated repair PR can be opened. The whole sequence – from INCONCLUSIVE verdict to test generation to repaired artifact – is described in the pillar post /blog/closing-the-loop-from-repository-audit-to-merged-repair. That article shows how a successful harness run unlocks the automatic merge step.

If you need a concrete example of the transformation, see the case study /blog/from-inconclusive-harness-verdict-to-automated-repair-pr. It walks through a real repository where a missing DB_URL caused an INCONCLUSIVE verdict, a single generated test supplied the URL, and the subsequent repair PR merged without manual intervention.

What to try on Monday

1. Pick the most recent audit that ended with INCONCLUSIVE.
2. Open the harness log and locate the first “missing input” line.
3. Write a one‑line test that provides that input and asserts a successful run.
4. Add the test to the repository, commit, and re‑run the harness.

If the harness now returns PROVISIONAL, you have turned a dead‑end audit into a repair candidate. The effort is a single test, but the payoff is an artifact that can be merged automatically. This small loop can be repeated for every INCONCLUSIVE verdict, converting audit cost into tangible coverage.