From INCONCLUSIVE Harness Verdict to Automated Repair PR

The audit dead‑end

You run the repository audit, the harness executes the candidate artifact, and the verdict comes back INCONCLUSIVE. The report tells you the harness could not exercise the input contract. Nothing else happens. The team files a ticket, hopes someone will write a test, and the audit sits in the backlog. The cost of the audit is already incurred; the value is lost because the repair never materialises.

Why the harness reports INCONCLUSIVE

The harness is strict about what it can observe. It loads the artifact, injects the inputs it knows about, and watches for a clean exit. When the artifact expects a file, an environment variable, or a network service that the harness does not provide, it aborts before reaching the verification point. The result is not a failure of the code, but a failure of the harness configuration. The verdict therefore signals a gap: the audit knows the artifact is potentially useful, but the test harness lacks the required stimulus.

Treating this gap as a defect in the artifact is a mistake. The artifact may be correct; the harness simply needs more data. The correct posture is to close the gap automatically, turning the missing stimulus into concrete test inputs or stubs, and then re‑run the harness.

Turning the verdict into a repair PR

Automation can bridge the gap. The harness emits a JSON payload that lists the missing inputs, for example:

{
  "artifact": "src/lib/parseConfig.js",
  "missingInputs": [
    { "type": "file", "path": "config/default.json" },
    { "type": "env",  "name": "API_KEY" }
  ]
}

A short script can consume this payload, generate the required files or environment‑variable mocks, commit them to a new branch, and open a pull request. The script lives in the CI runner, so the whole loop runs without human intervention.

// repair.ts – minimal example using @shpbl/sdk
import { createBranch, commitFile, openPR } from '@shpbl/sdk';
import * as fs from 'fs';
import payload from './harness-report.json';

async function main() {
  const branch = await createBranch(`repair/${payload.artifact}`);
  for (const input of payload.missingInputs) {
    if (input.type === 'file') {
      const stub = JSON.stringify({ placeholder: true }, null, 2);
      await commitFile(branch, input.path, stub);
    }
    if (input.type === 'env') {
      // create a .env stub in the repo root
      const envPath = '.env';
      const line = `${input.name}=placeholder`;
      const existing = fs.existsSync(envPath) ? fs.readFileSync(envPath, 'utf8') : '';
      await commitFile(branch, envPath, `${existing}\n${line}\n`);
    }
  }
  await openPR({
    source: branch,
    target: 'main',
    title: `Repair: add missing inputs for ${payload.artifact}`,
    body: 'Automated repair generated from an INCONCLUSIVE harness verdict.'
  });
}
main().catch(err => process.exit(1));

The script does three things:

1. Creates a branch named after the artifact.
2. Adds the missing inputs as stub files or .env entries.
3. Opens a PR with a concise description.

Because the script uses the typed TypeScript client, the same code works whether it runs locally, in a CI job, or via the HTTP API. The PR lands in the repository, the CI pipeline re‑runs the harness, and the verdict changes to CERTIFIED or PROVISIONAL. No manual ticket is required.

Integrating the fix into CI

Place the script in a CI stage that triggers on any INCONCLUSIVE result. A typical pipeline fragment looks like:

- name: Run harness
  run: npm run harness
- name: Detect inconclusive
  id: verdict
  run: |
    if grep -q INCONCLUSIVE harness-report.json; then
      echo "needs_repair=true" >> $GITHUB_OUTPUT
    fi
- name: Generate repair PR
  if: steps.verdict.outputs.needs_repair == 'true'
  run: node repair.ts

When the harness cannot exercise the artifact, the pipeline automatically creates a repair branch and opens a PR. The next CI run, triggered by the PR, executes the harness again. If the newly added inputs satisfy the contract, the harness returns CERTIFIED and the PR can be merged automatically.

This approach turns a dead‑end audit into a concrete improvement without extra human effort. It also keeps the audit cost low: the same harness run that produced the INCONCLUSIVE verdict now drives the repair. The loop described here aligns with the broader argument in the pillar post /blog/closing-the-loop-from-repository-audit-to-merged-repair, but focuses on the moment when the harness stalls.

What to do on Monday

Clone a repository that already uses the SHPBL harness, run the harness locally, and inspect the generated harness-report.json. Drop the repair.ts script into the repo, add the CI fragment, and push the changes to a feature branch. Observe the automated PR appear, merge it, and watch the harness move from INCONCLUSIVE to CERTIFIED. This single experiment will prove that the audit can close the loop without manual tickets.