Automate Repair PRs for Provisional Harness Verdicts

Why provisional verdicts stall

A provisional verdict tells you that the harness could run the artifact, but it could not yet assert correctness. In a typical CI run the job ends with “PROVISIONAL” and the pipeline stops. Engineers treat the result as a warning, file a ticket, and hope someone will later investigate. The signal never becomes a concrete action, so the same code sits in the repository awaiting manual triage. The cost is time spent reading logs, reproducing the environment, and deciding what change would satisfy the harness. The failure mode is clear: a provisional verdict is left as noise instead of a trigger.

Turning the signal into a repair PR

The remedy is to let the verdict itself generate a minimal change that moves the artifact into the next rung – usually a tighter contract or a missing dependency. The repair PR contains only the diff required to satisfy the harness’s expectations, leaving the rest of the code untouched.

A concrete example: the harness reports that a library entry is missing a required field in its manifest. The repair script reads the verdict payload, extracts the missing key, and adds a default value. The resulting commit is automatically pushed to a short‑lived branch and opened as a pull request labelled “repair‑provisional”. The PR description includes the original verdict, a link to the live run (/harness), and a checklist that the new build intent must pass.

## .github/workflows/provisional-repair.yml
name: Provisional Repair
on:
  workflow_run:
    workflows: ["Run Harness"]
    types:
      - completed
jobs:
  repair:
    if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.outputs.verdict == 'PROVISIONAL' }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Download verdict payload
        run: curl -s ${{ github.event.workflow_run.outputs.payload_url }} -o verdict.json
      - name: Generate repair PR
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          node scripts/repair-provisional.js verdict.json

The repair-provisional.js script parses verdict.json, applies the minimal edit, commits with a deterministic message, and uses the GitHub API to open the PR. Because the script runs in the same CI environment that produced the verdict, the repair is repeatable and auditable.

Putting the loop into CI

The loop consists of three parts that can be wired together with any CI system that supports webhook‑driven jobs:

1. Harness execution – the standard run that produces a verdict. The harness is model‑independent; its output is a plain JSON record that includes the verdict type and any missing contract elements.
2. Verdict listener – a job that triggers on the harness completion, filters for PROVISIONAL, and invokes the repair script. The listener can be a simple workflow step as shown above or a dedicated service that polls the /api-access endpoint for new provisional results.
3. Verification step – after the repair PR is merged, the pipeline re‑runs the harness on the updated artifact. If the verdict upgrades to CERTIFIED or at least PROVISIONAL disappears, the loop closes; otherwise the PR is marked with a failure label for further human review.

By keeping the repair logic in a typed TypeScript client (@shpbl/sdk) you guarantee that the same contract definitions used by the harness are also used to construct the fix. This eliminates the drift that often occurs when engineers hand‑code fixes based on memory of the error message.

The approach aligns with the layered‑verdict model described in the pillar post /blog/the-four-verdicts-between-green-tests-and-correct-code. Instead of treating a provisional verdict as a dead end, the pipeline treats it as a state transition that can be automated.

First step on Monday

1. Clone the repository that contains your harness workflow.
2. Add the provisional-repair.yml file shown above to .github/workflows.
3. Implement a minimal scripts/repair-provisional.js that logs the verdict payload and creates a dummy change (e.g., adds a comment file). Commit and push.

Run the CI pipeline once. When the harness emits a provisional verdict, the new workflow will open a repair PR automatically. Observe the PR, verify that the harness passes on the new commit, and iterate on the script until it produces a meaningful fix for your own contract failures.

Once the loop works for a single case, expand the script to cover the other provisional signals you encounter. The result is a CI pipeline that turns every provisional verdict into an actionable repair PR, removing the manual triage bottleneck and keeping the codebase moving toward certified status.