Prioritizing Refactoring with Provisional Verdicts

The hidden cost of a green test suite

Your CI pipeline shows a green bar. The code compiles. All unit tests pass. A recent production incident proves the behaviour is still wrong. The failure is not a missing test; it is a missing signal. When the test harness reports a PROVISIONAL verdict, the artefact is reproducible but its correctness is unverified. Treat that verdict as a warning, not a pass, to expose the gap between “it compiles” and “it is correct”.

A PROVISIONAL verdict means the harness could run the artefact but could not assert its correctness. An INCONCLUSIVE verdict means the harness could not exercise the input contract at all – a limitation of the harness, not a defect in the artefact. Keeping the two distinct prevents conflating a missing test with a missing execution path.

Developers often ignore provisional results because they sit beside a flood of passing tests. The result is a growing list of artefacts known to be flaky, but never prioritised. The list expands until a critical bug surfaces, and the team scrambles to locate the root cause. The cost is lost time, eroded confidence in CI, and a higher likelihood of regressions.

Turning a provisional verdict into a ticket

A provisional verdict is a concrete artefact of the test harness. It tells you two things: the artefact can be built and executed, and the harness could not assert correctness. That information is precise enough to be turned into a work item.

1. Capture the verdict – The harness writes a record that includes the artefact identifier, the commit hash, and the verdict label.
2. Enrich with context – Attach the failing harness log, the list of inputs that could not be exercised, and any metadata about licensing or invariants that the Build Intent gate resolved.
3. Create a CI‑driven ticket – A small automation script reads the record and opens an issue in your tracker, using a template that marks the issue as Refactor needed – provisional verdict. The template includes a link to the harness run (/harness) so reviewers can replay the scenario locally.

Because the ticket is generated automatically, it never gets lost in the noise of passing tests. The title is explicit: “Provisional verdict on lib‑utils@a1b2c3 – reproducible but unverified”. The body contains the exact steps to reproduce the harness run, so a developer can start investigating without hunting for logs.

Prioritising refactor work with CI signals

Not every provisional verdict deserves immediate attention. Use the CI signal itself to rank the work:

• Frequency – If the same artefact receives provisional verdicts on multiple commits, the signal is stronger. The automation can bump the ticket priority after the second occurrence.
• Impact – Cross‑reference the artefact’s dependency graph. An artefact that downstream services import is higher risk than an isolated utility.
• Complexity – The harness log often shows where the input contract failed. If the failure is a missing mock or an unhandled edge case, the fix may be a small test addition. If the harness could not exercise the contract at all, the underlying code likely needs a redesign.

Treating the provisional verdict as a first‑class CI signal helps you avoid chasing every failing test. You focus on artefacts that are reproducible yet unverified, and you allocate refactoring effort where it reduces the most risk.

A concrete workflow you can start on Monday

1. Add a hook to your harness – Extend the harness runner to emit a JSON line for every provisional verdict. Include the artefact ID, commit SHA, and a URL to the run (/harness).
2. Write a small script – Use the TypeScript client (@shpbl/sdk) to poll the harness output, filter for provisional verdicts, and call your issue‑tracker API to open a ticket. Keep the script under 50 lines; its purpose is to turn a machine‑readable signal into a human‑readable work item.
3. Run the script in CI – Add a step after the harness stage that executes the script. The step should never fail the pipeline; it only creates tickets.
4. Monitor the new tickets – During the first week, review the generated tickets in your sprint planning. Adjust the priority logic if you see too many low‑impact items.

The loop surfaces reproducibility problems as actionable tickets, not as ignored warnings. Over time you may observe a reduction in the backlog of provisional artefacts, and your green test suite can start to mean more than “nothing broke”. It will mean “nothing is known to be broken, and everything that could be verified has been verified”.

Note: The backlog reduction is an expected benefit of the process, not a guaranteed outcome. The same applies to any improvement in CI signal quality.

For a deeper look at the four verdicts and why provisional is a useful rung on the ladder, see the pillar post /blog/the-four-verdicts-between-green-tests-and-correct-code. Implement the hook and script on Monday, and you will have the first tickets ready for the next sprint.