The Gap Between Compilation and Correctness

The illusion of a green build

A CI run that ends with “build succeeded” is easy to celebrate. The compiler has accepted the source. The test runner has executed without crashing. That is all the signal tells you. It does not tell you that the program meets its specification.

In many teams the green bar becomes a proxy for “the code works”. The failure mode is clear: developers ship artifacts that compile and pass a shallow test suite, yet the behaviour in production diverges from intent. Bugs surface only under real workloads, after the fact, and the cost of rollback is high.

Verification levels beyond compilation

The gap between “it compiles” and “it is correct” can be expressed as a ladder of verification levels. Each rung adds a concrete, observable property.

• Compiled – the source parses and type‑checks. No runtime errors on load.
• Reproducible – the same inputs produce the same binary artefact, byte for byte. This is a repeatable step, independent of any model.
• Provisional – a test harness has exercised the artefact under a defined contract and returned a PROVISIONAL verdict. The harness confirms that the code runs, but it does not yet assert full correctness.
• Certified – the harness has verified the contract exhaustively and returned CERTIFIED. The artefact is now ready for code certification.

The PROVISIONAL rung is often overlooked. It is the point where a test harness has run the code, observed its behaviour, and recorded that the execution was successful, yet the harness could not cover the full specification. The verdict is honest: the artifact works as far as we have tested, but correctness remains unasserted.

When a provisional verdict is useful

Consider a library that implements a sorting routine. The test harness runs the routine on a handful of random arrays and observes that it returns a sorted list. The harness records PROVISIONAL. The code passes the green suite, but the guarantee that the algorithm works for all possible inputs is missing.

A PROVISIONAL verdict tells a team two things. First, the artefact is reproducible; the same source will always compile to the same binary. Second, the harness exercised the public contract without crashing. That information is valuable for a CI gate that wants to block only clearly broken artefacts while still allowing rapid iteration.

If the harness cannot exercise a particular input shape—say, a very large data set—the verdict becomes INCONCLUSIVE. That is not a defect in the code; it is a limit of the harness. The team can then write a property‑based test to expand coverage, turning INCONCLUSIVE into PROVISIONAL and eventually into CERTIFIED.

The ladder approach avoids the binary “green / red” mindset. It gives developers a language to discuss risk, and it lets CI pipelines make nuanced decisions based on the exact verification level.

How SHPBL makes the ladder concrete

SHPBL provides a single method that feeds all four access doors – an MCP server, a plain HTTP API, a typed TypeScript client, and an offline file‑based edition. The method runs a test harness that records one of the four verdicts: CERTIFIED, PROVISIONAL, INCONCLUSIVE, or FAILED. The harness execution is documented on the [/harness] page, and the same tools are available via the [/sdk] client. Because the library is model‑independent, the artefacts are computed rather than generated, and published checksums guarantee byte‑for‑byte reproducibility. Teams can start with a free repository evaluation at [/evaluation] to see the current verification level of their code, then decide whether to adopt the full verification pipeline.

Next steps for Monday

1. Open your CI configuration. Replace the simple “build succeeded” gate with a call to the SHPBL test harness via the HTTP API.
2. Record the verdict returned by the harness. Treat PROVISIONAL as a pass that still requires targeted tests before promotion.
3. Add a failing step that blocks promotion on FAILED or INCONCLUSIVE results until you expand the harness coverage.

By making the verification level explicit, you stop treating a green build as a certificate of correctness. Instead you build a ladder that climbs toward true code certification.

For a deeper dive into the four verdicts, see the pillar post [/blog/the-four-verdicts-between-green-tests-and-correct-code].