The Operational Cost of Provisional Test Verdicts

A test runner exits with code zero. The deployment pipeline turns green, the container builds, and the artifact reaches production. Twenty minutes later, a customer sends a batch payload with an unexpected null field, and the worker process deadlocks.

The automated suite did not fail before deployment because the test runner was never asked to verify the code. It was asked to execute it.

Treating reproducible execution as verification is one of the most expensive habits in software engineering. When a suite executes a code path without asserting its underlying invariants, it produces a provisional verdict: the code ran without crashing on the exact bytes supplied to it. Promoting that artifact to production as if it carried an executable proof of correctness guarantees brittle downstream systems.

The Gap Between Execution and Correctness

Consider a token-bucket rate limiter. A developer writes a quick suite: instantiate the bucket with ten tokens, call consume(1) ten times, assert that each call returns true, and exit.

func TestTokenBucket(t *testing.T) {
    tb := NewTokenBucket(10, 1*time.Second)
    for i := 0; i < 10; i++ {
        if !tb.Consume(1) {
            t.Fatalf("expected token at %d", i)
        }
    }
}

This test runs. It passes reliably across architectures. It yields a clean exit status. But it tests almost nothing about the contract:

• What happens if Consume is called with zero or negative tokens?
• Does the refiller drift under clock skew?
• What happens when twenty goroutines hit the bucket simultaneously with one token remaining?

The test suite merely proves that on a single thread, within one second, ten sequential subtractions succeeded. In an honest code certification system, this result receives a provisional verdict. The harness ran the artifact, and the execution completed reproducibly. But correctness was not asserted.

When a team lacks a vocabulary to distinguish between "it ran" and "it is correct", provisional artifacts get tagged as production-ready. The operational cost arrives later, paid in silent state corruption, unhandled edge cases, and middle-of-the-night rollbacks.

Why Provisional Verdicts Drift into Production

Provisional code slips into release branches for two reasons: test coverage metrics lie, and engineers optimize for green dashboards.

Line coverage measures whether an instruction was loaded into the CPU, not whether the result of that instruction was checked against a specification. If a function contains twenty branches and your test trips five of them without an assertion, your coverage tool reports progress. The CI runner reports green.

The danger is not that the code is broken. The danger is that nobody knows what it actually guarantees.

When a team merges provisional code, they inherit hidden operational assumptions:

1. Unbounded inputs: The code assumes callers will only supply standard, well-formed arguments because the tests only supplied standard, well-formed arguments.
2. Brittle dependencies: The code assumes external clocks, network responses, or file handles behave exactly as they did during the green local run.
3. Compounding uncertainty: When component A with a provisional verdict is composed with component B with a provisional verdict, the system's failure modes multiply rather than add.

At that point, your CI pipeline is no longer acting as a gate for verification. It has been demoted to a syntax-checker that runs code once before passing the risk to your users.

Quarantining the Provisional State

The remedy is straightforward: isolate provisional code until its input contracts are fully asserted.

In our own certification harness, an artifact that executes reproducibly but lacks comprehensive invariant checks is explicitly marked provisional. It is not marked failed—because the code did not break—and it is not marked inconclusive, which is reserved for cases when the harness fails instead of the code. It is simply quarantined. It cannot be promoted to the master catalog, and it cannot be claimed as certified capability.

Quarantine changes how teams handle implementation work:

First, it forces an honest accounting of the test suite. If an engineer submits a pull request with execution tests but no edge-case coverage or property assertions, the PR is categorized as provisional. It does not ship to production.

Second, it narrows the debugging surface. When an outage occurs, on-call engineers can immediately rule out certified components and focus their investigation on systems running on provisional or legacy boundaries.

Third, it provides a clear target for automated fuzzing and property testing. Provisional code is the primary candidate for generative harness runs that search for contract violations.

What to Change on Monday

Stop treating a green exit code as a binary pass/fail signal.

Open your core repository and inspect the three most critical modules handling data transformations or business invariants. Look at their test files. Remove the assertions mentally. If the test still runs to completion without failing, you do not have executable proof of correct behavior; you have an execution script.

Introduce a formal rule into your team's code review standard: a test must assert the negative cases, the boundary conditions, and the invalid inputs before an artifact can be promoted out of provisional status. If a component has only been proven to run on the happy path, keep it quarantined until the contract is verified.