The Four Verdicts Between Green Tests and Correct Code

A payment reconciliation pipeline ran at midnight, processed 12,000 pending settlements, and exited with status code 0. The CI pipeline stayed green. The monitoring dashboard showed zero unhandled exceptions.

By 06:00, finance noticed that not a single settlement had posted to the ledger.

The worker had received an empty payload from a misconfigured upstream queue, parsed the JSON array as valid, iterated over zero items, and returned cleanly. The test suite had a unit test asserting that an empty array returned a successful status code. In the eyes of the runner, the code was completely healthy. In reality, the system was inert.

Every engineering team has lived some version of this morning. It happens because software tooling collapses the entire spectrum of program behavior into a binary: pass or fail, 0 or 1, green or red.

A green build does not prove your code is correct. It proves only that the test harness executed a sequence of instructions without encountering an unhandled exception or an explicit assertion failure before the process terminated. Between a broken script and provably correct code sits an entire ladder of epistemic states. Collapsing those states into a single green checkmark creates a dangerous illusion of safety.

If we want rigor in software engineering, we have to stop treating "it ran without crashing" as proof of correctness. We need a taxonomy that names exactly what was checked, what was assumed, and what the harness failed to reach.

The Problem with the Binary Exit Code

Standard continuous integration relies on the POSIX exit code. If the process returns 0, the build is green. If it returns anything else, the build is red.

This design made sense in 1974 for chaining Unix utilities. It is poorly suited for code certification.

When a test suite runs, a dozen distinct things can happen:

1. The code executes against exhaustive boundary conditions and proves its invariants hold.
2. The code executes against a trivial happy path, leaving edge cases unprobed.
3. The test mocks out the only failure mode that matters in production.
4. The test environment lacks a dependent service, catches the network error, and skips the assertion entirely.
5. The assertions fail due to a genuine logic error.
6. The test runner crashes because the disk ran out of space.

Current tooling lumps scenarios 1, 2, 3, and 4 into "Green," while scenarios 5 and 6 are lumped into "Red."

This creates two fatal failure modes. First, developers assume that green means verified, shipping software with massive blind spots. Second, developers treat flaky harness infrastructure (scenario 6) with the same priority as broken business logic (scenario 5), breeding alert fatigue.

We need to divide execution outcomes into four distinct verdicts: FAILED, INCONCLUSIVE, PROVISIONAL, and CERTIFIED.

[ Execution Attempt ]
        │
        ├── Crashed / Assertion tripped? ───────────► FAILED
        │
        ├── Harness could not exercise contract? ──► INCONCLUSIVE
        │
        ├── Deterministic & clean, but unproven? ──► PROVISIONAL
        │
        └── Exhaustive invariants & boundary match ─► CERTIFIED

1. FAILED: The Contract Was Breached

A FAILED verdict is the cleanest state in software. It means the harness successfully exercised the code, the code ran in the target environment, and an explicit postcondition or invariant was violated.

func TestTransferDeductsBalance(t *testing.T) {
    acc := NewAccount(100)
    err := acc.Transfer(40, "recipient-id")
    if err != nil {
        t.Fatalf("unexpected error: %v", err)
    }
    if acc.Balance != 60 {
        t.Fatalf("expected balance 60, got %d", acc.Balance)
    }
}

If acc.Balance evaluates to 100, the invariant fails. We know precisely what failed, why it failed, and what contract was breached.

The value of a FAILED verdict depends on harness determinism. If a test fails because of a race condition in the test harness itself, it is not a code failure; it is an infrastructure defect. True failure requires that the fault lies entirely within the component under test.

2. INCONCLUSIVE: The Harness Failed Its Job

An INCONCLUSIVE verdict does not mean the code is broken. It means the harness was unable to evaluate the input contract.

Consider an integration test for an S3-compatible storage driver:

func TestStorageUpload(t *testing.T) {
    endpoint := os.Getenv("MINIO_ENDPOINT")
    if endpoint == "" {
        t.Skip("MINIO_ENDPOINT not set; skipping integration test")
    }
    // ... upload assertions ...
}

In standard CI, t.Skip() results in a green run. The dashboard displays a comforting green checkmark.

In truth, this result is inconclusive. The harness could not exercise the target contract because the necessary environment was missing. Marking this green is dishonest; marking it failed is inaccurate.

The same applies when mocks abstract away the failure surface. If you mock a database driver to always return (rows, nil), you have not tested your query parsing, your serialization, or your connection pool exhaustion. You have tested your mock.

An honest verification system labels these runs as INCONCLUSIVE. It tells the engineer: We executed the process, but we did not test the contract. No correctness claim can be made.

3. PROVISIONAL: Reproducible, but Unasserted

A PROVISIONAL verdict is where most production software actually lives, though few teams admit it.

A provisional verdict means:

• The component compiles cleanly.
• The build is byte-for-byte reproducible (sealed by published hashes or checksums).
• The artifact executes in a deterministic sandbox without crashing.
• Basic smoke tests pass.
• However, exhaustive invariant verification, formal proofs, or exhaustive edge-case property tests have not yet been asserted.

+-------------------------------------------------------------+
| PROVISIONAL VERDICT                                         |
+-------------------------------------------------------------+
| [x] Build reproducible from source                          |
| [x] Zero unhandled exceptions under standard smoke load     |
| [ ] Property-based fuzzing across input domain              |
| [ ] State-machine transition invariants mathematically proven|
+-------------------------------------------------------------+

Provisional code is not defective code. It is simply unverified code whose runtime stability has met a baseline standard of mechanical health. A provisional verdict separates the physical reality of software (does it build and run consistently?) from the logical claim of software (does it satisfy every business invariant across all edge states?).

When you ingest third-party packages or internal utilities, you are rarely getting certified software. You are getting provisional software. Calling it provisional keeps the risk visible instead of burying it under a green build badge.

4. CERTIFIED: Invariants Exhaustively Bound

A CERTIFIED verdict is the highest rung. It requires that the component's behavior is fully bounded against an explicit formal contract.

To reach a certified verdict:

1. Explicit Invariants: The component defines unambiguous pre-conditions, post-conditions, and state invariants.
2. Deterministic Execution: The artifact contains no non-deterministic runtime dependencies (e.g., hidden network calls, unseeded random number generators, unsynchronized concurrency).
3. Boundary Verification: The harness exercises not just the happy path, but the entire input boundary matrix—either through formal verification, exhaustive state-space enumeration, or bounded model checking.

// A certified state machine transition guarantees no unreachable or invalid state
pub fn transition(state: State, event: Event) -> Result<State, InvalidTransition> {
    match (state, event) {
        (State::Draft, Event::Submit)     => Ok(State::PendingReview),
        (State::PendingReview, Event::Approve) => Ok(State::Approved),
        (State::PendingReview, Event::Reject)  => Ok(State::Rejected),
        (current, invalid_event) => Err(InvalidTransition { current, invalid_event }),
    }
}

When this state machine is tested against every permutation of (State, Event) and proven to preserve balance conservation, idempotency, or authorization boundaries, it earns certification. You are no longer guessing whether an unhandled event might wedge the worker. The behavior is sealed.

For deeper architectural breakdowns of deterministic boundary modeling, see our work in the Volumes.

Why Naming the Rung Matters

When teams collapse these four verdicts into pass/fail, they create systemic technical debt.

| Rung | What the Team Thinks It Means | What It Actually Means | | :--- | :--- | :--- | | FAILED | "Our code is broken." | An assertion failed or the harness crashed. (Distinction required). | | INCONCLUSIVE | "Our tests passed." | The harness did not exercise the code path. | | PROVISIONAL | "The feature is completely done." | The artifact builds and runs, but invariants are unproven. | | CERTIFIED | (Rarely distinguished) | The component is provably bounded against its contract. |

If an engineering team cannot distinguish between an inconclusive run and a certified artifact, they will inevitably push broken software to production while pointing at their green CI dashboard in bewilderment.

Software reliability begins with epistemic honesty. Acknowledging that an integration suite is inconclusive because mocks masked the transport layer is not an admission of defeat—it is accurate reporting. It highlights where real engineering risk remains.

What to Change on Monday

You do not need to rewrite your entire testing infrastructure over a weekend to apply this discipline. You can introduce truth to your pipeline with three immediate changes:

1. Split Skipped Tests from Passing Tests: Configure your CI reporting to flag skipped suites, empty database queries, or unconfigured mocks as INCONCLUSIVE (amber) rather than green. If a test does not run its assertions, the build should not display a passing state.
2. Audit Your Mocking Boundaries: Identify tests where external interfaces (APIs, filesystems, database engines) are mocked to return constant success. Reclassify those unit tests as smoke verifications rather than semantic correctness proofs.
3. Tag Invariant Tests Separately from Smoke Tests: Separate your test suite into baseline execution (provisional validation) and boundary/property validation. If a service only passes smoke tests, label the artifact PROVISIONAL in your internal registry. Reserve full promotion gates for artifacts that carry verified assertions across their boundary matrix.

Stop accepting the binary lie of the green build. Name your rungs honestly, and build systems that know the difference between code that merely runs and code that is provably correct.