Dissecting the Boundary Between Passing Tests and True Proof

A test runner returning exit code zero guarantees exactly one thing: the interpreter or process reached the end of the script without an unhandled exception or a failed assertion. It does not prove the artifact satisfies its design requirements. It does not prove the edge boundaries hold under unexpected input. In many cases, it does not even prove the code under test executed in a meaningful way.

Treating exit 0 as a proxy for correctness is one of the most persistent failure modes in software delivery. A green suite proves that your tests ran. It does not provide verification that your software works.

The Gap Between Assertion and Invariant

Most test suites are collections of point checks. An engineer writes a function, thinks of three plausible inputs, writes three assert statements, and pushes the branch.

function parseRateLimit(header: string | null): number {
  if (!header) return 60;
  const parsed = parseInt(header, 10);
  return isNaN(parsed) ? 60 : parsed;
}

// The test suite:
test('parseRateLimit defaults and parses integers', () => {
  expect(parseRateLimit(null)).toBe(60);
  expect(parseRateLimit('120')).toBe(120);
});

This test suite passes. The CI pipeline turns green. Yet the implementation contains obvious defects: negative numbers ('-10') yield a negative rate limit, floats ('12.5') silently truncate, and extreme values ('999999999999999999999') overflow safe integer ranges.

A point check inspects a single coordinate on an input surface. An invariant defines the shape of the surface itself. When you only test specific coordinates, you verify that your assertions match your assumptions at those exact points. You have not tested the contract.

Bridging the boundary between passing tests and genuine verification requires moving from point checks to executable proof of invariants:

1. Domain boundaries: What happens at the absolute limits of the input type (empty strings, zero, negative infinity, non-canonical encodings)?
2. State transitions: Does the sequence of calls leave the system in an illegal state, even if every individual return value matches its local expectation?
3. Failure recovery: Does an invalid input fail closed, without leaking memory, mutating shared state, or masking underlying faults?

Why Exit Code Zero Is Provisional

In an engineering pipeline that takes code certification seriously, a passing test suite is only the first checkpoint. It establishes basic executability. It confirms that the runtime can load the artifact, resolve its dependencies, and execute a sequence of instructions without crashing.

In our classification model, this state is merely provisional. As we explore in the four verdicts between green tests and correct code, treating reproducible point tests as definitive proof of correctness conflates running code with correct code. A component whose tests run cleanly has earned the right to be evaluated; it has not earned a certificate of correctness.

To elevate an artifact from provisional to certified, a test harness must subject the artifact to adversarial verification. That means exercising properties the author likely did not write tests for:

• Property-based generation: Feeding thousands of pseudo-random inputs through the function to locate edge cases where invariants break.
• Contract enforcement: Explicitly evaluating pre-conditions, post-conditions, and frame conditions (what the function must not mutate).
• Environment isolation: Proving that the artifact does not depend on ambient disk state, system clocks, network availability, or order-dependent global state.

True verification is difficult and often computationally expensive. Conceding this limit is essential: you cannot run exhaustive formal verification across an entire multi-million-line business application on every pull request. But you can stop pretending that five unit tests per file constitute an executable proof.

What to Audit on Monday

You do not need to rewrite your entire test infrastructure overnight to begin closing the gap between green tests and verified software. You can start by auditing your team's most critical paths:

1. Search for tautological tests: Grep your codebase for tests with no assertions, tests that assert against mocked returns rather than calculated outputs, or tests wrapped in permissive try/catch blocks that swallow unexpected errors.
2. Replace two point tests with property invariants: Pick a high-risk utility—a parser, a monetary calculator, or an authorization check. Replace static unit assertions with a property-based invariant test that checks thousands of variations across the boundary conditions.
3. Name your verdicts honestly: Distinguish internally between a build that merely compiles and passes basic unit checks, and an artifact that has cleared deep invariant testing. Acknowledging that standard CI output is provisional prevents teams from shipping untested edge cases under the false confidence of a green checkmark. See also The Gap Between Compilation and Correctness.

A green test runner is an invitation to test further, not a demonstration of proof. Treat it accordingly.