Distinguishing Input Contract Failures from Component Defects

The failure mode is not always the component

A green suite can still lie to you.

The common mistake is simple. A test harness supplies an input the component never promised to handle, the run aborts, and the result gets filed as a defect in the component. That pollutes release signals. It makes the artifact look broken when the real failure is in the contract between the harness and the thing under test.

That distinction matters in code certification. A verdict is only useful if it names the thing that failed. If the harness missed the input contract, the correct posture is not “broken”. It is “inconclusive”.

What an input contract miss looks like

Think about a parser that accepts only UTF-8 text files with a header row. The harness feeds it a binary blob renamed as .csv. The process exits early. No useful behaviour is exercised. The harness has not shown that the parser fails on valid input. It has shown that the harness did not respect the input contract.

If you mark that run as FAILED, you create two problems.

First, you train people to ignore failures. They learn that some red runs are just harness noise.

Second, you destroy the signal you actually need. A FAILED verdict should mean the component was exercised on its promised inputs and did not meet the expected behaviour. If you use it for contract misses, you can no longer tell a defect from a bad test.

The right classification is INCONCLUSIVE. That does not mean “nothing happened”. It means the test harness could not exercise the input contract. That is a harness limit, not an artifact defect.

This is the same reason the pillar post on the four verdicts between green tests and correct code draws a hard line between verdicts. The line is not bureaucratic. It is operational.

Why this preserves verification rigor

Some teams worry that calling a run INCONCLUSIVE is a soft option. It is not.

A strict harness should still reject the run. It should still record what it tried, what contract it assumed, and where the exercise stopped. It should still force a human or an upstream fix to resolve the mismatch before the result can be treated as evidence.

The difference is where the blame lands.

• FAILED means the component was exercised within its declared contract and did not satisfy the check.
• INCONCLUSIVE means the harness could not establish that it had a valid basis to judge the component.

That is not a downgrade in rigour. It is rigour applied correctly.

The failure mode is especially common in verification pipelines that mix broad discovery with narrow certification. A harness may discover that an artifact exists, but not that it can be safely driven with the inputs it has been given. If the run is then stamped as a defect, the release process starts chasing ghosts. The result is noisy triage, slower review, and eventually distrust in the whole code certification flow.

A provisional verdict has its own place. It says the artifact is reproducible and has been exercised, but correctness has not yet been asserted. It is not a substitute for a contract-aware verdict. If the harness never got a valid input contract in the first place, provisional is too generous. INCONCLUSIVE is the honest label.

A worked example from a CI gate

Suppose your CI job certifies a command-line tool that reads a config file and writes a report.

The harness does three things:

1. It creates a temp directory.
2. It invokes the binary with --config /tmp/test.
3. It checks for a report file.

If the tool expects /tmp/test to be a file and the harness passes a directory, the invocation fails immediately. The harness has not discovered a product defect. It has discovered that it did not supply the required input shape.

The correct response is to fix the harness, not the component verdict.

A good harness records that the input contract was not satisfied. It should say, in effect: the artifact was not exercised on a valid config file, so no correctness claim is available. That keeps the release signal clean. It also gives the next engineer a concrete repair path.

This is where the test harness discipline pays off. The harness is not just a runner. It is part of the verification boundary. If it cannot establish the preconditions for a meaningful check, it should not pretend otherwise.

Monday’s move

Pick one flaky or noisy certification job and inspect the first failure, not the last red line.

Ask three questions:

• Did the harness supply an input the component actually promised to handle?
• Did the run reach the behaviour under test, or did it stop at contract setup?
• If the contract was missed, should the verdict be INCONCLUSIVE rather than FAILED?

Then change the gate so it distinguishes those cases explicitly. Keep the failure, but name it correctly. That one change makes your verification signal sharper without making it looser.