Layered Verdicts: Using PROVISIONAL as a Feature Flag Gate

A standard CI pipeline passes because nothing crashed during execution. Tests ran, mocks returned their canned objects, coverage thresholds checked out, and the artifact built without a warning. Then someone flips the feature flag in production, and the component fails on its first real database transaction because the mock hid an incompatible schema expectation.

Green suites prove that your test harness survived running your test files. They do not prove that an artifact executes cleanly against real boundary conditions.

Relying on a binary pass/fail status to authorise a production feature flag rollout is dangerous because it ignores the ladder of certainty between compilation and verified execution. To manage risk sensibly, you need layered certification states. Specifically, you need a rule that prevents any feature flag from being toggled on in production unless the underlying code has achieved at least a PROVISIONAL verdict from an execution harness.

What PROVISIONAL guarantees (and what it does not)

In a rigorous certification harness, verdicts are not binary. As detailed in our breakdown of the four verdicts between green tests and correct code, code lives along a spectrum:

1. FAILED: The code failed compilation, execution, or explicit assertions.
2. INCONCLUSIVE: The harness could not exercise the contract (often due to missing input definitions or environmental boundaries).
3. PROVISIONAL: The component executes repeatedly without runtime panics or harness violations across baseline executions, but formal semantic correctness has not yet been asserted.
4. CERTIFIED: Invariants, property tests, and boundary assertions have all completed successfully.

PROVISIONAL is an honest label. It admits that you have not yet proven mathematical or full business-logic correctness across every edge case. What it does verify is repeatability: the artifact can be loaded, its entry points invoked, and its baseline execution path traversed without crashing or violating runtime invariants.

This distinction is critical for release engineering. A feature guarded by a flag should never ship enabled if its artifact is merely INCONCLUSIVE or untested in isolation. Requiring a PROVISIONAL status establishes a non-negotiable floor: code behind a flag can sit dark in a binary, but the flag cannot be toggled to evaluate live traffic until that baseline execution proof exists.

Gating the flag manifest in CI

To enforce this, treat your feature-flag configuration as an artifact that depends directly on harness verdicts. If a developer sets enabled: true for a flag whose backing capability only scored INCONCLUSIVE or FAILED, CI fails the rollout step.

Here is a minimal pattern using a script in a pipeline that validates a rollout manifest against the harness verdict manifest:

import json
import sys

def validate_rollout(flags_path, verdicts_path):
    with open(flags_path) as f:
        flags = json.load(f)
    with open(verdicts_path) as f:
        verdicts = json.load(f)

    allowed_states = {"PROVISIONAL", "CERTIFIED"}
    violations = []

    for flag_name, config in flags.items():
        if not config.get("enabled", False):
            continue

        capability_id = config.get("capability_id")
        verdict = verdicts.get(capability_id, {}).get("verdict")

        if verdict not in allowed_states:
            violations.append(
                f"Flag '{flag_name}' targets capability '{capability_id}' "
                f"with invalid verdict '{verdict}'. Must be PROVISIONAL or CERTIFIED."
            )

    if violations:
        print("Feature flag gate failed:\n" + "\n".join(violations), file=sys.stderr)
        sys.exit(1)

    print("All enabled feature flags satisfy minimum harness certification.")

if __name__ == "__main__":
    validate_rollout(sys.argv[1], sys.argv[2])

By placing this check prior to deploying flag configurations to your control plane (whether LaunchDarkly, Unleash, or a flat configuration store), you separate trunk integration from production exposure.

Developers can merge incomplete or exploratory implementations behind dark flags. But the moment an operational intent changes—switching a flag from dark to active—the CI gate checks for the harness record. If the capability has not been executed through the baseline verification pipeline, the deployment stops. This closes the gap between compilation and correctness without slowing down trunk-based development.

What this changes on Monday

Do not try to rewrite your entire test suite to emit formal proofs this week. Start with the pipeline gate.

1. Map flags to capabilities. Add an explicit field in your feature-flag metadata that ties the flag to the specific module or package it exposes.
2. Record the execution verdict. Have your test runner output a machine-readable verdict file during test runs, distinguishing between skipped/inconclusive suites and suites that actually exercised the code.
3. Block active rollouts on unverified states. Allow developers to merge code with dark flags, but fail any PR that attempts to toggle enabled: true without a verified run on record.

Treating PROVISIONAL as your minimum production threshold gives you safety where it matters most: at the point of customer exposure.