Embedding Certification Harnesses to Verify SHPBL Catalog Entries

Stubs in the registry break catalog trust

A component count looks tidy on a dashboard. The numbers are easy to quote. The reality is that half of those rows are placeholders that never run. When a build asks for a library and receives a stub, the build fails or, worse, silently falls back to a no‑op. The catalog no longer represents a software inventory; it becomes a wish list. The failure mode is clear: registry execution returns “not found” or “no‑op” without a verdict. Without a concrete outcome, downstream teams cannot rely on the count.

Turn every entry into a verifiable artifact

The only way to move from claim to inventory is to execute each entry against a known contract. SHPBL provides a certification harness that runs an artifact and records one of four verdicts: CERTIFIED, PROVISIONAL, INCONCLUSIVE, or FAILED. The harness does not guess; it computes. When the harness reports CERTIFIED, the row is proven to be executable and correct. When it reports FAILED, the entry is a stub or broken implementation that must be fixed or removed.

A minimal verification step looks like this:

shpbl-harness run --artifact path/to/component.tar.gz

The command returns a JSON verdict. The harness also records the checksum of the artifact, linking the result to the exact byte‑for‑byte release. This creates a root of trust that is independent of any model or external inference.

Hook verification into the existing CI pipeline

Running the harness manually for every component is impractical. The solution is to embed the verification as a CI job that mirrors the build intent gate already used for writes. When a pull request adds or updates a component, the CI job:

1. Packages the component into the SHPBL archive format.
2. Calls the harness via the HTTP API (/api-access) to obtain a verdict.
3. Fails the CI run if the verdict is anything other than CERTIFIED or PROVISIONAL.

Because the same API powers both the agent and the offline client, the CI job can be written once and reused across environments. The result is a continuously refreshed catalog where every row has an up‑to‑date verification status. Teams downstream can query the catalog and filter on the verdict field, confident that a “CERTIFIED” row will run as expected.

Keep the verification scope realistic

The harness cannot exercise every possible input contract. When it returns INCONCLUSIVE, the failure is a limitation of the harness, not a defect in the component. Recognising this boundary prevents endless churn trying to force a verdict where the test harness simply lacks the required fixture. A pragmatic policy is to treat INCONCLUSIVE entries as “to be revisited” rather than “failed”. Over time, as test fixtures expand, those rows can be upgraded to PROVISIONAL or CERTIFIED.

The broader point is echoed in the pillar post [/blog/a-component-count-is-not-an-inventory] – a count alone is a claim, not an inventory. Only when each claim is backed by an execution verdict does the catalog become trustworthy.

Monday action: add a verification job

Pick a single repository that contributes to your catalog. Create a CI job that runs the harness against the latest component archive using the HTTP API. Configure the job to fail on any verdict other than CERTIFIED or PROVISIONAL. Merge the change and watch the CI result. By the end of the day you will have a concrete, reproducible proof that at least one registry entry is verified, and a template you can roll out to the rest of the catalog.