Post‑Merge Runtime Validation for Automated Repairs

The missing verification step

Audits expose defects and suggest repairs, but the suggested change often stops at the pull‑request. Teams merge the automated PR after static analysis passes, without exercising the repaired code in a realistic environment. The merged PR can still break a production path, leaving a regression undetected.

A lightweight runtime validation suite

Run a focused set of runtime checks immediately after the PR lands. The suite must:

• target only the code paths changed by the repair,
• execute in a sandbox that mirrors production configuration,
• finish in a few seconds so CI is not blocked.

Because the scope is narrow, the suite can be expressed as a handful of entry‑point calls with representative inputs. If the repair fixes a null‑dereference in processOrder, the validation script can invoke processOrder(sampleOrder) with a payload that triggers the previously unsafe branch. The harness records the outcome; a clean run yields a PROVISIONAL verdict, indicating that the repair passed a real‑world sanity check.

import { runValidation } from '@shpbl/sdk';

const changed = ['src/order.ts'];

const fixtures = [
  { orderId: 'test-001', items: [{ sku: 'A1', qty: 1 }] },
];

runValidation({ changed, fixtures })
  .then(r => console.log('Provisional verdict:', r.verdict));

The same typed client that powers the SHPBL harness runs the script, so the validation environment matches the one that produced the original CERTIFIED or PROVISIONAL status.

Integrating validation with the audit‑repair pipeline

A practical workflow inserts the validation step into the existing loop:

1. An audit produces a repair suggestion and creates an automated PR.
2. The PR merges automatically after static analysis reports PROVISIONAL.
3. A post‑merge job triggers the runtime validation suite.
4. If the suite passes, the harness records a PROVISIONAL verdict for the merged artifact.
5. If the suite fails, the job aborts, the PR is reverted, and the audit receives a new suggestion.

Because the same harness evaluates both static and runtime checks, the verdicts are comparable. The loop therefore moves from “audit‑repair‑report” to “audit‑repair‑validated”.

The approach does not remove all risk. Defects that appear only under load or with external services may remain hidden; the harness will return INCONCLUSIVE in those cases. Treat INCONCLUSIVE as a signal to expand the fixture set or to schedule a deeper integration test.

How SHPBL makes it practical

SHPBL offers four identical access points—MCP chatbot, HTTP API, typed TypeScript client, and offline file‑based edition—for static and runtime validation. The certification harness that records PROVISIONAL verdicts is available at /harness. Re‑using the harness after an automated PR merges provides a repeatable, model‑independent check that fits into any CI pipeline. The method is described in the pillar post /blog/closing-the-loop-from-repository-audit-to-merged-repair.

What to try on Monday

1. Pick a recent audit‑repair PR in your repository.
2. Add a post‑merge job that calls runValidation with the changed files and a minimal fixture set.
3. Observe the verdict recorded by the SHPBL harness.

If the job reports PROVISIONAL, the repair has passed a runtime sanity check. If it reports INCONCLUSIVE or FAILED, extend the fixture set until the repaired paths are exercised. This addition closes the audit‑repair loop without altering existing tooling.