Debt Scorecards: Prioritizing Refactors from Audit Data

Audits stall at the report

Teams run static analysis, collect lint warnings and architectural violations, then hand a PDF to management. The PDF never drives a change. The same module is revisited in the next audit, the debt re‑appears, and the cycle repeats. The failure is not the audit itself; it is the missing bridge between finding and fixing. Without a concrete signal that ranks the work, engineers default to “fix what feels urgent”, which usually means nothing gets fixed.

Turn findings into a debt scorecard

A scorecard assigns a numeric debt value to each module. The value is the sum of weighted violations:

{
  "module": "payment/processor",
  "score": 42,
  "breakdown": {
    "critical": 3 * 10,
    "high": 5 * 5,
    "medium": 8 * 2,
    "low": 12 * 1
  }
}

Critical violations carry the highest weight because they block releases or introduce security risk. High and medium items reflect maintainability pain. Low items capture style drift. The weights are configurable per organisation, but the principle stays the same: a single number reflects the repair effort required to bring the module to an acceptable state.

The scorecard is generated directly from the audit JSON that the existing pipeline already produces. No new analysis tool is required; the pipeline simply adds a reduction step that aggregates the counts and applies the weight table. Because the process is deterministic, the same input always yields the same score, making the metric repeatable across runs.

Surface the scores in a dashboard

A lightweight web view reads the score JSON files from the build artefacts and renders a sortable table. The table shows:

• Module name
• Debt score
• Trend over the last three audits

Sorting by score highlights the hottest spots. Colour‑coding (green, amber, red) gives an at‑a‑glance health indicator. The dashboard updates automatically whenever the CI pipeline finishes, so the data never lags behind the code.

Because the dashboard consumes the same artefacts that the certification harness runs against, the view is always consistent with the test results. When a module’s score drops from 42 to 15 after a repair, the dashboard reflects the change immediately, reinforcing the link between the repair and the measured improvement.

Hook the scorecard into the repair workflow

The next step is to turn a high‑score module into a repair branch automatically. The workflow is:

1. The dashboard flags a module whose score exceeds a configurable threshold.
2. A CI job invokes the repair generator, passing the module identifier.
3. The generator creates a branch that contains only the changes required to resolve the violations counted in the score.
4. The branch is submitted to the certification harness. If the harness returns CERTIFIED or PROVISIONAL, the branch can be merged automatically.

This loop closes the gap that the pillar post Closing the loop from repository audit to merged repair describes. The scorecard supplies the prioritisation signal; the harness supplies the safety net; the merge completes the repair. No manual triage is needed beyond setting the threshold.

The approach also respects the limits of the harness. If the harness cannot exercise a particular contract, it returns INCONCLUSIVE. The workflow treats that as a signal to defer the repair until the contract is clarified, rather than marking the module as “fixed”. This honesty prevents false confidence.

What to try on Monday

1. Export the latest audit JSON from your CI pipeline.
2. Write a small script that applies the weight table shown above and produces a debt-score.json file per module.
3. Add a step to your pipeline that publishes the JSON files as artefacts.
4. Point a simple static page at the artefacts and verify the table sorts correctly.

If the script runs without errors, you already have a quantitative debt scorecard. The next sprint can focus on wiring the scorecard into your existing repair generator. The effort required is a few hours of scripting, not a full redesign, and the payoff is a measurable reduction in repeat audits.