Skip to content
SHPBL

Evidence · Library v3.2.0 · Five public repositories, and our own

We ran it on code we don't control — then on our own

Five pre-purchase runs on repositories we don't control: a deliberately vulnerable teaching app, a frontier inference stack, a shipped Java game, a large commercial automation monorepo, and an autonomous agent platform. Same free download, same method, five different answers — including one that says hold, and several that tell the buyer not to buy for the headline features. Then a sixth, where an independent reviewer turned the instrument on this site and we published what it found, unedited.

Highlights come first on every run. The full sealed report is one click away if you want it, and stays out of your way if you don't.

Run this on your repo — free1.3 MB · ZIPHow the ten-minute run works

Six rules every run obeyed

01

Host first, always

The repository is inventoried and frozen before the catalog is opened. The library is never allowed to decide what your problems are.

02

Rejection is the main output

Across these five runs, the overwhelming majority of catalogued rows were screened out. Nine of 41. Eight of thousands. That ratio is the product working.

03

Claims get demoted, not promoted

Every run states its ceiling: no bitwise-replay claim, no publisher-authenticity claim, no security certification. Unsupported claims are removed rather than softened.

04

Composites are the real value

The strongest recommendation in four of five runs did not exist in the catalog or the host — it appeared only when compatible parts met a specific repository.

05

Money is stated both ways

Hard budget and counterfactual capability value are priced separately, including when hard budget is negative.

06

Evaluate-only means zero writes

With no purchase record, integration is blocked by the licence gate. Every run above changed nothing but its own report file.

The five runs

Run 01 · Damn Vulnerable Web Application · PHP · GPL-3.0-or-later

Verdict: Buy conditionally

DVWA

A host that is supposed to stay broken. The only way to prove the method does not lie about security.

Host repository: github.com/digininja/DVWA ↗ — read it yourself; we did not touch a line of it.

In 60 seconds

DVWA ships 57 module-level combinations that must remain exploitable — it is a teaching tool. A library that 'fixes' it destroys it. The run refused the easy claim: it did not certify anything secure, kept every lesson intact, and instead designed a defended comparison lane beside the vulnerable one, plus a rule that a real-world finding cannot be called closed until adversarial regression and an independent retest both pass.

Engineered rows screened
1,116 / 1,116
Discovery rows screened
6,479 / 6,479
Dedicated cyber pass
301 / 301
Host lines changed
0

What the Evaluation Library got right

  • Read the host first and recognised the vulnerabilities as the product, not as defects to remove.
  • Stated its own ceiling in the opening paragraph: one pass cannot truthfully certify a real website secure.
  • Screened all 1,116 engineered rows and all 6,479 Discovery rows, plus a dedicated 301-row cyber pass, and rejected most of them.
  • Changed zero host lines — the licence gate held at EVALUATE ONLY because no purchase record was supplied.

What was new about this run

  • The strongest output was a Remediation Closure Gate: refuse to mark a finding closed merely because code changed.
  • A vulnerability-preserving defensive twin is a capability neither side had alone — the host's lessons plus the library's evidence lane.
  • DVWA's own deployment warnings became executable assertions without breaking classroom mode.

Composites designed for this host

  • Vulnerability-Preserving Defensive Twin
  • Security Regression Matrix
  • Remediation Closure Gate
  • Exploit-to-Evidence Replay Capsule
  • Safe-Lab Exposure Lock
Read the full DVWA report+

Open in a new tab · Download the sealed HTML · 97 KB · prints to PDF · no network calls

Run 02 · Frontier inference stack · Python · MIT code + model licence

Verdict: Proceed — external layer, not a rewrite

DeepSeek-V3

The hardest possible flattery test: a world-class ML repository where the wrong answer is 'add our stuff to the model'.

Host repository: github.com/deepseek-ai/DeepSeek-V3 ↗ — read it yourself; we did not touch a line of it.

In 60 seconds

The run never touched weights. It found that the credible value sits around inference: durable cross-session memory with a generation-depth brake, a sealed evidence capsule for prompt/seed/checkpoint lineage that explicitly refuses to claim bitwise reproducibility across hardware, a drift monitor built from expert-routing counts DeepSeek already computes and throws away, and a redact-before-write envelope so adding persistence does not quietly create a retention surface.

Engineered rows parsed
1,116 / 1,116
Discovery rows parsed
6,479 / 6,479
Ranked recommendations
13
New composites
4

What the Evaluation Library got right

  • Scoped itself to an external layer around inference and said so up front.
  • Declined the deterministic-replay claim it could not support across precision modes and hardware.
  • Sequenced the work: persistence, erasure and replay proven before the larger consolidation engine is attempted.
  • Handled a split licence correctly — MIT code, separate model licence, and a separate licensor grant for the nonprofit twin.

What was new about this run

  • Value hid in telemetry that already existed: expert-selection counts were computed and discarded every forward pass.
  • Memory safety had to be designed with persistence, not after it — erasure is a day-one property.
  • Four host-specific composites came out of parts that individually did nothing for this repository.

Composites designed for this host

  • Persistent Dream Memory Loop
  • Inference Evidence Capsule
  • MoE Routing Drift Monitor
  • Memory Safety Envelope
Read the full DeepSeek-V3 report+

Open in a new tab · Download the sealed HTML · 100 KB · prints to PDF · no network calls

Run 03 · Shipped Java game · GPL-3.0-only · a4974ec

Verdict: Hold / conditional buy

Mindustry

The run that argues against the purchase — kept in unedited, because a library that never says no cannot be trusted when it says yes.

Host repository: github.com/Anuken/Mindustry ↗ — read it yourself; we did not touch a line of it.

In 60 seconds

Mindustry already owns the hard parts: packet abuse controls, admission checks, save rollback, ordered Steam backpressure. The report says so, prices the remaining value honestly at −$3,499 to +$14,501 net, and names a catalog hole rather than papering over it: nothing in the library is a correct general release-artifact authenticator, so the updater's executable-download boundary stays open. Language risk is stated too — TypeScript is canonical, Java is explicitly not claimed.

Recommended components
9 of 41 read in full
Composites
4
Net value range
−$3,499 to +$14,501
Licence break-even
3.33 engineer-hours

What the Evaluation Library got right

  • Refused a capability claim the catalog could not support, and marked the gap as the more important finding.
  • Published a negative net-value floor next to the positive ceiling instead of quoting only upside.
  • Made the buy conditional on a post-purchase Java bridge proof passing quickly.
  • Read 41 host-matched candidates in full before recommending nine.

What was new about this run

  • A mature host shrinks the honest surface area — and that is the useful signal, not a failure of the run.
  • The Update Evidence Gate composite ships with a deliberate stop: provenance and a host-owned SHA-256 gate, but no publisher-authenticity claim until a trusted digest source exists.
  • Break-even is stated two ways: 3.33 hours for the licence alone, 43–83 hours once integration is priced in.

Composites designed for this host

  • Packet Boundary Lab
  • Outbound Dependency Fuse
  • Admin Decision Black Box
  • Update Evidence Gate (with a deliberate stop)
Read the full Mindustry report+

Open in a new tab · Download the sealed HTML · 140 KB · prints to PDF · no network calls

Run 04 · Automation platform · TypeScript monorepo · sustainable-use licence

Verdict: Buy

n8n

A large commercial monorepo with 87 workspaces and mixed licences — the run that proves the method survives real-world licence complexity.

Host repository: github.com/n8n-io/n8n ↗ — read it yourself; we did not touch a line of it.

In 60 seconds

The report rejects most of the obvious matches: n8n's breaker, SSRF defences, encryption, API-key handling, evaluation tooling, supply-chain attestation and AI-credit accounting are already mature. What survives is forensic: a tamper-evident per-execution evidence capsule over the existing event bus, deterministic cause edges across workflow/node/queue/runner evidence, bounded failure domains for n8n-owned dependencies, and security-shaped fixtures turned into replayable regressions.

Recommended components
8
Composites
4
Counterfactual net
+$2.5k to +$22k
Workspaces licence-scanned
87

What the Evaluation Library got right

  • Froze a host inventory first and published the freeze hash (095b0387…) so the run is auditable.
  • Scanned sub-package licences — 69 sustainable-use, 15 MIT, 2 ISC, 1 Apache-2.0 — and routed attachment points away from the permissively published workspaces.
  • Stated an unresolved gap: the catalog's rate limiters expose no shared-state port, so no claim is made about multi-main limiter state.
  • Separated hard-budget net (negative) from counterfactual capability net (positive) instead of blending them.

What was new about this run

  • Maturity is not a dead end: the value moved from features to evidence — proving what an execution did, not doing it faster.
  • Attachment-point selection is a licence decision as much as an architectural one.
  • The strongest single item, the Execution Evidence Capsule, only exists when three catalog parts meet the host's event bus.

Composites designed for this host

  • Execution Evidence Capsule
  • Provider Failure Cell
  • Causal Run Ledger
  • Boundary Regression Replay Pack
Read the full n8n report+

Open in a new tab · Download the sealed HTML · 118 KB · prints to PDF · no network calls

Run 05 · Autonomous agent platform · TypeScript · MIT

Verdict: Buy conditionally

OpenClaw

A host that already has memory, agents, sandboxing and a dreaming engine — so the run had to find the delta or admit there wasn't one.

Host repository: github.com/openclaw/openclaw ↗ — read it yourself; we did not touch a line of it.

In 60 seconds

OpenClaw is not missing an agent platform. The report says that plainly and then narrows to five things it can defend: origin taint carried through assistant-derived text into memory admission, a provider-neutral spend governor built on existing cost telemetry, rehearsal of learned mutations before promotion, post-load plugin behaviour drift, and cross-agent dream federation of derived artifacts only — never raw cross-agent memory.

Verdict
Buy conditionally at $499
Condition
≥2 top composites on the roadmap
Host seams named
5
Host lines changed
0

What the Evaluation Library got right

  • Told the buyer not to buy for the headline features, in the summary, above the fold.
  • Used the host's own maturity ledger as evidence — Plugin Trust at 68/100, Alpha, not LTS — instead of asserting weakness.
  • Named a kit documentation defect: retired free/annual licence wording that conflicts with the authoritative $499 perpetual terms.
  • Attached each recommendation to a real seam in the source, not to a vague subsystem.

What was new about this run

  • Provenance is the hole strong agent stacks share: origin metadata dies the moment a model paraphrases tool output.
  • Federating derived artifacts is a way to share cognition without reopening private memory.
  • The conditional is explicit and checkable: buy only if at least two of the top composites are genuinely on the roadmap.

Composites designed for this host

  • Origin-Taint Memory Firewall
  • Spend-Safe Autonomy Governor
  • Runtime Plugin Immune Gate
  • Dream federation without replacing Dreaming
Read the full OpenClaw report+

Open in a new tab · Download the sealed HTML · 108 KB · prints to PDF · no network calls

Run 06 · This site and its MCP server · TypeScript · proprietary

Verdict: Method sound, hygiene not

SHPBL itself

Five runs read other people's repositories. This one reads ours — the only study where printing the result cost the author something.

In 60 seconds

An independent reviewer pointed the instrument at the instrument, using nothing but the public MCP surface. It held on staging and on refusals — a wrong repo name explains itself, a bad component ID returns the good rows plus a `missing` array, an empty vault result tells the model to say 'not present' plainly. It failed on hygiene: an ignore file that never named environment files, a literal substring search that made a 7,595-row library answer 'not present' to ordinary questions, and three disagreeing version numbers. Published unedited, with every repair shipped in server 1.13.0.

Findings
9, printed unedited
High severity
2 — ignore rules, multi-word search
Fixed in
MCP server 1.13.0
Host lines changed by the run
0

What the Evaluation Library got right

  • Ran on the published surface only — no privileged access, no author's help, no model called by the server.
  • Changed zero lines of the host: the run read, and every repair was written afterwards by hand.
  • Confirmed the arithmetic that carries the claims — the component classes sum to exactly 1,116.
  • Refused the flattering reading of a committed environment file: publishable keys, so not a breach, but a structural defect all the same.

What was new about this run

  • A single `.includes()` can make a whole library look empty — search was ANDing nothing and returning nothing for any plain-words phrase.
  • Protocol shipped inside every payload cost the caller more context than their own source; the method now lives in one free tool, with `brief: true` on the paid ones.
  • Metering and version strings must be derived from the one table the gate reads, or they drift and a customer cannot tell what charges.

Composites designed for this host

  • Ranked multi-term ledger search with closest-match fallback
  • Protocol-out-of-payload (`method_protocol` + `brief`)
  • Single-source version and metering derivation
  • Spine heuristic with generated and vendored noise filtered
Read the full SHPBL itself report+

Open in a new tab · Download the sealed HTML · 11 KB · prints to PDF · no network calls

Your turn

Ten minutes, your repository, your agent.

Download the Evaluation Master Library, drop it in the repository you care about, and tell your coding agent to follow the kit. It reads your code first, screens the catalog against what it found, and writes a report like the five above — with your file paths, your licences, and your numbers. Nothing is copied into your codebase, and a clean “leave it alone” is a legitimate result.

Download the Evaluation Master Library — FreeOwn the Complete Master Library — $499