Evidence · Library v3.4.0 · Five public repositories, and our own
We ran it on code we don't control — then on our own
Five pre-purchase runs on repositories we don't control: a deliberately vulnerable teaching app, a frontier inference stack, a shipped Java game, a large commercial automation monorepo, and an autonomous agent platform. Same free download, same method, five different answers — including one that says hold, and several that tell the buyer not to buy for the headline features. Then a sixth, where an independent reviewer turned the instrument on this site and we published what it found, unedited.
Highlights come first on every run. The full sealed report is one click away if you want it, and stays out of your way if you don't.
Approved repairs fit verified components into the broken structure, then prove the result before anything is allowed to move.
Six rules every run obeyed
01
Host first, always
The repository is inventoried and frozen before the catalog is opened. The library is never allowed to decide what your problems are.
02
Rejection is the main output
Across these six runs, the overwhelming majority of catalogued rows were screened out. Nine of 41. Eight of thousands. That ratio is the product working.
03
Claims get demoted, not promoted
Every run states its ceiling: no bitwise-replay claim, no publisher-authenticity claim, no security certification. Unsupported claims are removed rather than softened.
04
Composites are the real value
In most of the runs, the strongest recommendation did not exist in the catalog or the host — it appeared only when compatible parts met a specific repository.
05
Money is stated both ways
Hard budget and counterfactual capability value are priced separately, including when hard budget is negative.
06
Evaluate-only means zero writes
With no purchase record, integration is blocked by the licence gate. Every run above changed nothing but its own report file.
Rejection first, in shape
candidates found rejection gates kept
---------------- --------------- ----
░░░░░░░░░░░░░░░ ──▶ [ located? ]──✗─┐
░░░░░░░░░░░░░░░ ──▶ [ cited? ]──✗─┤──▶ discarded, on paper
░░░░░░░░░░░░░░░ ──▶ [ builds? ]──✗─┘
░░░░░░░░░░░░░░░ ──▶ [ all three ]───✓───▶ ███ named capability
░A candidate. Most of a repository never becomes reusable capability.
✓Located, cited and built green — the only path onto the shelf.
DVWA ships 57 module-level combinations that must remain exploitable — it is a teaching tool. A library that 'fixes' it destroys it. The run refused the easy claim: it did not certify anything secure, kept every lesson intact, and instead designed a defended comparison lane beside the vulnerable one, plus a rule that a real-world finding cannot be called closed until adversarial regression and an independent retest both pass.
Engineered rows screened
1,117 / 1,117
Discovery rows screened
6,479 / 6,479
Dedicated cyber pass
301 / 301
Host lines changed
0
What the Evaluation Library got right
Read the host first and recognised the vulnerabilities as the product, not as defects to remove.
Stated its own ceiling in the opening paragraph: one pass cannot truthfully certify a real website secure.
Screened all 1,117 engineered rows and all 6,479 Discovery rows, plus a dedicated 301-row cyber pass, and rejected most of them.
Changed zero host lines — the licence gate held at EVALUATE ONLY because no purchase record was supplied.
What was new about this run
The strongest output was a Remediation Closure Gate: refuse to mark a finding closed merely because code changed.
A vulnerability-preserving defensive twin is a capability neither side had alone — the host's lessons plus the library's evidence lane.
DVWA's own deployment warnings became executable assertions without breaking classroom mode.
The run never touched weights. It found that the credible value sits around inference: durable cross-session memory with a generation-depth brake, a sealed evidence capsule for prompt/seed/checkpoint lineage that explicitly refuses to claim bitwise reproducibility across hardware, a drift monitor built from expert-routing counts DeepSeek already computes and throws away, and a redact-before-write envelope so adding persistence does not quietly create a retention surface.
Engineered rows parsed
1,117 / 1,117
Discovery rows parsed
6,479 / 6,479
Ranked recommendations
13
New composites
4
What the Evaluation Library got right
Scoped itself to an external layer around inference and said so up front.
Declined the deterministic-replay claim it could not support across precision modes and hardware.
Sequenced the work: persistence, erasure and replay proven before the larger consolidation engine is attempted.
Handled a split licence correctly — MIT code, separate model licence, and a separate licensor grant for the nonprofit twin.
What was new about this run
Value hid in telemetry that already existed: expert-selection counts were computed and discarded every forward pass.
Memory safety had to be designed with persistence, not after it — erasure is a day-one property.
Four host-specific composites came out of parts that individually did nothing for this repository.
Mindustry already owns the hard parts: packet abuse controls, admission checks, save rollback, ordered Steam backpressure. The report says so, prices the remaining value honestly at −$3,499 to +$14,501 net, and names a catalog hole rather than papering over it: nothing in the library is a correct general release-artifact authenticator, so the updater's executable-download boundary stays open. Language risk is stated too — TypeScript is canonical, Java is explicitly not claimed.
Recommended components
9 of 41 read in full
Composites
4
Net value range
−$3,499 to +$14,501
Licence break-even
3.33 engineer-hours
What the Evaluation Library got right
Refused a capability claim the catalog could not support, and marked the gap as the more important finding.
Published a negative net-value floor next to the positive ceiling instead of quoting only upside.
Made the buy conditional on a post-purchase Java bridge proof passing quickly.
Read 41 host-matched candidates in full before recommending nine.
What was new about this run
A mature host shrinks the honest surface area — and that is the useful signal, not a failure of the run.
The Update Evidence Gate composite ships with a deliberate stop: provenance and a host-owned SHA-256 gate, but no publisher-authenticity claim until a trusted digest source exists.
Break-even is stated two ways: 3.33 hours for the licence alone, 43–83 hours once integration is priced in.
A large commercial monorepo with 87 workspaces and mixed licences — the run that proves the method survives real-world licence complexity.
Host repository: github.com/n8n-io/n8n ↗ — read it yourself; we did not touch a line of it.
In 60 seconds
The report rejects most of the obvious matches: n8n's breaker, SSRF defences, encryption, API-key handling, evaluation tooling, supply-chain attestation and AI-credit accounting are already mature. What survives is forensic: a tamper-evident per-execution evidence capsule over the existing event bus, deterministic cause edges across workflow/node/queue/runner evidence, bounded failure domains for n8n-owned dependencies, and security-shaped fixtures turned into replayable regressions.
Recommended components
8
Composites
4
Counterfactual net
+$2.5k to +$22k
Workspaces licence-scanned
87
What the Evaluation Library got right
Froze a host inventory first and published the freeze hash (095b0387…) so the run is auditable.
Scanned sub-package licences — 69 sustainable-use, 15 MIT, 2 ISC, 1 Apache-2.0 — and routed attachment points away from the permissively published workspaces.
Stated an unresolved gap: the catalog's rate limiters expose no shared-state port, so no claim is made about multi-main limiter state.
Separated hard-budget net (negative) from counterfactual capability net (positive) instead of blending them.
What was new about this run
Maturity is not a dead end: the value moved from features to evidence — proving what an execution did, not doing it faster.
Attachment-point selection is a licence decision as much as an architectural one.
The strongest single item, the Execution Evidence Capsule, only exists when three catalog parts meet the host's event bus.
OpenClaw is not missing an agent platform. The report says that plainly and then narrows to five things it can defend: origin taint carried through assistant-derived text into memory admission, a provider-neutral spend governor built on existing cost telemetry, rehearsal of learned mutations before promotion, post-load plugin behaviour drift, and cross-agent dream federation of derived artifacts only — never raw cross-agent memory.
Verdict
Buy conditionally at $499
Condition
≥2 top composites on the roadmap
Host seams named
5
Host lines changed
0
What the Evaluation Library got right
Told the buyer not to buy for the headline features, in the summary, above the fold.
Used the host's own maturity ledger as evidence — Plugin Trust at 68/100, Alpha, not LTS — instead of asserting weakness.
Named a kit documentation defect: retired free/annual licence wording that conflicts with the authoritative $499 perpetual terms.
Attached each recommendation to a real seam in the source, not to a vague subsystem.
What was new about this run
Provenance is the hole strong agent stacks share: origin metadata dies the moment a model paraphrases tool output.
Federating derived artifacts is a way to share cognition without reopening private memory.
The conditional is explicit and checkable: buy only if at least two of the top composites are genuinely on the roadmap.
Run 06 · This site and its MCP server · TypeScript · proprietary
Verdict: Method sound, hygiene not
SHPBL itself
Five runs read other people's repositories. The sixth reads ours — the only study where printing the result cost the author something.
In 60 seconds
An independent reviewer pointed the instrument at the instrument, using nothing but the public MCP surface. It held on staging and on refusals — a wrong repo name explains itself, a bad component ID returns the good rows plus a `missing` array, an empty vault result tells the model to say 'not present' plainly. It failed on hygiene: an ignore file that never named environment files, a literal substring search that made a 7,596-row library answer 'not present' to ordinary questions, and three disagreeing version numbers. Published unedited, with every repair shipped in server 1.13.0.
Findings
9, printed unedited
High severity
2 — ignore rules, multi-word search
Fixed in
MCP server 1.13.0
Host lines changed by the run
0
What the Evaluation Library got right
Ran on the published surface only — no privileged access, no author's help, no model called by the server.
Changed zero lines of the host: the run read, and every repair was written afterwards by hand.
Confirmed the arithmetic that carries the claims — the component classes sum to exactly 1,117.
Refused the flattering reading of a committed environment file: publishable keys, so not a breach, but a structural defect all the same.
What was new about this run
A single `.includes()` can make a whole library look empty — search was ANDing nothing and returning nothing for any plain-words phrase.
Protocol shipped inside every payload cost the caller more context than their own source; the method now lives in one free tool, with `brief: true` on the paid ones.
Metering and version strings must be derived from the one table the gate reads, or they drift and a customer cannot tell what charges.
Combinations designed for this host
Ranked multi-term ledger search with closest-match fallback
Download the Evaluation Master Library, drop it in the repository you care about, and tell your coding agent to follow the kit. It reads your code first, screens the catalog against what it found, and writes a report like the six above — with your file paths, your licences, and your numbers. Nothing is copied into your codebase, and a clean “leave it alone” is a legitimate result.
