Validating Upstream License Invariants in Vendored Tarballs

A vendored tarball lands in your repository under vendor/ or third_party/. Its top-level manifest claims MIT. The build passes, the archive is committed to version control, and your compliance checklist is marked complete.

Six months later, an audit reveals that four performance-critical source files inside that tarball were lifted from an upstream repository distributed under GPL-3.0-only. The root LICENSE file declared one thing; the individual file headers declared another.

When you treat vendored code as an opaque blob, your software provenance is built on optimism. As argued in our analysis of why vendoring code without a paper trail is unsecured debt, unverified third-party code accumulates liability silently. Checking only the root package metadata does not validate your vendored code license. It only validates what someone typed into a manifest.

Traversing the Archive Beyond the Root Manifest

Package registries and archive manifests describe build intentions, not file-level reality. Maintainers bundle snippets from secondary sources, copy test fixtures with restrictive licenses, or change terms without updating the archive root.

A reliable invariant check must inspect the concrete files inside the unpacked tree. This requires integrating a static extraction step directly into your license compliance CI pipeline:

1. Unpack the tarball into an ephemeral verification directory.
2. Parse the Abstract Syntax Tree (AST) comments or lexical tokens of every source file.
3. Extract explicit SPDX-License-Identifier expressions and copyright blocks.
4. Compare every extracted token against an explicit policy matrix.
5. Fail the build if any file contains an incompatible, undeclared, or missing license header.

{
  "archive": "vendor/cache/codec-core-2.1.0.tar.gz",
  "sha256": "8f434346648f6b96df89dda901c5176b10f607624f37d1e7f707e7c7e53da82f",
  "declared_license": "Apache-2.0",
  "rules": {
    "allow": ["Apache-2.0", "MIT", "BSD-3-Clause"],
    "deny": ["GPL-2.0-only", "GPL-3.0-only", "SSPL-1.0"],
    "require_spdx_per_file": true
  }
}

This model shifts the verification boundary from trust to proof. It turns vendoring from an informal file copy into a verifiable step toward replacing folder-of-hope vendoring with verified SBOMs.

Where Automated Parsing Hits Real Limits

Automated header inspection is not free of edge cases. Three concrete failure modes require disciplined handling:

First, legacy and bespoke grants. Many older repositories do not use standard SPDX expressions. They include plain-text headers like "Licensed under the same terms as Perl itself" or custom multi-sentence permissions. An AST parser encountering non-standard text cannot reliably infer a machine-readable identifier.

Second, dual-license expressions. An upstream file marked SPDX-License-Identifier: MIT OR GPL-2.0-or-later is acceptable if your policy permits MIT. But if the author wrote MIT AND GPL-2.0-or-later, your downstream distribution must satisfy both grants. Parsers must evaluate Boolean license expressions correctly rather than matching on simple substrings.

Third, generated or minified code. Tools that concatenate or transpile sources often drop header comments entirely or aggregate multiple conflicting notices into a single comment block.

When the scanner cannot determine the provenance of a file, the correct posture is not to guess. The CI job must mark the invariant check as failed. If a human reviewer confirms that a bespoke header satisfies your legal criteria, record that exception in the invariant declaration alongside the file's SHA-256 hash. Never allow the scanner to ignore unparsed files by default.

Implementing the Invariant Gate

You do not need a massive enterprise platform to stop license leaks. You need a strict CI gate that treats vendored archives as untrusted inputs.

On Monday, put three controls in place:

1. Pin and hash the archive. Store the upstream tarball with an explicit SHA-256 digest. Verify the checksum before unpacking. If the archive digest changes upstream, the build halts before any file reaches your repository tree.
2. Scan headers per file in CI. Run lexical extraction across every unpacked source file. Evaluate SPDX identifiers and Boolean expressions against your policy allowlist, failing the build on any missing, denied, or unparseable license expression.
3. Record hash-pinned exceptions. For files with bespoke grants approved by legal review, check in an explicit override file containing the target file path, the reviewed license classification, and the file's SHA-256 digest. Any subsequent upstream modification invalidates the hash and forces a re-review.