Extracting Origin Commits from Unversioned In-Tree Vendor Dumps

You inherit a repository with third_party/zstd/ committed directly into the main branch. There is no git submodule, no release tag in the commit log, and no vendor manifest. Someone downloaded a snapshot on a Tuesday in 2021, deleted the upstream test suite to keep the repository light, and committed it with the message "import zstd".

Now an audit lands on your desk, or a CVE is published for that library between versions 1.4.5 and 1.5.0. You cannot prove which upstream commit this code derived from, whether local patches were applied, or which license text governed that exact point in time.

Vendoring code without a paper trail is unsecured debt. Leaving unversioned source folders in your tree turns basic software provenance into guesswork.

Reconstructing the origin commit is not guesswork. It is a deterministic search across git tree objects.

Why Naive Diffing Fails

The obvious approach—checking out successive upstream release tags and running diff -r against your local folder—almost always fails on real-world vendor drops.

First, engineers rarely copy clean trees. They strip documentation, ignore build scripts, or reformat line endings. Second, teams often copy from an untagged master branch rather than a formal release. Third, developers often make small, silent in-tree edits—such as changing an #include path or patching a compiler warning—without recording the diff.

Comparing file by file against tags leaves you with hundreds of partial diffs and no clear parent commit. To perform a rigorous git audit across your supply chain, you must isolate what was upstream from what was modified locally.

Isolating Upstream Trees with Git Index Primitives

Git tracks snapshots of directory states as tree objects. If the vendored code was copied without local modifications (even if subdirectories were deleted), the remaining files still share SHA-1/SHA-256 blob identifiers with upstream objects.

To locate the exact upstream commit:

1. Clone the canonical upstream repository into a separate workspace.
2. In your local repository, generate a synthetic git tree object containing only the vendored directory, re-rooted to the repository base.
3. Compare the subtree hash against every tree object in the upstream history.

You can isolate the tree hash of your local vendored directory directly from your current branch:

VENDOR_TREE=$(git rev-parse HEAD:third_party/zstd)

In the upstream clone, you then search for a commit that matches this tree hash exactly:

git log --all --format="%H" | while read -r COMMIT; do
  UPSTREAM_TREE=$(git rev-parse "$COMMIT^{tree}")
  if [ "$VENDOR_TREE" = "$UPSTREAM_TREE" ]; then
    echo "Exact match found: $COMMIT"
    break
  fi
done

If the engineer copied the entire repository root without altering a single byte, this search terminates in seconds with the exact upstream commit SHA.

Handling Pruned Trees and Local Patches

If files were deleted or modified during the import, an exact tree match will return nothing. You must drop down to blob intersection and patch matching.

When files are pruned, you find the commit that maximises shared blob hashes. You can extract the list of blobs in your vendored tree and score upstream commits by how many matching blobs they contain:

git ls-tree -r "$VENDOR_TREE" | awk '{print $3, $4}' | sort > local_blobs.txt

git log --all --format="%H" | while read -r COMMIT; do
  MATCHES=$(git ls-tree -r "$COMMIT" | awk '{print $3, $4}' | sort | comm -12 local_blobs.txt - | wc -l)
  echo "$MATCHES $COMMIT"
done | sort -nr | head -n 5

The commit with the highest score is your candidate base.

Once you have the candidate base, run a diff between that upstream commit and your vendored directory, filtering out deleted files:

git diff "$CANDIDATE_COMMIT" "$VENDOR_TREE" --no-renames --diff-filter=M

If this diff is empty, your local code is an exact, pruned snapshot of that candidate commit. If the diff produces output, you have just isolated the local in-tree patches that your team applied on top of upstream.

Sealing the Lineage

Once you extract the base commit SHA and any local modifications, do not leave them in someone's notes. Formalise the lineage:

1. Document the upstream remote and full 40-character commit SHA in a vendor manifest file next to the code (e.g., third_party/zstd/VENDOR_METADATA.json).
2. If local patches exist, extract them into explicit .patch files stored alongside the manifest, or submit them upstream.
3. Check the exact license file associated with that candidate commit SHA to ensure no license transition occurred between the release you assumed you were using and the commit actually pulled.

Treating unversioned directories as a verifiable series of base-commit-plus-patches establishes an immutable root of trust for every external line in your repository. You can verify whether the upstream invariants match your legal requirements by validating upstream license invariants in vendored tarballs before checking them back in.

What to Do on Monday

Run a quick inventory across your codebase for any directory inside vendor/, third_party/, or lib/ that lacks a submodule link or package manager lockfile entry.

Pick the oldest unversioned directory. Run the blob-matching loop above against its upstream repository. Find the exact parent commit, diff the local modifications, and write the base SHA into an explicit vendor manifest.

Stop treating vendored code as a black box of inherited risk. If it runs in your build, you need to know the commit that built it.