Artifact Reproducibility Triage
Verifying Research Code Against Its README
Captured from the live deployment, not a mockup
01 · The problem
A model asked whether a repository contains what its README promises will simply agree, so the pipeline runs deterministic filesystem checks first and allows exactly one model call afterwards.
02 · How it works
- 01
Moved detection of fabricated file claims from 0% to 100%, measured across 742 real research artifacts, by refusing to let the model assert a file exists without a filesystem check behind it.
- 02
Built a verifier for whether a paper's repository actually contains what its README claims, running deterministic checks ahead of a single model call so the model never asserts an unverified fact.
03 · What it cost, and what it returned
Measured fabricated-claim detection moving from 0% to 100% across 742 research artifacts.