ReproBridge Turns Reproduction Into a Reusable Artifact — and Lifts Agent Success From 76% to 93%
The reproducibility gap
Reproducing an ML experiment means reconstructing a procedure from information scattered across a paper, its released code, and supporting artifacts. When something is missing, whoever is reproducing has to decide what the authors probably meant. For an autonomous coding agent, that decision is the whole game.
The paper’s central hypothesis is that preserving the procedures and decisions recovered during reproduction can improve computational reproducibility. That reframes the artifact from a static snapshot of code into a living record of what it took to make the code produce the reported numbers.
What a refined reproducible artifact contains
The authors define a standardized “refined reproducible artifact” (RRA) with four components:
- A refined codebase
- Environment setup
- Experiment launchers
- A reproduction guide
The guide is the piece that carries the recovered decisions: settings that were ambiguous, repairs that were needed, and the reasoning behind them. The hypothesis is that later attempts, human or agent, spend less time rediscovering what an earlier attempt already worked out.
Building ReproBridge-Bench
The benchmark was assembled with a documented sampling chain. The authors started from 742 ICML 2026 Spotlight papers, narrowed to 241 with code and non-purely-theoretical content, and randomly sampled 50. Excluding six for artifact or code availability, four for compute requirements, and four for closed-model access left 36 retained test papers.
A task is one own-method measurement in a main table, identified by dataset, metric, reporting statistic, and location. Baselines, stand-alone ablations, figures, and appendix-only results were excluded. The 36 papers yielded 1,463 curated tasks from 502 experiments. Of those, 1,029 tasks (70.3%) had numerically compatible reference results and formed the downstream evaluation set, spanning 387 experiments.
Each task carries five human-verified reference observations. The rationale is simple: a single flaky run should not be able to define what counts as correct. The five observations were produced by five reproducers on separate servers starting from the same pinned raw commit, with separate execution directories and separate coding-agent sessions. Reproducers could use agent assistance, but humans inspected scripts, configurations, edits, and measurement extraction.
How a task is scored
A task succeeds when its result lands inside a fixed compatibility interval and is not invalidated by an independent Auditor. The interval is a Student-t prediction interval with a 1% tolerance floor at a default q of 0.95; in plain terms, it is wide enough to absorb ordinary run-to-run noise but narrow enough that a wrong number still fails.
The Auditor exists to catch results that look right for the wrong reasons. It traces every reported value to its output location, execution record, and generating script using ledger timestamps and file hashes. A Finder proposes violations and a separate Refuter tries to overturn them, with a panel handling contested cases. Five violation classes are eligible: copying a value without measuring it, substituting the measurement instrument, altering an operand, choosing the reported outcome conditional on observed values, and computing the wrong statistic. The Auditor gives no feedback to the Reproducer.
Six agents, three stages
Construction runs through six agent roles across three stages. Specify covers Intake and Planner; Execute covers the Builder and a construction Reproducer; Preserve covers the Distiller and Packager. A Reviewer works in separate contexts. Humans define the workflow, delivery standards, and protocols, while agents perform all construction, review, and packaging.
Two design choices matter for anyone thinking of adopting this. When paper and code conflict, agents follow the paper’s setup and flag the discrepancy rather than silently picking a side. And every measurement must link to an actual output file, with blocked tasks retaining an explicit failure reason. That insistence on traceability is the same discipline that separates useful agent workflows from plausible-looking output, a theme that runs through work like Scientist World Turns 1,000 Papers Into a Training Gym for AI Researchers.
The main results
Opus 5 constructed the refined artifacts. Fresh Reproducer sessions then received the paper, selected tasks, and either raw code or the refined artifact, with construction conversations and hidden reference observations withheld. Sonnet 4.6, Sonnet 5, Opus 5, and Gemini 3.8 Flash attempted all 1,029 tasks across 36 papers, one fresh session per paper-model-condition, under a 24-hour wall clock limit per paper and eight H100-80GB GPUs for GPU experiments.
| Model | Raw success | RRA success | Gain |
|---|---|---|---|
| Sonnet 4.6 | 72.3% | 91.6% | +19.3 pp |
| Sonnet 5 | 76.5% | 93.3% | +16.8 pp |
| Opus 5 | 89.2% | 94.7% | +5.5 pp |
| Gemini 3.8 Flash | 67.6% | 90.8% | +23.2 pp |
Across the four models, refined artifacts raised mean reproduction success from 76.4% to 92.6% and cut downstream API spending by 19.7% on average. Token reductions ranged from 6.3% to 19.8%, and API savings from 16.8% to 23.7%, including failed attempts. Perhaps the most striking number is the narrowing of the cross-model spread, from 21.6 percentage points to 3.9. All four models cleared 90% mean success with refined artifacts. Better handoffs compress the gap between strong and weak agents, because less of the job depends on improvisation.
External check and ablations
On CORE-Bench Hard, across 45 capsules with Sonnet 5 constructing the artifacts, refined artifacts achieved the highest success, cut median duration from 33 to 14 minutes, and lowered spending per successful capsule from $8.43 to $4.34 (USD, as quoted in the source). A comparison against VERITAS, whose patched codebase and fix log sit between raw and refined handoffs, placed the refined artifact at or above raw success in each area.
Ablations removed one component at a time. Every removal reduced success, with the largest losses from removing the code or the environment setup. Removing guides or launchers also hurt even when the other components stayed in place.
What the paper-code contradictions show
An agent-assisted static analysis recorded 238 in-scope contradiction findings touching 669 distinct selected tasks. A contradiction required an explicit statement in the paper plus conflicting implemented behavior; absent implementations and silent choices were logged separately. One cited example is the Manifold-Aware Perturbations guide, which identifies a learning rate of 10⁻⁴ in the paper against 10⁻³ in the released code.
The authors are careful here. They describe these findings as possible sources of execution deviation, note that their effect on measured performance needs controlled comparisons, and state plainly that they are neither human-validated error rates nor explanations of the measured gains. The contradictions were not supplied to the evaluated Reproducers.
Limitations and open questions
The evaluation conditions on retained papers, fixed reference regions, and the tested agents, and the authors stress it is not a conference-wide reproduction rate. Five reference observations are a protocol choice, not a guarantee of adequate uncertainty estimation. Numerical compatibility does not certify scientific validity, and refined artifacts can preserve incorrect assumptions as faithfully as correct ones.
Automated audits miss some violations, quantified against human verification. The comparison evaluates the construction workflow and the resulting handoff together, so it does not isolate the effect of reorganizing identical outputs. API savings cover downstream Reproducer sessions only; construction, independent verification, GPU execution, maintenance, and human review carry separate costs. Human time savings and comprehension need their own study, and human reuse of the artifacts is left to future work.
How the artifacts are meant to be used
The intended first use is upstream: authors running ReproBridge as a reproducibility check before submission, where executing the specified setup exposes missing details, execution barriers, and paper-code inconsistencies while they can still be fixed. Third-party reproducers can preserve their work in the same format to lower the cost of the next attempt. Readers can follow the setup route and launchers to run experiments, then inspect the guide and change record to judge settings and repairs.
That is the paper’s practical claim. A reproduction that ends in a number is a one-off. A reproduction that ends in a guide, a launcher, and a change record is infrastructure that the next attempt, human or agent, can build on.
For more on this, see give agent memory practical.
