Scientist World Turns 1,000 Papers Into a Training Gym for AI Researchers

Ask an AI agent to improve a machine learning method and it may well produce something that looks like progress. Whether the number on the evaluation actually moves is another matter. That gap between the appearance of research and research itself is what a new paper, Grow a Research World for AI-Scientists, sets out to close.

The authors’ framing is blunt: nailing one problem is a trick, but research is the ability to discover again and again, problem after problem. If that is the target, a fixed benchmark of a few hundred tasks will not do. It measures yesterday, and it slowly turns into training data. So instead of shipping another leaderboard, the team built a world: 1,000 executable scientific environments, each grown from a real paper, scored on whether an agent actually beats the human method.

The problem with a fixed benchmark

Most AI-Scientist evaluations hand an agent a task and check the answer. That works for coding puzzles and math competitions, where the ground truth is stable. Research does not behave that way. The frontier moves, the literature grows, and a method that wins in 2024 may be a footnote by 2026.

The paper argues that AI-Scientists need an evaluation surface as large as the scientific literature to grow in, and that it should keep growing. Scientist World is their attempt: a research environment built directly from published work, where every task is anchored to a real method and a real reported result. It is positioned as three things at once: a benchmark, a dataset of research traces, and a training ground.

The distinction matters for anyone who has watched agent benchmarks saturate. A world that keeps absorbing new literature measures research rather than recall. That is a different claim on an agent’s capabilities, and a harder one to game.

How a paper becomes an environment

The conversion pipeline has no human in the loop, and it runs on four roles that share a workspace. Each paper gets nine iterations, split across three reproducers, three designers, one packager and two verifier passes, with the model family rotating across GPT, Claude, MiniMax and GLM.

The Reproducer reimplements the paper’s method and original evaluation, working from the authors’ code where it was released and from scratch otherwise. What it leaves behind is the source method and the source result, the anchor for everything that follows.

The Designer then writes the problem statement and task rules: which metric to improve, which settings are fixed, what solutions are allowed. It sets the reproduced source result as the bar, and it prepares two evaluation sets, one visible and one held out, so that shortcuts and overfitting show up as a gap between them.

The Packager strips out local files, replaces them with download scripts, and declares dependencies, so that a fresh clone can rebuild and rerun the environment on its own.

The Verifier audits the result as an independent agent. The paper’s phrasing is worth keeping: the builder that grades the work never built it.

A ladder of milestones tracks progress, and any agent can audit and demote the previous milestone claim, rolling work back. Compute sits in a shared pool, builders submit jobs and are woken with logs, and each paper is capped at 70 GPU-hours enforced outside the builder. Throughout, a GPU-hour means one GPU occupied for one hour from that shared pool, and a compute job is a single submitted task in it, so 76 jobs on a median environment is a batch of short runs rather than 76 hours of wall-clock work.

The design principles are as interesting as the mechanics. Experiments can be shrunk in scale without changing the scientific question. Tasks must stay non-trivial, meaning the source method has to leave a meaningful gap over simple baselines. And the reproduced method is treated as one solution rather than the solution, with the data interface and evaluator written independently of it.

Scoring with Frontier-Normalized Improvement

The scoring metric is where the paper earns its keep. Frontier-Normalized Improvement (FNI) measures how much an agent’s method improves on the human researchers’ reproduced method. Zero FNI means you matched it. Positive FNI means you beat it, and the value reads as a percentage improvement over the human baseline, so an FNI of 41 is a 41% gain and 106 is roughly double the human result.

That single choice reframes the whole exercise. An agent cannot score by writing a confident report, by describing a plausible approach, or by reproducing what the paper already did. It has to run experiments and produce a number that clears the human bar. Recall of the literature earns nothing; only executed improvement counts.

From 124,816 papers to 1,000 environments

The corpus did not start small. The authors pulled arXiv’s core AI categories over fourteen months and ended up with 124,816 unique papers. Two fast screens using MiniMax M2.7 did the first cut: a quality screen keeping roughly the top 25% band, and a type screen keeping empirical-method papers. That left 21,354.

Detailed judgments on research quality and reproduction difficulty followed, combined into a geometric-mean utility over quality and feasibility. A category-balanced top 1,000 became the first release.

The corpus keeps growing. The source categories add roughly 9,000 unique papers per month, of which about 1,500 survive screening and remain feasible. That is the refresh mechanism: the benchmark updates itself as the literature does.

The cost of building a research world

Turning a paper into a runnable environment is not cheap, and the paper is unusually candid about the bill. The median environment takes 0.4 GPU-hours and 76 compute jobs. Across the 1,000-paper release, that adds up to 239,506 compute jobs, 2,286 GPU-hours, 28,338 CPU-hours, 947,056 agent turns, 51 billion input tokens and 231 million output tokens.

The dataset side carries its own weight. Each stored attempt is a version-controlled repository preserving the full development history, averaging 3.7 idea revisions, 6 method revisions, 25 experiments, 35 commits and 5.4 hours of work. The corpus records roughly 6,400 hours of research in total.

The result is not a spreadsheet of scores but a library of attempts, including the ones that failed.

Benchmark results

Evaluation runs on a fixed cohort of ten lower-compute environments, with every model attempting every environment under the same compute budget. There are two modes. In improve, the agent is given the reproduced source method. In innovate, it gets no source code and must design from scratch, with no leakage of the original solution. Scores are averaged over visible and held-out splits.

AI-Scientists beat the paper’s reproduced method in 84% of improve attempts and 74% of innovate attempts. Fable 5 is strongest at improving. Fable 5.1 and Fable 5 take the top two places at innovating, with GLM 5.2 third among open models.

The two modes turn out to be related but distinct abilities. Ranks across the improve and innovate leaderboards are only loosely correlated, which suggests that improving an existing method and inventing a new one draw on different skills. Anyone building an agent stack around a single “research capability” number should take note.

Effort is not the explanation

The most useful negative result in the paper concerns effort. Compute hours and agent turns explain almost none of the score variance, with R² at 0.00 for compute hours and 0.02 for turns. Spending more time and more tokens does not buy a better score.

The authors put it plainly: the models that win are the ones doing better research, not the ones trying hardest. For teams tempted to scale test-time compute as a substitute for method quality, that is a direct challenge.

A second check cuts the other way. An LLM judge, blind to the actual scores, rated attempts on creativity, engineering, rigor, significance and depth. Those ratings tracked the measured scores at ρ = 0.89. Qualitative judgment and execution agree closely here, which is reassuring for anyone who has to triage agent output before running it.

Training a small scientist

Scientist World also serves as a training set. The authors took Qwen3-8B and trained it with Tinker in two stages: supervised fine-tuning on 1,000 examples drawn from the trajectories, cleaned so that no fabricated result survives, then off-policy GRPO on 1,500 already-scored ideas, rewarded by measured outcomes with updates clipped to a trust region around the SFT policy.

To isolate ideation from execution, they split the roles. The trained scientist proposes ideas; a single fixed frontier implementer, GPT 5.6 Sol, executes every idea under identical conditions on ten held-out papers, prompted to follow the ideator’s idea exactly. Those ten held-out papers are a separate set from the ten lower-compute environments used for the benchmark evaluation, so the two figures should not be read as the same cohort.

Median implemented-idea FNI rose from 41 for the base scientist to 84 after SFT and 106 after GRPO, with the mean climbing from 51 to 90. Ideas from the trained scientist, faithfully implemented, reach the anchor of the frontier cohort. An 8B model, in other words, can learn to propose research directions that hold up when someone else does the work.

That separation is not just an experimental convenience. It reflects a real division of labor between proposing a direction and carrying it through.

Limitations and open questions

The authors are careful about what the result does not show. A model as small as Qwen3-8B cannot yet carry the whole loop: implementing and debugging a method over a long horizon is frontier work, which is precisely why ideation was separated from implementation in training.

Research is also contingent. A creative idea can be implemented well and still fail empirically, and the response is aggregation across many environments rather than elimination of outcome variance. The benchmark cohort is ten environments because running every model on all 1,000 is too costly to repeat, and each environment is a scaled-down paper whose score represents one trajectory. The full propose, execute and update loop, the conclusion concedes, is just beginning.

Why engineers should care

Three reusable ideas remain once the framing is set aside. First, a standing benchmark judged by execution rather than rhetoric, refreshed as new literature appears. Second, 1,000 runnable environments spanning multiple research areas, each closer to a small research project than a single script, with visible and held-out splits and baselines for comparison. Third, roughly 6,400 hours of released research traces with real experimental outcomes as rewards.

For anyone building evaluation harnesses, the FNI formulation is the transferable piece: normalize against a reproduced human result and let the agent’s own experiments decide the score. For anyone training small models, the two-stage recipe and the ideation-implementation split offer a template that separates proposing a direction from executing it, which is also how the paper gets a usable signal out of an 8B model.

None of it replaces the fundamentals. The benchmark simply gives those skills a place to be measured, one executed experiment at a time.

Related: reprobridge turns reproduction reusable.