Graph ML’s Benchmark Problem, Quantified: 885 Papers, 27,723 Results and a Living Leaderboard

Graph machine learning has a benchmarking problem, and a new paper puts a number on it. The authors turn 885 peer-reviewed papers into a searchable record of 27,723 results across 111 datasets, then use that record to show how often reported gains rest on comparisons that were already out of date when the paper was written. They call the artifact a living leaderboard for graph learning, and the audit built on top of it is the more interesting half.

The benchmarking problem, in graph ML terms

Graph learning has no universally accepted modern benchmark set and no defined tuning process. Results are scattered across papers that use different dataset subsets, baselines that were tuned lightly or not at all, and comparisons against numbers that have since been improved. The paper points to a recent position paper warning that weak benchmarks will hinder progress in the field, and it treats that warning as something to test rather than repeat.

There has been progress: OGB, LRGB and GraphBench each pushed the field toward more standardized evaluation, and GraphBench in particular includes guidance on how baselines should be tuned. But none of these gives a complete picture of what has actually been reported, on which datasets, with which baselines. That gap is what the authors set out to close.

What the paper introduces

Two contributions. The first is a graph-specific leaderboard, organized by dataset, that places any new result in the context of everything previously reported on that dataset. The second is a quantitative analysis of benchmarking practice at scale, built from a corpus assembled by recursively following incoming and outgoing citation links from a set of seed papers.

That corpus is not a keyword search. It is a citation crawl, which matters for coverage claims: the authors are explicit that the pipeline only sees what it can reach through links and through available sources.

How the LLM extraction pipeline works

The pipeline runs in four stages. Candidate screening looks only at title and abstract, asking a single question: is this paper about graph learning? For papers that pass, the pipeline downloads the source and identifies table environments. Dataset discovery then sends the introduction, the experiments section, table captions, and the first row and column of each table to a language model, which works out which table holds results for which dataset. Table selection narrows to the relevant tables, and result extraction runs once per detected dataset.

The model doing the work is Gemma4-26B-A4B at 4-bit precision. The first two stages are single calls; the final extraction is one call per dataset. Extracted entries include standard deviations, and ablation rows are skipped. A row only reaches the leaderboard when the extracted task matches the dataset’s task, and custom, out-of-distribution, few-shot, continual and attack settings are excluded entirely. Metadata such as codebase URLs and method type (message-passing GNN, graph transformer, or MLP) is captured for filtering.

On the site, dataset pages carry a state-of-the-art-over-time plot with baselines and a filterable results table. Paper pages show that paper’s results alongside GCN and MLP baselines, added if the paper omitted them, plus the top three leaderboard entries. Simple MPGNN and MLP performance is highlighted, which turns out to be a pointed design choice. The authors plan weekly integration of new papers, arguing this gives far better coverage than manually maintained baselines and lets the field watch models and baselines evolve over time.

What lands on the leaderboard

As of the 15 October 2026 snapshot, the pipeline had extracted 27,723 eligible results across 111 datasets from 885 peer-reviewed papers. That snapshot is the basis for everything below, including the discussion of 2025 conference papers and the two-year trend in architectures. The scale is what makes the findings statistically legible: individual anecdotes about weak baselines are common, but a corpus this size can put a percentage on the practice.

Finding 1: stale baselines are the norm

Two thirds of analyzed papers, 67.1%, include at least one result for which a significantly better value for the same method was already recorded in another paper at the time of publication. Not a value discovered later, but one that existed in the literature when the paper went out.

The gaps are not marginal. In 38.5% of cases, the best literature value for a baseline was at least halfway to the state of the art. In practice, halfway means the recorded value closes at least half the gap between the paper’s reported baseline score and the best result on that dataset, so the baseline method is substantially stronger than the paper’s table suggested. At the individual-result level, up to 29.6% of results on Actor fell below their corpus value, the largest such shortfall the authors report across datasets. The abstract singles out two datasets: 45% of Amazon-ratings tables and 37% of Peptides-struct tables contain baseline results drastically lower than values already available.

There is a reassuring signal buried here. Stale-baseline rates are lower for OGB, which maintains a manual leaderboard, and for the relatively recent LRGB benchmark. Where the community has invested in shared infrastructure, the problem shrinks.

Finding 2: MLPs are underrated baselines

On the legacy heterophilic datasets Texas, Cornell, Actor and Wisconsin, graph techniques provide no benefit over plain MLPs. The same holds for datasets with textual node features such as PubMed and Coauthor-Physics and Coauthor-CS, even without LLM embeddings. On 57.3% of papers reporting Cornell results, the best known MLP outperforms every non-MLP method that paper reports on that dataset.

The paper’s own phrasing is blunt: one should first take MLPs seriously as baselines. For anyone who has shipped a graph model on the strength of a two-point improvement over GCN, that sentence is worth sitting with. The leaderboard makes the check cheap, which is the point of building it.

Finding 3: legacy datasets persist, and no architecture dominates

TU datasets are being used less, but legacy heterophilic graphs are not fading. They appear in 42.4% of 2025 conference papers that use at least one of the five plotted benchmark groups, making them the second-most-used group. The authors call this an alarming trend and point to known problems with these datasets: size, test data leakage, and the MLP performance just described. Their recommendation is to move to modern alternatives such as Roman-Empire and Tolokers.

On architecture, the picture is a near-even split. Across datasets with at least five papers, each dataset unit-weighted with ties split evenly, graph transformers lead on 39.1% versus 47.2% for message-passing GNNs. Neither dominates, and the authors note the picture was stable over the last two years. MPGNNs lead on node-level tasks; graph transformers hold a slightly larger share on graph-level tasks. The remainder of the share falls to traditional methods such as SVMs and boosting, which remain strong contenders on several datasets, especially around molecules. That is a useful reminder that the architecture debate is not the whole field.

Limitations and caveats

The authors are candid about what the pipeline cannot see. Only about 80% of NeurIPS, ICLR and ICML papers from 2022 to 2024 are available on arXiv, which limits source-based coverage; a PDF extraction path is in development for papers without source files.

Compatibility is the harder problem. Results using different evaluation protocols or metrics can end up merged into one leaderboard, and the authors concede that judging compatibility is non-trivial even for humans. Current guards include matching on metric, outlier detection, manual exclusion, and rules covering typical cases. Improving this is where they plan to invest next.

The statistics carry their own caveats. Comparisons use a two-sided Welch’s t-test at the 5% level when both results report a standard deviation; when only one does, the other is treated as fixed and compared against the 95% confidence interval for the mean; when neither does, unequal values are treated as significantly different. There is no correction for multiple comparisons, and when repetition counts are unavailable, four runs are assumed. These are reasonable defaults for an audit at this scale, but they are defaults.

What the leaderboard changes for engineers and reviewers

The practical value is a fast check. Before trusting a claimed gain, look up the dataset: does the number beat a published baseline, or a plain MLP? The leaderboard answers that in seconds, and the audit suggests the answer will sometimes be no. Reviewers get a centralized record of prior results and a way to track how suggested baselines develop over time. The authors close by arguing that extended baseline tuning is necessary and that the community should shift toward modern datasets.

The leaderboard is live at graphlearningbenchmarks.github.io, the code is on GitHub, and the paper is A Living Leaderboard for Graph Machine Learning.