When Your arXiv Query Returns Nothing: A Reproducibility Checklist for Paper Discovery

Every engineer who has built a literature-scanning pipeline has a version of this story. You fire off a request for recent papers, wait for the page to render, and get back something that looks like a result page but contains no results. No titles. No authors. No abstracts. Just a heading, a few lines of interface text, and the quiet suspicion that you have wasted a morning.

The instinct in that moment is dangerous. An empty response feels like a gap in the literature, and the temptation is to fill it, either by guessing at what should be there or by asking a language model to summarize “the relevant work.” Both moves break the one property that makes a literature scan worth doing: reproducibility. An empty arXiv response is a data point about your retrieval pipeline, not a statement about the state of machine learning research. Treat it as a bug to be caught. Your retrieval should be boring, verifiable and loud when it breaks.

This checklist walks through the failure mode, the fix and the record-keeping that makes a scan reproducible. A captured page from a real run serves as the worked example.

The failure mode: a page that looks like a result

The captured page in question was the product of a specific, reasonable-looking request: a query string of search_query=cat:cs.LG&id_list=&start=0&max_results=50, asking for 50 papers in the cs.LG (Machine Learning) category beginning at result zero. What came back was a page heading reading “Computer Science > Computation and Language,” the label for the cs.CL category, followed by boilerplate describing arXivLabs: a framework letting collaborators build and share features on arXiv’s site, with partners accepting arXiv’s values of openness, community, excellence and user data privacy, plus a call for project ideas and a link to learn more.

That is the entire inventory. No paper titles, no authors, no abstracts, no IDs, no dates, no results list at all. Either the query returned nothing rendered, or only the page chrome was captured before the entries loaded. This is not a subtle failure. It is a complete one, dressed up in the visual language of a successful page load.

Why an empty page is not a finding

There is a real difference between “the literature contains nothing on this” and “my request did not return the literature.” A genuine absence is rare and interesting. A failed retrieval is common and boring. Confusing the two is how pipelines start producing fiction.

A category listing page on arXiv is, in principle, a query result, and when it contains no papers the correct reading is that the page failed to deliver, not that the subfield went quiet. The same logic applies here. If you summarize an empty page as “no relevant recent work,” you have made a claim about the literature that your evidence does not support. If you go further and let a model invent plausible-sounding citations to fill the space, you have manufactured evidence. Neither survives a second run of the same query, which is exactly the test reproducibility demands.

For background on how arXiv structures its category listing pages, see our earlier piece What arXiv’s stat.ML Listing Page Actually Tells Us – And What It Doesn’t.

Query versus category mismatch

The most informative detail in the captured page is the disagreement between the request and the response. The query asked for cs.LG. The heading announced cs.CL. Those are different categories, and the mismatch tells you something about how result pages are assembled: the heading is often page furniture, rendered from a template or a default category context, while the entries themselves come from the query. When the entries fail to load, the furniture remains and the substance vanishes, leaving a page that misdescribes itself.

This is why the request in question seemed so reasonable in the first place. Asking arXiv for up to 50 recent papers in a single category, whether cs.LG or cs.CL, is an entirely sensible thing to do. cs.CL, the Computation and Language category, covers areas such as language models, tokenizers, evaluation harnesses and retrieval systems. The problem was never the ambition. It was that the response could not be trusted to reflect the request, and nothing in the pipeline checked.

The arXivLabs boilerplate deserves a note of its own, because it is the part of the page most likely to be mistaken for content. As explained in Inside arXivLabs: How arXiv Opens Its Platform to Outside Builders – and What It Means for Machine Learning Research, arXivLabs is a framework that lets collaborators develop and share new arXiv features directly on the arXiv website. It is not an external integration, a partner API or a separate product bolted onto arXiv from the outside. It is also not a research finding. Treating its presence as a signal about the literature is a category error of the most literal kind.

Switch to the API

The fix for rendered-page fragility is to stop parsing pages. arXiv exposes a programmatic interface at export.arxiv.org/api/query, which accepts explicit parameters: search_query for the category or field query, start for the offset and max_results for the count. A well-formed request for 50 machine learning papers looks like https://export.arxiv.org/api/query?search_query=cat:cs.LG&start=0&max_results=50.

What comes back is Atom XML, a structured feed with entries you can parse deterministically. Each entry carries an ID, a title, authors, a publication and update date, and an abstract. There is no page chrome to confuse with results, no template heading to contradict your query, and no JavaScript rendering step between the request and the data. The response is also archivable: you can store the raw XML alongside your analysis, which is the foundation of everything that follows.

Three details matter in practice. First, the feed includes an opensearch:totalResults field reporting how many records match the query in total, which lets you distinguish “zero results” from “the parser found nothing in a non-empty response.” Second, arXiv can legitimately return fewer entries than max_results asks for, so never assert equality between the two. Third, the API is a shared public service: arXiv’s terms ask callers to space out requests rather than hammering the endpoint in tight loops, and to identify their client. Build a delay between calls and cache responses you have already fetched. If you care about recency, the API accepts sortBy (for example submittedDate) and sortOrder (ascending or descending) parameters, so you can ask for the newest papers first rather than filtering dates after the fact.

Validate before you summarize

Structured output makes validation cheap, and validation is the step that turns a silent failure into a loud one. Before any downstream summarization, check four things programmatically.

First, count the entries. If max_results was 50 and the feed contains zero entries, raise an error rather than proceeding. Second, verify the IDs. Every entry should carry an arXiv identifier matching the expected format, and the set should be non-empty. Third, check the dates. If you asked for recent work, confirm that the publication or update timestamps fall within your intended window; a feed full of 2019 papers when you wanted last month is a different kind of retrieval failure. Fourth, confirm the abstracts are present and non-trivial. An entry with an empty summary field is a partial record, and partial records should be logged, not silently summarized.

The principle is simple: any condition that would make your summary misleading should halt the pipeline. An empty response should fail loudly at the validation gate, not quietly at the writing stage.

Reproducibility notes for literature scans

A literature scan is an experiment, and it deserves the same record-keeping as one. For every run, capture the exact query string, the endpoint URL, the timestamp in a fixed format such as 2025-03-05, and the raw response body. Store these alongside any downstream analysis, so that a reader, or a future version of yourself, can re-run the request and compare outputs field by field.

This matters most precisely when things go wrong. If a scan returns nothing, the archived raw response is the evidence that distinguishes a genuine absence from a broken request. Without it, you have a claim and no provenance. With it, you have a reproducible observation about your own tooling, which is often more valuable than the papers you were looking for.

Note that Atom feeds are not stable artifacts: result sets change as new papers are submitted, and the feed may reorder or update entries between runs. Compare structurally, checking which IDs appear in each response, rather than expecting an identical file.

A pre-flight checklist before you write a word

Run this sequence every time, and treat a failure at any step as a stop, not a warning.

1. Confirm the category. Decide whether you want cs.LG, cs.CL or something else, and make sure the search_query parameter says so. Do not let a page heading overrule your own request.
2. Confirm the endpoint. Use export.arxiv.org/api/query with explicit search_query, start and max_results parameters. Do not scrape rendered pages for structured data.
3. Confirm non-zero entries. Parse the Atom XML and assert that the entry count is greater than zero. Do not assert equality with max_results, since arXiv may return fewer results than requested.
4. Confirm the records. Check that IDs, dates and abstracts are present and within your intended window.
5. Archive the raw response. Store the query, endpoint, timestamp and XML together.
6. Then write. Only after the data has passed every gate should summarization begin, and every claim in the summary should trace back to an archived entry.

An empty arXiv response is not a finding about machine learning. It is a finding about your pipeline, and the correct response is to fix the pipeline, log the failure and re-run. The literature is large. Your retrieval should be boring, verifiable and loud when it breaks.