AI-Generated Fake Papers Are Testing Scientific Publishing’s Defenses — And the Arms Race Is Just Beginning

The Trust Problem at the Heart of Science

Goldstein’s editorial in the issue, under the heading “Guarding the Gates of Knowledge,” treats AI-generated fakery as a problem of epistemic infrastructure rather than a niche concern for publishers. Scientific publishing works because readers extend what economists would call a credit relationship to strangers. A reader in one field trusts a result from another because the review system vouched for it. Break that chain and the cost is not measured in retractions. It is measured in the collapse of the assumption that a cited result means anything at all.

That framing matters because the usual coverage of AI slop focuses on volume. The more consequential question is plausibility. A flood of obviously broken manuscripts is an annoyance for editors. A slow trickle of convincing ones is an existential problem, because it corrupts the record without announcing itself.

The Experiment: How an Applied Math PhD Used AI to Fake a Paper

To test the gates directly, IEEE Spectrum turned to freelance writer and applied mathematics PhD Benjamin Skuse, whose feature “I Used AI to Fake a Scientific Paper” documents an attempt to push his own AI-generated work past IEEE’s defenses. Skuse had no intention of committing research fraud; he wanted to know how hard it would be. As covered in this analysis of Skuse’s free-LLM experiment, the answer unsettled him.

Skuse was surprised at how effortless it was to produce a paper that was, in his phrase, “more or less plausible and more or less science.” Features and AI beat editor Eliza Strickland, who worked on the project, described the workflow with visible unease: “He basically hit ‘run experiment’ and ‘write paper.’ And then he didn’t do a damn thing to it. He just launched it into the maw of the publishing engine.”

That detail is the crux. The fake paper was not a masterpiece of deception. It was the output of a straightforward pipeline, generated without hand-tuning, and it still cleared the bar of surface plausibility. The gap between “obviously fake” and “worth a reviewer’s time” has narrowed to the point where a motivated amateur can cross it in an afternoon.

The Finding: Human Review Plus Detection Tools Are Holding, For Now

Here is the genuinely reassuring part of the experiment: the paper did not sail through. IEEE’s human-review processes, combined with research-integrity checker programs, are currently keeping pace with this kind of submission.

The qualifier is doing enormous work in that sentence. The defenses held against someone using free tools. More advanced large language models no longer make the same obvious mistakes as their free counterparts, which means the detection signals that integrity checkers rely on, from telltale phrasing to hallucinated citations to statistical oddities in generated text, are getting quieter with each model generation. Publishers are not standing still, but neither are the tools. What the experiment established is a snapshot of a moving front line, not a settled outcome.

Goldstein’s own experience illustrates the asymmetry. He still finds the outputs of current chatbots useful, yet he does not fully trust them, and that combination, useful but not fully trustworthy, is precisely the condition under which review systems have to operate.

Wikipedia’s Parallel Battle: Volunteers Fighting AI Slop With Fewer People and Fewer Tools

Commercial publishers have budgets, dedicated integrity staff and commercial detection software. Wikipedia has volunteers, and the comparison is not flattering to the encyclopedia’s odds. In the issue’s Five Questions, Strickland interviewed Ellia Lebleu, a founding member of WikiProject AI Cleanup, the volunteer effort to detect and delete AI-generated material from the site.

Lebleu’s team faces the same adversary as IEEE with a fraction of the resources. “Wikipedia is known as a repository of human-generated information, and they want to hold on to that,” Strickland said. “Who knows how long they’ll be able to do that.” The project is a small, human-scale attempt to preserve a genuinely human-generated corpus, and it is running on volunteer time against adversaries who can generate content at machine speed. That same imbalance, human caretakers against machine-speed producers, runs through the far less glamorous work of taggers reverting edits at two in the morning.

From ‘Hallucination’ to the Loop: How We Got Here and Where It Leads

The term “hallucination” was a useful way to frame the problem as a bug in a system that was otherwise doing something real. Six years later, the same underlying behavior, models producing fluent, well-formed text that is not grounded in fact, has become the raw material for fake manuscripts, fake encyclopedia entries and fake citations.

Strickland’s closing concern in the issue points somewhere stranger still. “Doesn’t it just feel crazy that we’ll have AI writing for other AIs at some point?” she asks. “We’ll just get left out of the loop.” The scenario is not science fiction. Automated summarization of papers, automated review assistance and agentic research pipelines are all being built now, and each one adds a layer where machine-generated text is consumed by another machine rather than a human reader. Once that loop closes, the plausibility of the text matters less than its format, and the human review step that currently catches fake papers becomes the only remaining check.

What the Experiment Suggests

The IEEE exercise leaves a few practical lessons for anyone who reads, reviews or publishes technical work.

• Plausibility is no longer evidence of effort. A well-formatted manuscript with sensible section headings and a coherent abstract tells you almost nothing about whether an experiment was run. Reviewers should weight methodological detail and raw data availability more heavily than prose quality.
• Verification is a skill worth building. Goldstein’s “verify you must” applies at every level: checking citations against primary sources, confirming that cited datasets exist, and treating unusually smooth writing with mild suspicion rather than admiration.
• Detection tools are a moving target, not a solution. Integrity checkers work today against free tools. Do not assume they will work against next year’s models, and do not treat a clean detector result as proof of human authorship.

The arms race is real, and the current defenses held against a determined amateur with free tools. The open question is whether they can hold against a determined amateur with better ones, and whether the humans doing the reviewing can keep pace with machines generating plausible text at scale. Goldstein’s stance, six years into the chatbot era, remains “Trust but verify. And verify you must.”