How scientists caught AI writing their colleagues’ papers

A massive new study scanned more than one million scientific papers for signs of AI and the results were overwhelming: AI is everywhere.

The study, published this week in Nature Human Behaviour, analyzed preprints and papers published from 2020 to 2024 by searching for some of the signature traces that AI-generated tools leave behind.

While some authors deceptively using AI might slip up and leave obvious clues in the text – chunks of a prompt or conspicuous phrases like “regenerate response,” for instance – they have become savvier and more subtle over time.

In the study, the researchers created a statistical model of word frequency using snippets of abstracts and introductions written before the advent of ChatGPT and then fed them through a large language model. By comparing the texts, the researchers found hidden patterns in the AI-written text to look for, including a high frequency of specific words like “pivotal,” “intricate,” and “showcase,” which aren’t common in human-authored science writing.

Some research areas rely on AI, others don’t

The authors found widespread use of large language models across research topics, but some fields appeared to rely on AI much more heavily than others. In computer science abstracts, an estimated 22.5% of sentences showed evidence of AI usage, compared to 9.8% for physics papers and 7.8% for mathematics.

“We see the biggest increases in the areas that are actually closest to AI,” co-author and Stanford computational biologist James Zou said. Shorter papers and papers in crowded fields of research showed more signs that they were partially written by AI.

Many academic journals have added rules requiring authors to disclose the use of generative AI tools like ChatGPT. Prominent journals including Science and Nature explicitly state that AI cannot author a paper, but AI can be used for more minor processes like copyediting and grammar. For some journals, the AI prompts used in crafting a paper must be detailed in depth in a paper’s methods section.

If the threat of introducing hallucinations into a paper isn’t enough of a deterrent, authors trying to sneakily use AI run the risk of getting called out by running databases dedicated to documenting AI transgressions in research.

“Machines play an important role, but as tools for the people posing the hypotheses, designing the experiments, and making sense of the results,” Science Editor-in-Chief wrote in a letter on AI use in research. “Ultimately the product must come from—and be expressed by—the wonderful computer in our heads.”