Academic reviewers sometimes deliver the feedback that a scientific paper can be published, but with major revisions. Credit: NuPenDekDee/GettyScientists aren’t usually thrilled when reviewers of their research papers ask for extensive revisions ahead of publication. But the stress that authors experience might pay off in the long run. An analysis of publicly available peer reviews for thousands of papers finds that papers receiving tough reviews go on to have a higher impact in science than those that sail through the review process.The study, posted to the arXiv preprint server last month1, evaluated the peer-review correspondence associated with a selection of papers published in Nature Communications. The journal has been making these files public for papers that the journal accepts since 2016, as long as the papers’ authors give their consent. It does this for transparency and to inform discussion of published papers in the research community, the journal says.Co-author of the preprint, An Zeng, who is a specialist in complexity science at Beijing Normal University in China, says that he and his colleagues “thought these files could tell us a lot about the ‘negotiation’ between reviewers and authors” to get papers published.No pain, no gainTo draw insights from the review correspondence, the team prompted a large language model (LLM) to evaluate these files for 8,000 published papers — 1,000 randomly selected per year from the period 2017 to 2024.The peer-review crisis: how to fix an overloaded systemThe authors asked the LLM — a version of Claude, which was created by artificial-intelligence company Anthropic, in San Francisco, California — to score reviewers’ comments for how constructive they were and for how strong their opinions were. For example, a comment that the main conclusion of the paper “is not supported by the evidence” would score highly in opinion strength, Zeng says. But it would not score highly as a constructive comment without further suggestions for how the authors could address the problem. The model also assessed the ‘quality’ of comments, which was a measure of how specific and well-reasoned they are. To evaluate the response of the authors to reviews, the LLM assessed the ‘cost’ of revisions — defined as how much work the authors did to address the comments. The model also measured whether the authors accepted or rebutted reviewers’ criticisms and whether those issues were resolved in later review rounds.By considering how many times a paper was cited in the literature during the three years after its publication, the researchers found that papers eliciting stronger criticism from reviewers and requiring more-extensive revisions received more citations than did papers that drew light comments and sailed through the peer-review process. The quality of reviewer comments was also associated with high citation counts, whereas how constructive comments were did not seem to affect a paper’s citations.The findings are “consistent with the idea that more ambitious or consequential work may be questioned more intensely”, Zeng says. But they could also suggest that a demanding review process helps authors to improve their papers before publication, boosting the likelihood of having a high-impact paper, he adds.