Is AI review better than human review?

When I tweeted my concerns with AI review, many responded that AI reviews are better than the average human review. I was surprised as the AI-generated reviews I have seen are clearly worse. That disconnect motivated this post.

I raised my concerns in the context of standard review cycles: reviewers write their reports either from their own thoughts or through AI prompts; authors are required to provide point-by-point responses to every comment, regardless of whether it was written by a human or an AI. This is what matters to most of human authors like me. I am less concerned with using AI-generated reviews as optional secondary opinions when authors are not forced to rebut.

I started by reading a few papers referenced in responses to my tweet. King & Worthington (2026) used ChatGPT to judge human vs Opus review. This approach has been criticized by Baumann et al (2026) as AIs think alike. King and Worthington also asked former editors to choose their preferred reviews, but without reading through papers as closely as their authors or dedicated peer reviewers, the editors wouldn’t truly evaluate the reviews’ scientific depth. In another study, Brian et al (2026) injected deliberate errors to clinical trials and checked who could find more of them. I agree error-spotting is a strength of AI but this setup has little in common with typical paper review in life sciences. Meanwhile, Thakkar et al (2026) found providing AI feedback on human review is helpful. This work is not about autonomous AI review.

I then asked Claude and Gemini to collect more papers evaluating AI-generated review. They found around 20 papers or preprints. Like King & Worthington (2026), many use AI to judge reviews. Only a handful involve paper authors or peer reviewers as judges. These include Lemberger et al (2026), who found authors rate human review better except on “clearness” and “politeness” (Fig. 2). I like Biswas et al (2026) for its scale: the authors deployed a multi-stage review system, rather than a single prompt, to review 23k conference papers submitted to AAAI-26 and surveyed the authors. They found authors preferred AI review across more categories (Fig. 2) but ranked “weak big-picture judgement”, “nitpicking” and “excessive verbosity” as the top three weakness of AI review (Fig. 3), consistent with my previous post. Importantly, the AAAI-26 authors were not required to provide point-by-point rebuttals to AI-generated reviews. In the end, no study evaluated how AI reviews affect standard review cycles in a realistic setting.

One might argue that if we let authors judge, they will simply choose the review asking them to do less. I agree this is a potential confounding factor but a lengthy review without hitting the core scientific issues is far worse. Furthermore, peer review is also a social process. Human emotions matter as much as the science. I have witnessed AI generating 30+ bullet points on a paper I reviewed. It is painful to imagine the authors’ despair as they read and are forced to respond point by point. And this will happen to me someday.

Is AI review better than human review? Generally “no” when you ask the authors or the reviewers. The answer would likely be more negative if the authors were required to provide point-by-point responses. My bottom line remains the same: human expert review without the influence of AI is essential to scientific publishing.

Afterword

While reading papers and the lengthy AI summary of a lot more papers, I realized that fully AI-generated reviews are already so pervasive that journal bans won’t stop them. AI detection reports are not smoking guns. Even if watermarked detection could reach 100% specificity, a reviewer could easily claim: “I didn’t feed the paper to AI; I only used AI to polish my own draft”. Journals can’t hold reviewers accountable. Nonetheless, I believe editors should be given the power to discard a review if they detect or strongly suspect it was fully generated by AI. Sparing human authors from having to rebut low-effort AI output would be a massive relief. Additionally, journals may consider deploying automatic AI feedbacks on human-submitted reviews as suggested by Thakkar et al (2026) who showed such feedback helped human reviewers. In fact, an automatic AI feedback like “this review appears to be generated by AI” would discourage most human reviewers from submitting raw AI output.