PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers

1. Introducing PRISM

Major ML venues now receive tens of thousands of submissions, which makes reviewer assignment, workload, and review quality increasingly difficult to manage. LLM reviewers offer scale, but common evaluation methods often rely on surface similarity metrics or broad LLM-as-a-judge scores. These approaches can blur together fluency, factuality, and scientific rigor.

PRISM asks a stricter question:

Does a review provide grounded analysis, calibrated novelty judgment, valid flaw detection, and actionable feedback?

To answer it, PRISM evaluates each manuscript-review pair through four independent and interpretable pipelines. Each pipeline extracts small review units, verifies them against the manuscript or prior literature, and computes metrics from those structured decisions instead of relying on a single opaque judge rating.

PRISM uses constrained LLM judging for extraction and labeling. The final scores, however, are computed analytically from labels, retrieval evidence, consensus verification, and the structure of review comments.

2. Methods

3. Results and Analysis

PRISM shows that LLM reviewers tend to specialize in different review responsibilities. No single system dominates all four dimensions.

Headline Results

Key Takeaways by Dimension

Depth of Analysis: Humans lead (DoA = 0.494) through premise density and methodology focus. DeepReview and CycleReviewer nearly match by generating many grounded premises, while TreeReview falls into surface-level reviewing patterns.

Novelty Assessment: All systems operate within 0.750–0.830 for evidence-grounding. SEA achieves the highest score (0.833), showing that retrieval-oriented pipelines help verify literature claims.

Flaw Detection: Reviewer2 is an exhaustive flaw scanner with Critical Recall of 0.591 (vs. human 0.343). LLMs can surface more candidate issues, but human oversight is needed to manage precision.

Constructiveness: DeepReview significantly outperforms others (MCS = 0.634 vs. human 0.566). Human reviewers are specific but often stop at diagnosis; DeepReview closes the loop from “this is a problem” to “here is how to fix it.”

Core Insight

Human and LLM reviewers are complementary. Humans excel at balanced judgment and calibration, while LLMs excel at exhaustive scanning and systematic verification. Their union covers more diagnostic ground than either group alone.

4. Conclusion

Since no single system dominates all four dimensions, the evidence points toward targeted ensemble deployment:

• Use Reviewer2 for exhaustive flaw scanning, as it catches critical issues that human reviewers may miss.
• Use DeepReview for constructive feedback drafting, as it provides actionable and solution-oriented suggestions.
• Use SEA for novelty-grounding checks, as it verifies claims against prior literature effectively.
• Use human reviewers for final judgment, as they provide the most balanced and cognitively aligned assessment.

These systems are most effective as specialist assistants within a human-led pipeline rather than autonomous reviewers.

Related Systems

PRISM belongs to public benchmarks and reviewer-assistance projects that emphasize inspectable evaluation. Related systems include Reviewer2, SEA, DeepReview, TreeReview, and CycleReviewer.

BibTeX

@article{prism2026,
  title={PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers},
  author={Ngoc Phan, Toan Huynh, Tran Khanh Thanh, Duy A. Nguyen, Nguyen Pham Tuan Anh, Thanh Nguyen, Nitesh V. Chawla, Wray Buntine, Kok-Seng Wong, Khoa D Doan, Binh Nguyen},
  journal={arXiv preprint},
  year={2026}
}