GitHub Releases ReviewBench to Compare What AI Code Reviewers Catch and Miss

Teams comparing AI code reviewers can now adjust a public leaderboard to favor broader issue coverage or fewer false alarms. GitHub released ReviewBench on October 5, 2026, as a research preview, with common test cases, scoring rules and tools for evaluating an agent. The benchmark also gives reviewers credit for valid problems missing from its reference answer set.

Teams can filter results by severity and issue category, then adjust the balance between precision and recall. Precision measures the share of surfaced issues that are valid; recall measures the share of known valid issues found. F1 balances both equally. The leaderboard reorders to reflect those preferences.

• Grounded scores use only existing reference labels, keeping the answer set fixed for comparisons between reviewers.
• Augmented scores also judge unmatched findings, so valid new discoveries can earn credit rather than automatically count against an agent.

There is a comparison limit: augmented recall changes its denominator according to each agent's discoveries. GitHub therefore uses grounded recall as the headline cross-system measure and augmented results as diagnostics for individual systems.

ReviewBench contains 219 pull requests—proposed code changes—from 187 public, open-source-licensed repositories across 19 languages. GitHub says it analyzed 103.9 million pull requests to model the workload, aligning the benchmark's language and repository-size distributions with GitHub overall.

The sample deliberately gives more weight to substantive, multi-file changes, reducing the dominance of tiny, single-file edits. Its reference findings come from human reviews, author follow-up commits, deterministic analysis tools and multiple model families.

Overlapping findings are merged before validation. A finding must be true, relevant and non-trivial to count; GitHub uses Claude Sonnet 5 to judge it under a published rubric.

96.6%Engineer agreement

GitHub says senior engineers who did not build the dataset independently relabeled every ground-truth finding in an internal audit, agreeing with its true-or-false-positive judgments 96.6% of the time.

GitHub says ReviewBench's offline trends consistently matched the direction of later Copilot code-review production experiments. It versions the dataset, judge and finding-matching system, and publishes its validation methodology and known threats to validity.

The public research preview includes the complete dataset, evaluation methodology, judge prompt and configuration, plus a self-serve runner. Developers can inspect the test cases, reproduce results and evaluate their own agents under the same benchmark configuration.