GitHub has released ReviewBench for evaluating AI code review

GitHub has released ReviewBench, a benchmark for evaluating the quality of AI systems for code review. In its research preview phase, it allows users to explore a public dataset, compare agents and submit their own system for evaluation. The dataset contains 219 pull requests from 187 public repositories with open-source licenses and covers 19 programming languages.

According to GitHub, the selection is based on an analysis of 103.9 million pull requests. The distribution of languages and repository sizes is intended to reflect overall activity on the GitHub platform. In terms of change size, however, the benchmark deliberately limits the predominance of small edits and gives more weight to larger changes spanning multiple files. Results can be filtered by the severity and category of findings, and the balance between detecting issues and limiting false positives can be adjusted.

According to GitHub, an internal audit by senior engineers who were not involved in creating the dataset agreed with the original evaluation in 96.6 % of cases. Versions of both the dataset and components of the evaluation procedure are tracked to enable repeatable comparisons. The company says that, when developing the Copilot code review feature, the benchmark helps it track improvements, detect declines in quality and predict the direction of results from experiments in production; this reflects the experience of the company with its own product.