Perch on Greptile and ReviewBench
Recorded bug detection on 50 public pull requests
Perch's OpenAI Decisions configuration flagged the function containing the known bug in 38 of 50 pull requests at a 60% cutoff. The saved run used 33.5¢ of model inference across all 50. The Jev configuration flagged 22 of 50 with 14.9¢ of model inference.
| Configuration | Known bugs located | Median flagged functions per PR | Measured model cost, all 50 PRs |
|---|---|---|---|
| Perch · OpenAI Decisions | 38 / 50 (76%) | 7 | 33.5¢ |
| Perch · Jev | 22 / 50 (44%) | 2 | 14.9¢ |
Perch is credited for flagging the function holding a known bug; published competitor results credit a judged review comment identifying it. These scoring units differ. The cost is measured model inference for each named saved run, separate from Perch Cloud billing. Four known bugs sit outside the functions Perch reads. Authenticated CodeRabbit and Greptile reviews for the separate order-service experiment remain unscored.
Pinned scoring scripts and configuration · ReviewBench precision, thresholds and full reproduction steps · Public saved answers and labels
ReviewBench: recall and false alerts together
| Configuration and cutoff | Recall | Precision | Correct alerts | False alerts | Measured model cost |
|---|---|---|---|---|---|
| Perch · OpenAI Decisions, 60% | 40.0% | 28.8% | 603 | 1491 | 415.69¢ |
| Perch · Jev, 60% | 7.9% | 45.8% | 119 | 141 | 179.82¢ |
Recall measures how many labelled buggy methods were flagged; precision measures how many flagged methods were labelled buggy. The OpenAI Decisions rerun at 60% flagged 603 of 1,506 labelled buggy methods, with 1,491 false alerts. At 80%, recall is 23.2% and precision is 41.7%, with 349 correct alerts and 487 false alerts. That higher threshold scores the same saved answers and has the same model cost.
The full ReviewBench dataset has 219 PRs. This rerun scanned 171; unread targets still count as not flagged. Its labels combine reference findings and additional judgments of alerts from an earlier frozen run. The source publishes reference-only scoring and development/validation splits. Perch scores methods, while the leaderboard judges review comments, so these figures are not an identical-protocol ranking of products.
Download the inputs and reproduce the scores
Pinned ReviewBench scoring source · Pinned public dataset revision · Structured results for both benchmarks
From the matching public Perch checkout, download the saved data. The commands below score existing answers and make no model requests.
hf download perchscan/benchmark-results --repo-type dataset --revision 40285b3a4a91e3aeefd0c8f5544f5f8e94a4c3ac --include 'greptile/*' 'reviewbench/*' --local-dir .benchmarks/hf
node .benchmarks/greptile/score.mjs
node .benchmarks/reviewbench/rerun-score.mjs
node .benchmarks/reviewbench/rerun-score.mjs --results .benchmarks/hf/reviewbench/results/rerun-jev.jsonl.gz --summary .benchmarks/hf/reviewbench/rerun-jev-summary.jsonSeparate model cost from a service quote
The tables report measured model input-token costs from these saved runs. Hosted Perch analysis has its own rule-check billing, and other tools have their own allowances and meters. These results do not measure human review time, application correctness or every future model configuration.
Pair broad detection with focused regression checks
Requests authentication: executable controls and a revised guard rule shows a requirement missed by the first rule and caught after a focused revision. Four saved order-service requirements retains one recorded Perch check per input, with separate clean and broken versions. These experiments answer different questions from a public PR benchmark.
Results reproduced October 23, 2026. Published by Perch / Verglas LLC.