Four recurring order-service mistakes, one committed regression patch and one clean refactor patch. Every reviewer receives the same requirements and public code.
Perch flagged all four targeted regressions and none of the four clean requirements in this run. The executable contract tests pass on the baseline and clean source and fail on the regression source. Competitor reviews without completed authenticated runs remain unscored.
| Service | Targeted regressions flagged | Clean requirements flagged | Time | Cost |
|---|---|---|---|---|
| Perch | 4/4 | 0/4 | 0.436s regression; 0.389s clean | 0.08 US cents / eight fresh checks across both runs; rate-derived estimate, actual bill unavailable |
| CodeRabbit | Not run | Not scored | Not available | Not available |
| Greptile | Not run | Not scored | Not available | Not available |
Perch CLI DEVELOPMENT (4af67af), model jev-latest, four scoped file rules, forced fresh checks. The 0.08¢ customer-price estimate uses eight fresh checks at 10¢/1,000; it is separate from an observed billing receipt. Times exclude installation, login and repository indexing. The other installed CLIs are CodeRabbit 0.9.0 and Greptile 3.6.2.
Perch used the scoped requirements from the downloadable rule pack and forced fresh model checks through its configured direct SystemOne endpoint, https://api.typesafe.ai/v1/systemone. The elapsed times describe that execution path. This is the actual command captured for both public input variants:
perch check review-evaluation/order_service.py --rules eval-all-stock-lines,eval-malformed-order-errors,eval-adjacent-expirations,eval-complete-subtotal --force --jsonDownload the exact Perch rules · Download the scoped rule pack · CodeRabbit CLI command documentation · Greptile CLI command documentation
Read can_fulfil only. It returns true only when every requested item has sufficient stock; a shortage on any line returns false, including when an earlier line has enough stock.
Perch regression score 95%; gate 50%. Clean control: no rule finding.
Read load_order only. Missing files and malformed JSON propagate an exception to the caller instead of being converted into an empty order.
Perch regression score 96%; gate 50%. Clean control: no rule finding.
Read release_expired only. It releases every expired reservation, including adjacent expired entries, removes those entries, and leaves unexpired reservations intact.
Perch regression score 69%; gate 50%. Clean control: no rule finding.
Read subtotal only. It includes price times quantity for every cart line, including the last line, and returns Decimal zero for an empty cart.
Perch regression score 96%; gate 50%. Clean control: no rule finding.
Base: 772ea00fd54bf96499ada83ccbb50d459d5d8a0d. The instruction file and patch hashes are recorded in the structured report.
Identical reviewer instructions · Perch raw JSON and commands · Executable contract tests · Runtime control output
Identical committed base and head code; exact SHA values and patch hashes recorded before reviews. The same four behavior requirements are provided to every reviewer. No clean/broken labels are included in the instruction file or reviewed checkout.
One regression-suite review and one clean-control review per available service. No repeated runs selected for a better score. For each requirement, report whether the regression was identified and whether the clean control was flagged. Other comments are retained for separate adjudication; unavailable or incomplete reviews are not scored as clean.
This is a focused disclosed-requirement suite, not a representative all-bug accuracy benchmark or a complete security evaluation. Fresh judgments can vary. Different review surfaces and model choices are reported. Four disclosed behavior requirements on one Python helper file. Perch results are complete; unavailable competitors are not counted as misses. This suite is not representative all-bug accuracy or proof of comparative superiority.
CodeRabbit and Greptile are installed but do not currently have authenticated review sessions for these inputs. Their missing results are not treated as findings, misses or clean reviews.