Source: https://reviewbenchmarks.dev/evaluations/order-service/

Captured 2026-10-14 · public commits · disclosed requirements

# Check the same behavior requirements.

Four recurring order-service mistakes, one committed regression patch and one clean refactor patch. Every reviewer receives the same requirements and public code.

Perch flagged all four targeted regressions and none of the four clean requirements in this run. The executable contract tests pass on the baseline and clean source and fail on the regression source. Competitor reviews without completed authenticated runs remain unscored.

[Download inputs and captured results](https://reviewbenchmarks.dev/evaluations/order-service/inputs.zip)[Structured evaluation report](https://reviewbenchmarks.dev/evaluations/order-service/report.json)

## Results, costs and review time

| Service | Targeted regressions flagged | Clean requirements flagged | Time | Cost |
| --- | --- | --- | --- | --- |
| Perch | 4/4 | 0/4 | 0.436s regression; 0.389s clean | 0.08 US cents / eight fresh checks across both runs; rate-derived estimate, actual bill unavailable |
| CodeRabbit | Not run | Not scored | Not available | Not available |
| Greptile | Not run | Not scored | Not available | Not available |

Perch CLI DEVELOPMENT (4af67af), model jev-latest, four scoped file rules, forced fresh checks. The 0.08¢ customer-price estimate uses eight fresh checks at 10¢/1,000; it is separate from an observed billing receipt. Times exclude installation, login and repository indexing. The other installed CLIs are CodeRabbit 0.9.0 and Greptile 3.6.2.

## Recorded reviewer configuration

Perch used the scoped requirements from the downloadable rule pack and forced fresh model checks through its configured direct SystemOne endpoint, https://api.typesafe.ai/v1/systemone. The elapsed times describe that execution path. This is the actual command captured for both public input variants:

```
perch check review-evaluation/order_service.py --rules eval-all-stock-lines,eval-malformed-order-errors,eval-adjacent-expirations,eval-complete-subtotal --force --json
```

[Download the exact Perch rules](https://reviewbenchmarks.dev/evaluations/order-service/rules.yaml) · [Download the scoped rule pack](https://reviewbenchmarks.dev/evaluations/order-service/rule-pack.zip) · [CodeRabbit CLI command documentation](https://docs.coderabbit.ai/cli/reference) · [Greptile CLI command documentation](https://www.greptile.com/docs/code-review/greptile-cli)

## Four requirements with executable controls

### can_fulfil

Read can_fulfil only. It returns true only when every requested item has sufficient stock; a shortage on any line returns false, including when an earlier line has enough stock.

Perch regression score 95%; gate 50%. Clean control: no rule finding.

### load_order

Read load_order only. Missing files and malformed JSON propagate an exception to the caller instead of being converted into an empty order.

Perch regression score 96%; gate 50%. Clean control: no rule finding.

### release_expired

Read release_expired only. It releases every expired reservation, including adjacent expired entries, removes those entries, and leaves unexpired reservations intact.

Perch regression score 69%; gate 50%. Clean control: no rule finding.

### subtotal

Read subtotal only. It includes price times quantity for every cart line, including the last line, and returns Decimal zero for an empty cart.

Perch regression score 96%; gate 50%. Clean control: no rule finding.

## Use immutable base and head inputs

Base: `772ea00fd54bf96499ada83ccbb50d459d5d8a0d`. The instruction file and patch hashes are recorded in the structured report.

- [Public regression change](https://github.com/lakeday-org/perch/pull/371) · [source at 46dd1d0](https://github.com/lakeday-org/perch/blob/46dd1d0d0ea7ceb0e690203d24fca21c428818ec/review-evaluation/order_service.py) · [download patch](https://reviewbenchmarks.dev/evaluations/order-service/regression.patch)
- [Public clean change](https://github.com/lakeday-org/perch/pull/372) · [source at b9ce63a](https://github.com/lakeday-org/perch/blob/b9ce63ac935739a998ebb40ca6881a261571d208/review-evaluation/order_service.py) · [download patch](https://reviewbenchmarks.dev/evaluations/order-service/clean.patch)

[Identical reviewer instructions](https://reviewbenchmarks.dev/evaluations/order-service/shared-instructions.md) · [Perch raw JSON and commands](https://reviewbenchmarks.dev/evaluations/order-service/perch-results.json) · [Executable contract tests](https://reviewbenchmarks.dev/evaluations/order-service/test_behavior.py) · [Runtime control output](https://reviewbenchmarks.dev/evaluations/order-service/runtime-results.json)

## Compare the same requirement and scope

Identical committed base and head code; exact SHA values and patch hashes recorded before reviews. The same four behavior requirements are provided to every reviewer. No clean/broken labels are included in the instruction file or reviewed checkout.

One regression-suite review and one clean-control review per available service. No repeated runs selected for a better score. For each requirement, report whether the regression was identified and whether the clean control was flagged. Other comments are retained for separate adjudication; unavailable or incomplete reviews are not scored as clean.

This is a focused disclosed-requirement suite, not a representative all-bug accuracy benchmark or a complete security evaluation. Fresh judgments can vary. Different review surfaces and model choices are reported. Four disclosed behavior requirements on one Python helper file. Perch results are complete; unavailable competitors are not counted as misses. This suite is not representative all-bug accuracy or proof of comparative superiority.

CodeRabbit and Greptile are installed but do not currently have authenticated review sessions for these inputs. Their missing results are not treated as findings, misses or clean reviews.

**Review Benchmarks**

Methods, controls & recorded results.

Published by Perch · © 2026 Verglas LLC. Review Benchmarks is a Perch publication.

Perch’s MIT license covers the CLI; hosted model analysis is a paid service. Relevant source is sent to the configured model service.

[Perch CLI MIT license](https://github.com/lakeday-org/perch/blob/main/LICENSE) · [Article text](https://reviewbenchmarks.dev/evaluations/order-service/index.md) · [Resource index](https://reviewbenchmarks.dev/llms.txt)