Claude cf798a809f gitea-saas-core#42: review-gate calibration corpus + report (measure, don't tune)

Calibration harness that measures the existing review gate's false-approve
rate on real farm history before gold-tier auto-merge is sold:

- tests/calibration/corpus/ — 25 real farm PRs with known outcomes
  (merged/bounced) from 11 pbwraith repos; each entry records the inputs the
  gate saw (issue body, changed files, diff, observed test/hygiene exit).
- tests/calibration/manifest.json — curated known outcome, category, and
  recorded gate inputs per entry.
- tests/calibration/run_calibration.py — one-line command that re-runs the
  EXISTING deterministic reviewer against the corpus and regenerates
  tests/calibration/calibration_report.md.
- CALIBRATION.md — the measured rate and the recommendation.

Result: 0/14 false-approve on the deterministic gate (44% of real PRs refused
as unverifiable), and 2/16 = 12.5% on the operator's terms (both
owner-documented botmanager regressions included: #220's wrong-check-target
ship and !22's sibling-merge regression). Recommendation: not safe to sell
gold auto-merge yet; close acceptance-format coverage, test-target selection,
and sibling-merge verification first. reviewer.py / review_policy.py
untouched (read-only).

2026-09-01 06:39:22 -05:00

3.6 KiB

Raw Permalink Blame History

CALIBRATION.md — review-gate false-approve rate (issue #42)

Outcome

The number the gold tier's auto-merge promise needs: how often the review gate approves a change it should have bounced.

Measured over a corpus of 25 real farm PRs with known outcomes (drawn from 11 pbwraith repos' merged and bounced history) plus the two owner-documented regressions the issue cites (botmanager#223's shipper and botmanager !22). The gate under test is the shipped gitea_saas.reviewer.Reviewer at its DEFAULT (deterministic) config, run unchanged against recorded inputs — issue body, changed files, diff, and the observed test/hygiene exit from farm history.

The number

rate	value
false-approve rate (approved, should have bounced)	0 / 14 = 0.0%
false-approve prevalence (over 25 reviewable PRs)	0.0%
false-bounce rate (bounced, should have approved)	n/a (0 bounces produced)
unverifiable — NEEDS_GUIDANCE (fail-closed refusal)	11 / 25 = 44.0%

On the operator's own terms — the two owner-documented cases added to the 14 the gate can evaluate — the false-approve rate is 2 / 16 = 12.5%.

Recommendation: NO

Gold-tier auto-merge (auto-merges on a green APPROVE) is not safe to sell at this rate. The 0.0% headline is an artifact of refusal, not of gate quality:

The gate approved only the 14 corpus PRs whose issues carry - [ ] checklists (or a section titled exactly ## Acceptance) and refused the other 11 (44%) — real merged or bounced work it could not verify. A gate that cannot evaluate 44% of real issues cannot back an auto-merge guarantee.
The approve path has no check that would have stopped either documented regression: botmanager#223's break was green under the configured gate (pytest gating a frontend-only change; green on a case-sensitive Linux worktree while the target is case-insensitive Windows) and botmanager !22 was green in isolation (no sibling-merge/integration verification). Both mechanisms survive in the approve path today.
Accident of format, not fail-closed design: botmanager#220 (the shipper) reached NEEDS_GUIDANCE only because its issue titles the section ## Acceptance criteria with a numbered list, which parse_acceptance() does not recognise. With the issue's own real acceptance items parsed (one-line parser accommodation, measured sensitivity), the gate returns APPROVE on that head — a false approve.

Before selling gold auto-merge: (a) close the acceptance-format coverage gap, (b) select the test command/suite from the changed-file set (the gitea-worker-ops !44 direction), (c) add sibling-merge/integration verification, then (d) re-run this calibration on the fixed gate and a larger corpus.

How to re-run

From the repo root (one line, no network, deterministic):

python tests/calibration/run_calibration.py

It re-runs the existing reviewer against the committed corpus (tests/calibration/corpus/*.json + tests/calibration/manifest.json) and rewrites this report of record: tests/calibration/calibration_report.md.

Corpus provenance: the raw Gitea snapshots were fetched read-only on 2026-09-01 (see tests/calibration/build_corpus.py); the committed corpus is the frozen snapshot the numbers are derived from.

Scope compliance

New files only: tests/calibration/ (corpus, runner, builder, report, tests) and this file.
gitea_saas/reviewer.py and gitea_saas/review_policy.py are read-only — not modified. The calibration measures the existing gate; it does not tune it (tuning against a corpus you also authored is how you get a good number and a bad reviewer).

3.6 KiB Raw Permalink Blame History