Calibration harness that measures the existing review gate's false-approve
rate on real farm history before gold-tier auto-merge is sold:
- tests/calibration/corpus/ — 25 real farm PRs with known outcomes
(merged/bounced) from 11 pbwraith repos; each entry records the inputs the
gate saw (issue body, changed files, diff, observed test/hygiene exit).
- tests/calibration/manifest.json — curated known outcome, category, and
recorded gate inputs per entry.
- tests/calibration/run_calibration.py — one-line command that re-runs the
EXISTING deterministic reviewer against the corpus and regenerates
tests/calibration/calibration_report.md.
- CALIBRATION.md — the measured rate and the recommendation.
Result: 0/14 false-approve on the deterministic gate (44% of real PRs refused
as unverifiable), and 2/16 = 12.5% on the operator's terms (both
owner-documented botmanager regressions included: #220's wrong-check-target
ship and !22's sibling-merge regression). Recommendation: not safe to sell
gold auto-merge yet; close acceptance-format coverage, test-target selection,
and sibling-merge verification first. reviewer.py / review_policy.py
untouched (read-only).
