gitea-saas-core#42: review-gate calibration — measure the false-approve rate before selling auto-merge #47

pbwraith · 2026-09-01T13:39:58+02:00

pbwraith commented

2026-09-01 13:39:58 +02:00

Closes #42

What

A calibration harness that measures the existing review gate's false-approve rate on real farm history — before gold-tier auto-merge is sold. Measure first; reviewer.py / review_policy.py are NOT touched (read-only).

Deliverables

Corpus — tests/calibration/corpus/: 25 real farm PRs with KNOWN correct outcomes, drawn from 11 pbwraith repos' merged and bounced history. Each entry records the inputs the gate saw (linked issue body, changed files, diff, observed test/hygiene exit). The two botmanager cases from the issue are in the corpus with their outcomes recorded:
- botmanager-220 (!220, the undefined-component crash shipper, botmanager#223): known outcome should_bounce, recorded green gate inputs.
- botmanager-bang22 (!22, sibling-merge regression): known outcome should_bounce, recorded as owner-documented (original head predates the repo's renumbering; closest surviving record botmanager#201).
Runner — tests/calibration/run_calibration.py: one-line command (python tests/calibration/run_calibration.py) re-runs the existing deterministic reviewer against the corpus and regenerates the report. No network, deterministic.
Committed report — tests/calibration/calibration_report.md.
Recommendation — CALIBRATION.md.

Measured numbers

False-approve rate (deterministic gate): 0 / 14 = 0.0% — but 11/25 (44%) of real PRs were refused as unverifiable (NEEDS_GUIDANCE), including both owner-documented regressions.
With the acceptance-format gap closed (botmanager#220's real '## Acceptance criteria' items parsed): verdict flips to APPROVE on the crash-shipping head — FALSE-APPROVE.
On the operator's terms (both documented cases + the 14 the gate can evaluate): 2 / 16 = 12.5%.

Recommendation

NO — gold-tier auto-merge is not safe to sell yet. 0% is an artifact of refusal, not quality: the approve path has no check that would have stopped either documented regression (wrong test target; no sibling-merge verification), and the gate cannot evaluate 44% of real farm issues (acceptance-format blindness). Close coverage + test-target selection + sibling-merge verification first, then re-run this calibration.

Gate

python -m pytest -q: 239 passed, 1 skipped (pre-existing skip)
git diff --check: clean
New files only: tests/calibration/ + CALIBRATION.md

Closes #42 ## What A calibration harness that measures the existing review gate's false-approve rate on real farm history — before gold-tier auto-merge is sold. Measure first; reviewer.py / review_policy.py are NOT touched (read-only). ## Deliverables 1. **Corpus** — `tests/calibration/corpus/`: 25 real farm PRs with KNOWN correct outcomes, drawn from 11 pbwraith repos' merged and bounced history. Each entry records the inputs the gate saw (linked issue body, changed files, diff, observed test/hygiene exit). The two botmanager cases from the issue are in the corpus with their outcomes recorded: - `botmanager-220` (`!220`, the undefined-component crash shipper, botmanager#223): known outcome **should_bounce**, recorded green gate inputs. - `botmanager-bang22` (`!22`, sibling-merge regression): known outcome **should_bounce**, recorded as owner-documented (original head predates the repo's renumbering; closest surviving record botmanager#201). 2. **Runner** — `tests/calibration/run_calibration.py`: one-line command (`python tests/calibration/run_calibration.py`) re-runs the existing deterministic reviewer against the corpus and regenerates the report. No network, deterministic. 3. **Committed report** — `tests/calibration/calibration_report.md`. 4. **Recommendation** — `CALIBRATION.md`. ## Measured numbers - False-approve rate (deterministic gate): **0 / 14 = 0.0%** — but 11/25 (44%) of real PRs were refused as unverifiable (NEEDS_GUIDANCE), including both owner-documented regressions. - With the acceptance-format gap closed (botmanager#220's real '## Acceptance criteria' items parsed): verdict flips to **APPROVE on the crash-shipping head** — FALSE-APPROVE. - On the operator's terms (both documented cases + the 14 the gate can evaluate): **2 / 16 = 12.5%**. ## Recommendation **NO — gold-tier auto-merge is not safe to sell yet.** 0% is an artifact of refusal, not quality: the approve path has no check that would have stopped either documented regression (wrong test target; no sibling-merge verification), and the gate cannot evaluate 44% of real farm issues (acceptance-format blindness). Close coverage + test-target selection + sibling-merge verification first, then re-run this calibration. ## Gate - `python -m pytest -q`: **239 passed, 1 skipped** (pre-existing skip) - `git diff --check`: clean - New files only: `tests/calibration/` + `CALIBRATION.md`

pbwraith added 1 commit 2026-09-01 13:39:58 +02:00

gitea-saas-core#42: review-gate calibration corpus + report (measure, don't tune) cf798a809f

Calibration harness that measures the existing review gate's false-approve
rate on real farm history before gold-tier auto-merge is sold:

- tests/calibration/corpus/ — 25 real farm PRs with known outcomes
  (merged/bounced) from 11 pbwraith repos; each entry records the inputs the
  gate saw (issue body, changed files, diff, observed test/hygiene exit).
- tests/calibration/manifest.json — curated known outcome, category, and
  recorded gate inputs per entry.
- tests/calibration/run_calibration.py — one-line command that re-runs the
  EXISTING deterministic reviewer against the corpus and regenerates
  tests/calibration/calibration_report.md.
- CALIBRATION.md — the measured rate and the recommendation.

Result: 0/14 false-approve on the deterministic gate (44% of real PRs refused
as unverifiable), and 2/16 = 12.5% on the operator's terms (both
owner-documented botmanager regressions included: #220's wrong-check-target
ship and !22's sibling-merge regression). Recommendation: not safe to sell
gold auto-merge yet; close acceptance-format coverage, test-target selection,
and sibling-merge verification first. reviewer.py / review_policy.py
untouched (read-only).

pbwraith referenced this pull request

2026-09-01 13:40:29 +02:00

Measure the review gate's false-approve rate before selling auto-merge #42

pbwraith commented

2026-09-01 22:48:03 +02:00

Held — `main` is red, and it is not this PR's fault

Gated against the real merge result. The comparison that matters:

tree	pytest
`origin/main` @ `71cc263` (no PR)	10 failed, 225 passed
`main` + this PR	10 failed, 229 passed

The same 10 failures, and this PR adds 4 passing tests and zero new
failures. Every failure is in tests/test_review_workspace.py /
tests/test_reviewer.py — files this PR does not touch.

So this is not a bounce and nothing is being asked of you. The work is good; it
is queued behind a pre-existing defect filed as #48: on Windows the shell
eats the ^ in git rev-parse --verify <sha>^{commit}, so ReviewWorkspace
cannot build a synthetic merge on the review box at all.

I will not merge onto a red main — with the baseline broken there is no way to
tell a genuine regression from the existing noise, which is exactly how
botmanager shipped a broken master last week. Once #48 is green I will
re-gate and merge.

## Held — `main` is red, and it is not this PR's fault Gated against the real merge result. The comparison that matters: | tree | pytest | |---|---| | `origin/main` @ `71cc263` (**no PR**) | **10 failed**, 225 passed | | `main` + this PR | **10 failed**, 229 passed | **The same 10 failures, and this PR adds 4 passing tests and zero new failures.** Every failure is in `tests/test_review_workspace.py` / `tests/test_reviewer.py` — files this PR does not touch. So this is not a bounce and nothing is being asked of you. The work is good; it is queued behind a pre-existing defect filed as #48: on Windows the shell eats the `^` in `git rev-parse --verify <sha>^{commit}`, so `ReviewWorkspace` cannot build a synthetic merge on the review box at all. I will not merge onto a red `main` — with the baseline broken there is no way to tell a genuine regression from the existing noise, which is exactly how botmanager shipped a broken `master` last week. Once #48 is green I will re-gate and merge.

pbwraith referenced this pull request

2026-09-02 00:22:53 +02:00

gitea-saas-core#48: pass git commands as arg lists, never shell strings (Windows caret fix) #49

pbwraith referenced this pull request

2026-09-03 10:50:40 +02:00

gitea-saas-core#48: pass git commands as arg lists, never shell strings (Windows caret fix) #49

pbwraith commented

2026-09-03 11:05:33 +02:00

Merged — verified against green `main`

240 passed, 1 skipped, 0 failed; main after !49 is 236 passed. 0 conflicts, confirmed with a real git merge after Gitea briefly reported this unmergeable — that was stale metadata from main moving, not a genuine conflict.

This PR was held, never bounced: it was queued behind the Windows caret defect (#48). A 24-PR corpus from this farm's real history, including botmanager-220 and botmanager-231 — the two cases that would have auto-merged under gold tier. The number is the deliverable, not a passing test.

## Merged — verified against green `main` **240 passed, 1 skipped, 0 failed**; `main` after !49 is 236 passed. **0 conflicts**, confirmed with a real `git merge` after Gitea briefly reported this unmergeable — that was stale metadata from `main` moving, not a genuine conflict. This PR was *held*, never bounced: it was queued behind the Windows caret defect (#48). A 24-PR corpus from this farm's real history, including botmanager-220 and botmanager-231 — the two cases that would have auto-merged under gold tier. The number is the deliverable, not a passing test.

pbwraith merged commit 2ef1e668fa into main

2026-09-03 11:05:36 +02:00

pbwraith referenced this issue from a commit

2026-09-03 11:05:37 +02:00

gitea-saas-core#42: review-gate calibration (#47)

Sign in to join this conversation.

No reviewers