pbwraith/gitea-saas-core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gitea-saas-core#39: eval farm HTTP surface — register_eval_routes #44

Merged
pbwraith merged 1 commits from issue-39 into main 2026-09-03 11:05:23 +02:00
Conversation 3 Commits 1 Files Changed 2 +814 -15
pbwraith commented 2026-09-01 10:59:42 +02:00
Owner
Copy Link

Closes #39

Implements the eval-farm HTTP surface (register_eval_routes), symmetric with the factory front-end. Files touched: gitea_saas/intake_eval.py, tests/test_intake_eval.py — nothing else (app.py registration stays with the assembly issue #38).

Routes

  • POST /api/eval/batches — funded: 201 {ok, batch_id, repo, issues, hold_id}; unfunded: 202 {ok: False, reason: unfunded, queued: True, pending_id} (queued locally, no Gitea issue).
  • GET /api/eval/batches/<id> — status + per-task state; cancelled batches read as cancelled.
  • GET /api/eval/batches/<id>/results — the results table (per-task status/title/summary).
  • POST /api/eval/batches/<id>/cancel — stop draining + release the batch hold.
  • GET /api/eval/health — public liveness (handler never calls _require_auth, like /api/factory/health).

Auth (issue #24 discipline, no sixth implementation)

  • Missing/blank/wrong X-API-Key -> canonical 401 UNAUTHORIZED_BODY from gitea_saas.auth.
  • No key configured at all -> 503 SERVICE_UNAVAILABLE (per #38's assembly acceptance, which demands 503 app-wide when SAAS_API_KEY is unset; #24's consolidation converted the old per-route 503 to 401 for bad keys but the owner's #38/#39 acceptance explicitly wants 503 for unconfigured — implemented here as a config-state branch around the same key_is_valid check).

Acceptance mapping

  1. Each authed route: success test, missing/blank-key 401 test, unconfigured-key 503 test — TestEvalRoutes (health public in both configs).
  2. Unfunded submit -> queued row, no Gitea issue; funding then flush_pending ships it (issues created, row shipped) — TestQueueAndFlush, both halves asserted.
  3. Cancel releases the hold, balance returns to its pre-submit value exactly (intake-level + route-level tests); cancel is idempotent; poll/emit refuse a cancelled batch.
  4. Full suite green: 258 passed, 1 skipped (pre-existing #4 extension guard). git diff --check clean.

Design notes

  • Hold: eval_batch_hold_minor config attr / SAAS_EVAL_BATCH_HOLD_MINOR env / default = the debit floor (500); 0 disables. Placed via the billing core's reserve_hold (idempotency key eval-batch-<id>); rolled-back intake releases it (eval-rollback-<id>), so a failed submit never locks money.
  • Schema: pending_eval_batches + eval_batch_cancels (idempotent CREATE IF NOT EXISTS); hold_id column on eval_batches with a PRAGMA-guarded ALTER TABLE for pre-existing DB files.
  • Flush has no route yet — it is the mechanism acceptance 2 asserts (factory's unlock has a route; if the owner wants a client-visible flush endpoint that is a trivial follow-up, or the funding webhook can call flush_pending).
Closes #39 Implements the eval-farm HTTP surface (`register_eval_routes`), symmetric with the factory front-end. Files touched: `gitea_saas/intake_eval.py`, `tests/test_intake_eval.py` — nothing else (app.py registration stays with the assembly issue #38). Routes - `POST /api/eval/batches` — funded: 201 `{ok, batch_id, repo, issues, hold_id}`; unfunded: 202 `{ok: False, reason: unfunded, queued: True, pending_id}` (queued locally, no Gitea issue). - `GET /api/eval/batches/<id>` — status + per-task state; cancelled batches read as `cancelled`. - `GET /api/eval/batches/<id>/results` — the results table (per-task status/title/summary). - `POST /api/eval/batches/<id>/cancel` — stop draining + release the batch hold. - `GET /api/eval/health` — public liveness (handler never calls `_require_auth`, like `/api/factory/health`). Auth (issue #24 discipline, no sixth implementation) - Missing/blank/wrong `X-API-Key` -> canonical 401 `UNAUTHORIZED_BODY` from `gitea_saas.auth`. - No key configured at all -> 503 `SERVICE_UNAVAILABLE` (per #38's assembly acceptance, which demands 503 app-wide when SAAS_API_KEY is unset; #24's consolidation converted the old per-route 503 to 401 for *bad* keys but the owner's #38/#39 acceptance explicitly wants 503 for *unconfigured* — implemented here as a config-state branch around the same `key_is_valid` check). Acceptance mapping 1. Each authed route: success test, missing/blank-key 401 test, unconfigured-key 503 test — `TestEvalRoutes` (health public in both configs). 2. Unfunded submit -> queued row, no Gitea issue; funding then `flush_pending` ships it (issues created, row `shipped`) — `TestQueueAndFlush`, both halves asserted. 3. Cancel releases the hold, balance returns to its pre-submit value exactly (intake-level + route-level tests); cancel is idempotent; poll/emit refuse a cancelled batch. 4. Full suite green: 258 passed, 1 skipped (pre-existing #4 extension guard). `git diff --check` clean. Design notes - Hold: `eval_batch_hold_minor` config attr / `SAAS_EVAL_BATCH_HOLD_MINOR` env / default = the debit floor (500); 0 disables. Placed via the billing core's `reserve_hold` (idempotency key `eval-batch-<id>`); rolled-back intake releases it (`eval-rollback-<id>`), so a failed submit never locks money. - Schema: `pending_eval_batches` + `eval_batch_cancels` (idempotent CREATE IF NOT EXISTS); `hold_id` column on `eval_batches` with a PRAGMA-guarded `ALTER TABLE` for pre-existing DB files. - Flush has no route yet — it is the mechanism acceptance 2 asserts (factory's unlock has a route; if the owner wants a client-visible flush endpoint that is a trivial follow-up, or the funding webhook can call `flush_pending`).
pbwraith added 1 commit 2026-09-01 10:59:44 +02:00 
Five routes symmetric with the factory front-end:
  POST /api/eval/batches                (funded 201 / unfunded 202 queued)
  GET  /api/eval/batches/<id>           status + per-task state
  GET  /api/eval/batches/<id>/results   the results table
  POST /api/eval/batches/<id>/cancel    stop draining + release the hold
  GET  /api/eval/health                 liveness, no auth

Auth delegates to gitea_saas.auth (issue #24 single path — no sixth
implementation): missing/blank key -> canonical 401; no key configured
at all -> 503 SERVICE_UNAVAILABLE (issue #38's assembly acceptance).

Unfunded submits now queue in pending_eval_batches (no Gitea issue is
ever created); flush_pending ships them after funding, exactly like a
fresh funded submit. Funded submits reserve eval_batch_hold_minor
(SAAS_EVAL_BATCH_HOLD_MINOR, default = debit floor) via the billing
core's reserve_hold; cancel releases it so the balance returns to its
pre-submit value exactly, and marks the batch cancelled (poll/emit
become no-ops). Rolled-back intake releases the hold too.

Schema: pending_eval_batches + eval_batch_cancels tables (idempotent
CREATE IF NOT EXISTS), hold_id column with a PRAGMA-guarded ALTER for
pre-existing DB files.
pbwraith added the
in-review
 label 2026-09-01 11:00:21 +02:00
pbwraith commented 2026-09-01 11:00:45 +02:00
Author
Owner
Copy Link

[lane2] gitea-saas-core#39 implemented, PR open for manual review.

Commit: 8620d4d on branch issue-39 (based on origin/main @ 71cc263, no existing branch/PR for #39 before creation).
PR: #44 (state=open, merged=false, base=main, head=issue-39), changed files: gitea_saas/intake_eval.py, tests/test_intake_eval.py — exactly the two files the issue owns.

Verification evidence:

  • New tests (TestEvalRoutes / TestQueueAndFlush / TestHoldAndCancel): each authed route covered for success, missing/blank key (401 canonical UNAUTHORIZED_BODY), unconfigured key (503 SERVICE_UNAVAILABLE); /api/eval/health public in both configurations.
  • Acceptance 2 (both halves asserted): unfunded submit -> pending_eval_batches row, zero Gitea issues/repos; after funding, flush_pending ships the batch (issues created, row marked shipped, batch holds reserved).
  • Acceptance 3: submit reserves eval_batch_hold_minor (default 500) via BillingCore.reserve_hold; cancel releases it — balance returns to its pre-submit value exactly (asserted at intake and route level); cancel idempotent (second call already_cancelled, no double release); poll_completed/emit refuse a cancelled batch.
  • Full suite: 258 passed, 1 skipped (pre-existing #4 extension guard) on the committed tree; git diff --check clean.

One deliberate deviation to flag for review: acceptance 1 says "unconfigured key (503)" — the #24 consolidation ships 401 for empty configured keys across the other front-ends, but issue #38 (assembly) explicitly requires 503 app-wide when SAAS_API_KEY is unset, so register_eval_routes returns 503 SERVICE_UNAVAILABLE when no key is configured while keeping the canonical 401 for missing/blank/wrong keys. No new auth implementation: still key_is_valid + API_KEY_HEADER + unauthorized_response from gitea_saas.auth.

[lane2] gitea-saas-core#39 implemented, PR open for manual review. Commit: 8620d4d on branch issue-39 (based on origin/main @ 71cc263, no existing branch/PR for #39 before creation). PR: #44 (state=open, merged=false, base=main, head=issue-39), changed files: gitea_saas/intake_eval.py, tests/test_intake_eval.py — exactly the two files the issue owns. Verification evidence: - New tests (TestEvalRoutes / TestQueueAndFlush / TestHoldAndCancel): each authed route covered for success, missing/blank key (401 canonical UNAUTHORIZED_BODY), unconfigured key (503 SERVICE_UNAVAILABLE); /api/eval/health public in both configurations. - Acceptance 2 (both halves asserted): unfunded submit -> pending_eval_batches row, zero Gitea issues/repos; after funding, flush_pending ships the batch (issues created, row marked shipped, batch holds reserved). - Acceptance 3: submit reserves eval_batch_hold_minor (default 500) via BillingCore.reserve_hold; cancel releases it — balance returns to its pre-submit value exactly (asserted at intake and route level); cancel idempotent (second call already_cancelled, no double release); poll_completed/emit refuse a cancelled batch. - Full suite: 258 passed, 1 skipped (pre-existing #4 extension guard) on the committed tree; git diff --check clean. One deliberate deviation to flag for review: acceptance 1 says "unconfigured key (503)" — the #24 consolidation ships 401 for empty configured keys across the other front-ends, but issue #38 (assembly) explicitly requires 503 app-wide when SAAS_API_KEY is unset, so register_eval_routes returns 503 SERVICE_UNAVAILABLE when no key is configured while keeping the canonical 401 for missing/blank/wrong keys. No new auth implementation: still key_is_valid + API_KEY_HEADER + unauthorized_response from gitea_saas.auth.
pbwraith commented 2026-09-01 22:48:00 +02:00
Author
Owner
Copy Link

Held — main is red, and it is not this PR's fault

Gated against the real merge result. The comparison that matters:

tree pytest
origin/main @ 71cc263 (no PR) 10 failed, 225 passed
main + this PR 10 failed, 248 passed

The same 10 failures, and this PR adds 23 passing tests and zero new
failures. Every failure is in tests/test_review_workspace.py /
tests/test_reviewer.py — files this PR does not touch.

So this is not a bounce and nothing is being asked of you. The work is good; it
is queued behind a pre-existing defect filed as #48: on Windows the shell
eats the ^ in git rev-parse --verify <sha>^{commit}, so ReviewWorkspace
cannot build a synthetic merge on the review box at all.

I will not merge onto a red main — with the baseline broken there is no way to
tell a genuine regression from the existing noise, which is exactly how
botmanager shipped a broken master last week. Once #48 is green I will
re-gate and merge.

## Held — `main` is red, and it is not this PR's fault Gated against the real merge result. The comparison that matters: | tree | pytest | |---|---| | `origin/main` @ `71cc263` (**no PR**) | **10 failed**, 225 passed | | `main` + this PR | **10 failed**, 248 passed | **The same 10 failures, and this PR adds 23 passing tests and zero new failures.** Every failure is in `tests/test_review_workspace.py` / `tests/test_reviewer.py` — files this PR does not touch. So this is not a bounce and nothing is being asked of you. The work is good; it is queued behind a pre-existing defect filed as #48: on Windows the shell eats the `^` in `git rev-parse --verify <sha>^{commit}`, so `ReviewWorkspace` cannot build a synthetic merge on the review box at all. I will not merge onto a red `main` — with the baseline broken there is no way to tell a genuine regression from the existing noise, which is exactly how botmanager shipped a broken `master` last week. Once #48 is green I will re-gate and merge.
pbwraith commented 2026-09-03 11:05:20 +02:00
Author
Owner
Copy Link

Merged — verified against green main

259 passed, 1 skipped, 0 failed; main after !49 is 236 passed. 0 conflicts, confirmed with a real git merge after Gitea briefly reported this unmergeable — that was stale metadata from main moving, not a genuine conflict.

This PR was held, never bounced: it was queued behind the Windows caret defect (#48). The eval farm was library-only while the factory front-end had five routes; ARCHITECTURE.md calls them the two front-ends the billing core serves. Both are now reachable over HTTP.

## Merged — verified against green `main` **259 passed, 1 skipped, 0 failed**; `main` after !49 is 236 passed. **0 conflicts**, confirmed with a real `git merge` after Gitea briefly reported this unmergeable — that was stale metadata from `main` moving, not a genuine conflict. This PR was *held*, never bounced: it was queued behind the Windows caret defect (#48). The eval farm was library-only while the factory front-end had five routes; ARCHITECTURE.md calls them the two front-ends the billing core serves. Both are now reachable over HTTP.
pbwraith merged commit 5fda4c09b6 into main 2026-09-03 11:05:23 +02:00
pbwraith removed the
in-review
 label 2026-09-03 11:05:25 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked

   
bounced

   
eval-task

   
in-progress

   
in-review

   
needs-guidance

   
p1

   
p2

   
p3

   
tracking

   
unfunded
No Label
blocked
bounced
eval-task
in-progress
in-review
needs-guidance
p1
p2
p3
tracking
unfunded
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: pbwraith/gitea-saas-core#44
No description provided.
Delete Branch "issue-39"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?