Eval farm: batch intake -> lane drain -> results emitter #8

New Issue

pbwraith · 2026-08-25T11:11:26+02:00

pbwraith commented

2026-08-25 11:11:26 +02:00

Goal

Eval farm front-end (option 2): batch task intake, drain through the lane system, collect results, emit to a callback (webhook or file drop). Turns the lane machine into a paid eval/QA grind.

Depends on: issues #1-2 (billing core). Files allowed: gitea_saas/intake_eval.py, tests/test_intake_eval.py. Do NOT modify anything else.

Spec

Schema additions: eval_batches(id, client_id, status, callback_url, created_at), eval_tasks(id, batch_id, repo, issue_number, task_id, payload_json, status, result_json, created_at).
EvalIntake(core, gitea, emitter):
- submit_batch(client_id, tasks: [{task_id, title, body}], callback_url=None):
  - funded check like factory (issue #4 semantics): funded → proceed; unfunded → 402-style {ok: False, reason: "unfunded"}
  - creates ONE internal eval repo per batch (eval-<client_id>-<batch_id>, or reuse a shared eval-runs repo config SAAS_EVAL_REPO), creates one Gitea issue PER TASK with eval-task label, body = task spec + acceptance + batch: <id> marker
  - returns {batch_id, issues: [{task_id, issue_number}]}
- poll_completed(batch_id) — for each task issue, if closed (lane merged): read final comment / labels, store result_json (from PR/lane report), mark done
- emit(batch_id) — once all tasks done: JSON report {batch_id, client_id, results: [{task_id, status, summary}]} POSTed to callback_url (retry x3, backoff), or written to SAAS_EVAL_OUTPUT_DIR/<batch_id>.json when callback_url empty. Mark batch emitted.
Server additions (issue #2 file allowed to extend): POST /api/eval/batches, GET /api/eval/batches/{id}, POST /api/eval/batches/{id}/emit.

Acceptance

funded batch creates one issue per task with eval-task label + batch marker
unfunded batch → 402, no Gitea issues created
poll_completed stores results from closed issues; skips open ones
emit posts full JSON to callback (mock emitter verifies payload + retry on 500 x3); file-drop mode writes file
batch marked emitted only after all tasks done; idempotent re-emit
tests hermetic (MockGitea, mock emitter, tmp DB)

Verification: python -m pytest tests/test_intake_eval.py -q green.

## Goal Eval farm front-end (option 2): batch task intake, drain through the lane system, collect results, emit to a callback (webhook or file drop). Turns the lane machine into a paid eval/QA grind. **Depends on:** issues #1-2 (billing core). **Files allowed:** `gitea_saas/intake_eval.py`, `tests/test_intake_eval.py`. Do NOT modify anything else. ## Spec - Schema additions: `eval_batches(id, client_id, status, callback_url, created_at)`, `eval_tasks(id, batch_id, repo, issue_number, task_id, payload_json, status, result_json, created_at)`. - `EvalIntake(core, gitea, emitter)`: - `submit_batch(client_id, tasks: [{task_id, title, body}], callback_url=None)`: - funded check like factory (issue #4 semantics): funded → proceed; unfunded → 402-style `{ok: False, reason: "unfunded"}` - creates ONE internal eval repo per batch (`eval-<client_id>-<batch_id>`, or reuse a shared `eval-runs` repo config `SAAS_EVAL_REPO`), creates one Gitea issue PER TASK with `eval-task` label, body = task spec + acceptance + `batch: <id>` marker - returns `{batch_id, issues: [{task_id, issue_number}]}` - `poll_completed(batch_id)` — for each task issue, if closed (lane merged): read final comment / labels, store `result_json` (from PR/lane report), mark `done` - `emit(batch_id)` — once all tasks done: JSON report `{batch_id, client_id, results: [{task_id, status, summary}]}` POSTed to `callback_url` (retry x3, backoff), or written to `SAAS_EVAL_OUTPUT_DIR/<batch_id>.json` when callback_url empty. Mark batch `emitted`. - Server additions (issue #2 file allowed to extend): `POST /api/eval/batches`, `GET /api/eval/batches/{id}`, `POST /api/eval/batches/{id}/emit`. ## Acceptance - [x] funded batch creates one issue per task with eval-task label + batch marker - [x] unfunded batch → 402, no Gitea issues created - [x] poll_completed stores results from closed issues; skips open ones - [x] emit posts full JSON to callback (mock emitter verifies payload + retry on 500 x3); file-drop mode writes file - [x] batch marked emitted only after all tasks done; idempotent re-emit - [x] tests hermetic (MockGitea, mock emitter, tmp DB) **Verification:** `python -m pytest tests/test_intake_eval.py -q` green.

pbwraith added the

p2

label 2026-08-25 11:11:26 +02:00

pbwraith referenced a pull request that will close this issue

2026-08-25 13:22:21 +02:00

Eval farm: batch intake -> lane drain -> results emitter (#8) #20

pbwraith referenced this issue

2026-08-25 13:22:40 +02:00

Eval farm: batch intake -> lane drain -> results emitter (#8) #20

pbwraith referenced this issue

2026-08-25 13:33:15 +02:00

gitea_saas/gitea_client.py has three competing authors: give it one owning issue #21