Move per-PR eval-gate to pytest fake-LLM; run real Gemini eval nightly

[augustineuzokwe]

Problem

The CI regression job currently makes ~30 sequential live Gemini calls on every PR + every push-to-main. Today's main runs failed back-to-back on 503 (attempt 1) and 504 (attempt 2, during the judge layer) — different status codes on consecutive attempts confirm real backend flakiness, not a code bug. Earlier runs today hit 429 RESOURCE_EXHAUSTED on the project monthly spend cap. The retry-classifier merged in 760999d correctly distinguishes these, but the structural problem — gating PRs on ~30 sequential calls to a third-party API — remains.

At a ~1% per-call transient rate, the workload has ~25% failure odds per run, dropping to ~6% with nick-fields/retry@v4 max_attempts=2. That is "blocker every other busy day," not "reliable."

Solution — industry-standard two-tier eval pattern

• PR + push-to-main runs only uv run pytest -q, which already includes the Epic 8 fake-LLM coverage of the deep_clean LangGraph scenario plus full branch coverage for split, PO checkpoint, story review, and error paths via direct mocks. Deterministic, free, <90s wall-clock, 100% reliable. Catches structural regressions: graph wiring, schema changes, prompt template breakage, contract drift.
• Nightly cron runs the existing evals/run_evals.py against real Gemini Flash with the existing retry classifier, then the existing check_thresholds.py + check_budgets.py gates. Catches prompt-quality drift, model behavior changes, real cost/latency regressions. Failures surface via GitHub Actions email — within 24h, not in the middle of a PR review.

This matches the eval pattern documented in Anthropic's own evals workflow, LangSmith experiment runners, and OpenAI's evals framework: PRs gate on deterministic offline checks; live-model quality regressions run on a schedule with alerting. None of those gate PRs on a live third-party model's hourly uptime — and after today, neither will we.

Honesty note: we considered keeping evals/run_evals.py on the PR tier under fake-LLM, but the fake provider returns canned JSON keyed by scenario (not by sample), so all 3 golden samples would produce identical outputs. That would be dressing up a smoke test as something it isn't. Pytest is what's actually being checked — so pytest is what we run.

Phased delivery

• PR 1 (this issue) — delete the regression job from .github/workflows/ci.yml. Stops the bleeding today.
• PR 2 (follow-up) — create .github/workflows/nightly-eval.yml modeled on nightly-safety-regression.yml, running the existing live eval on 03:00 UTC cron with the existing threshold/budget gates. Will be tracked in a separate issue after PR 1 merges, per the user's stop-gate request.

Plan

/Users/auzokwe/.claude/plans/so-you-mean-we-parallel-nova.md

[augustineuzokwe]

Reopening — PR #333 deleted the regression job; per follow-up user feedback, we want to preserve the YAML and disable it via if: false instead, so the configuration stays as a flip-switch fallback while the nightly cron is being built. Follow-up PR opening now.