Derive openai-codex models from OpenAI catalog, 272k context, opt-in live tests

[m-aebrer]

Closes #538

Derive the openai-codex model list from the OpenAI catalog (adds GPT-6 Sol/Luna, 6.1 Sol, Daybreak), use a 272k context window for every Codex model, and make live provider tests opt-in so npm test never spends real quota by default.

Implementation plan posted as a comment below.

[m-aebrer]

Implementation Plan

Analysis

1. Codex catalog — openai-codex was a hand-typed list in packages/ai/scripts/generate-models.ts. models.dev doesn't track the Codex surface, but it does track the OpenAI Platform catalog, which Codex mirrors. Deriving from it removes the manual step. Context: Codex's default window is 272k (prompts above that are billed at 2x); Pi uses 272k for all Codex models. Users can raise it with modelOverrides.
2. Live tests — every live test is gated by DREB_SKIP_LIVE_API === "1" || !credential, so plain npm test / bash test.sh spend real money on every provider the developer is logged into (including context-overflow.test.ts, which sends more than a full context window). Only the pre-commit hook passes --no-live-api.

Deliverables

Stage 1 — Codex catalog and Copilot test fixes (already implemented locally)

• generate-models.ts: derive Codex models from openai entries (GPT >= 5.5, named variants, gpt-daybreak-*-latest, plus the explicit extra gpt-5.4-mini; exclude -pro/-mini/-nano/-chat/-codex and bare family ids when variants exist). Copy pricing, context min(catalog, 272k), max output 128k. Throw if expected models go missing.
• src/utils/overflow.ts: recognise Copilot 413 Request Entity Too Large as overflow.
• test/fixtures/copilot-models.ts: prefer broadly-entitled Copilot models.
• test/github-copilot-anthropic.test.ts: interleaved-thinking test uses a non-adaptive model.
• Tests: openai-codex-registry.test.ts (membership, exclusions, pricing parity, 272k), supports-xhigh.test.ts, overflow-detection.test.ts, context-overflow.test.ts.
• Docs: root README, coding-agent README, docs/providers.md.

Stage 2 — Opt-in live tests

• Replace DREB_SKIP_LIVE_API with opt-in DREB_LIVE_API=1 (removed entirely, no compatibility shim).
• Shared isLiveApiEnabled() helper in each package's test utilities; every live skipIf becomes !isLiveApiEnabled() || ... across about 36 files in ai, agent, coding-agent.
• resolveApiKey / hasAuthForProvider (in packages/ai/test/oauth.ts, packages/coding-agent/test/utilities.ts) don't read or refresh auth.json unless opted in.
• Bedrock shouldRunBedrockExtensiveTests requires opt-in.
• Expensive tier: context-overflow.test.ts also requires DREB_LIVE_API_EXPENSIVE=1.
• test.sh: default offline; --live-api opts in (remove --no-live-api). .husky/pre-commit: drop the flag.

Acceptance Criteria

See the issue. Key checks: the new GPT-6/Daybreak models appear on openai-codex with no hand-typed entries; every Codex model is 272k; Codex pricing matches OpenAI; npm test with a populated auth.json makes zero provider calls; live tests run only with DREB_LIVE_API=1; overflow tests also need the expensive flag.

Files

• packages/ai/scripts/generate-models.ts, packages/ai/src/utils/overflow.ts
• packages/ai/test/oauth.ts, packages/coding-agent/test/utilities.ts, packages/{ai,agent}/test/bedrock-utils.ts
• All test files containing DREB_SKIP_LIVE_API
• test.sh, .husky/pre-commit
• README.md, packages/coding-agent/README.md, packages/coding-agent/docs/providers.md, packages/coding-agent/docs/development.md, AGENTS.md

Testing Approach

• Registry tests for the derived Codex catalog (membership, exclusions, pricing parity, context).
• Update oauth-no-live.test.ts / utilities-no-live.test.ts / bedrock-utils.test.ts: the default (unset) makes no credential reads; DREB_LIVE_API=1 restores loading.
• New guard test: scan test sources and fail if a file references provider credentials (resolveApiKey, *_API_KEY, *_OAUTH_TOKEN) in a skipIf/runIf without the opt-in helper.
• npm run build + npm test (offline by default).

Risks

• Tests reading process.env.*_API_KEY directly could bypass the helper; the guard test covers this.
• Codex models not enabled on a user's plan still appear in the list and fail at the server with a clear error; this is documented.
• The generator depends on models.dev naming; the loud missing-model check catches drift.

---

Plan created by mach6

[github-actions]

Vitest coverage

Metric	Covered	Total	Coverage
Statements	42732	58460	73.09%
Branches	23352	37076	62.98%
Functions	9222	12370	74.55%
Lines	30757	41952	73.31%

View full coverage run

[m-aebrer]

Progress Update

Both plan stages are implemented.

Stage 1 — Codex catalog

• openai-codex is now derived from the OpenAI catalog in generate-models.ts: it adds gpt-6-sol, gpt-6-luna, gpt-6.1-sol, gpt-daybreak-blue-latest and gpt-daybreak-red-latest, excludes -pro/-mini/-nano/-chat/-codex variants and bare family ids, and keeps gpt-5.4-mini as an explicit extra.
• Pricing is copied from OpenAI. Every Codex model has a 272k context window (the Codex default; above 272k is billed at 2x) and can be raised with modelOverrides. Generation throws if expected models go missing.
• Copilot live-test fixes: entitlement-safe model selection, 413 Request Entity Too Large recognised as overflow (unit tested), interleaved-thinking test uses a non-adaptive model.

Stage 2 — Opt-in live tests

• DREB_SKIP_LIVE_API removed. Live tests run only with DREB_LIVE_API=1; context-overflow tests also need DREB_LIVE_API_EXPENSIVE=1.
• Shared isLiveApiEnabled() / isExpensiveLiveApiEnabled() in each package's test/live-api.ts, used across about 30 test files, the credential helpers (no auth.json reads unless opted in) and the Bedrock checks.
• test.sh is offline by default; --live-api opts in, an inherited opt-in is cleared, and unknown arguments are rejected. The pre-commit hook now runs plain bash test.sh.
• New live-api-gating.test.ts fails if any credential-gated skipIf/runIf bypasses the helper or uses the removed flag (verified with a deliberately unguarded probe).
• Docs: root README, coding-agent README, docs/providers.md, docs/development.md, AGENTS.md.

Verification: build, tsgo --noEmit, biome and bash test.sh pass (6326 passed, 0 failed, 741 skipped). No live provider calls were made. The large line count is mostly biome re-wrapping the shortened skipIf conditions (the whitespace-ignoring diff is about +716/−1105).

Commit: 7de9b99

---

Progress tracked by mach6

[m-aebrer]

Unverified Review Candidates — Pending Assessment

Review round: 1
Reviewed commit: 7de9b99

These are unverified candidates. Severity reflects reviewer confidence; do not treat any item as a merge blocker until the assessment comment is posted.

Critical

• Finding 1 (error-auditor, 98) — generate-models.ts terminal .catch(console.error) swallows the new expected-model check failure; exit code 0, build may reuse stale models.generated.ts.

Important

• Finding 2 (code-reviewer, 91) — hasNamedVariants counts dated snapshots (e.g. gpt-5.5-2026-10-01) as named variants, suppressing the bare family ID and failing/altering generation.
• Finding 3 (error-auditor, 97) — Expected-model check covers only four IDs; new GPT-6/Daybreak models could silently disappear.
• Finding 4 (error-auditor, 94) — Math.min() with models.dev 4096 default limits can shrink Codex context/output silently when upstream metadata is missing.
• Finding 5 (error-auditor, 88) — overflow.ts 413 pattern treats any request-body-too-large (e.g. big image via proxy) as context overflow → futile compaction.
• Finding 6 (test-reviewer, 99) — openai-codex-registry.test.ts asserts an exact 11-ID list, so a correctly auto-added new model breaks the test.
• Finding 7 (test-reviewer, 96) — Codex derivation not tested against synthetic catalog input (new model inclusion, excluded variant, missing expected ID throws).
• Finding 8 (test-reviewer, 97) — live-api-gating.test.ts accepts isLiveApiEnabled() for context-overflow tests; expensive gate could regress unnoticed.
• Finding 9 (test-reviewer, 94) — Gating scan only inspects skipIf/runIf; unguarded tests or module-init calls would pass.
• Finding 10 (test-reviewer, 95) — test.sh opt-in env handling has no shell-level test.

Suggestions

• Finding 11 (simplifier, 88) — live-api.ts duplicated verbatim across agent, ai, coding-agent test dirs.

Strengths

• Completeness-checker confirmed every acceptance criterion met; no residual DREB_SKIP_LIVE_API; build passes; docs updated across root README, coding-agent README, providers.md, development.md, AGENTS.md.

Agents run: code-reviewer, error-auditor, test-reviewer, completeness-checker, simplifier

---

Reviewed by mach6

[m-aebrer]

Review Assessment

#539 (comment)

Classifications

Finding	Classification	Reasoning
1	merge blocker	Factual: generateModels().catch(console.error) exits 0; the new throw precedes writeFileSync, and the ai build then compiles the stale gitignored models.generated.ts. Scope: defeats the issue's "generation fails loudly". Practical: local/release builds go green with wrong models.
2	useful follow-up	Factual: a dated snapshot like gpt-5.5-YYYY-MM-DD would suppress the bare ID. Scope: PR logic. Practical: no such snapshots for gpt-5.x+ today; latent. Cheap to bundle.
3	merge blocker	Factual: expected-ID check covers only gpt-5.4-mini, gpt-5.5, gpt-5.6-luna, gpt-6-astra. Scope: issue requires loud failure when expected Codex models disappear. Practical: the PR's headline models (GPT-6 Sol/Luna, 6.1 Sol, Daybreak previews) can vanish silently.
4	useful follow-up	Factual: 4096 default flows through Math.min. Scope/Practical: all current models derive 272k/128k; only triggers on an upstream metadata gap.
5	false positive	413 → overflow is explicitly mandated by the issue, anchored, and unit-tested.
6	merge blocker	Factual: exact 11-ID toEqual. Generated file is rebuilt from live models.dev in CI. Scope: contradicts "new model appears after rebuild with no code change". Practical: next upstream frontier addition breaks CI on every PR.
7	useful follow-up	Synthetic-catalog tests need generator refactor; not an acceptance criterion.
8	useful follow-up	Guard hole for expensive tier; current overflow tests correctly use the expensive gate.
9	nitpick	Credential helpers independently refuse without opt-in; offline CI green.
10	nitpick	Flag semantics are unit-tested; shell wrapper is thin.
11	nitpick	16-line env helper; duplication matches existing bedrock-utils.ts pattern.

Action Plan

1. Finding 1 — set process.exitCode = 1 (or rethrow) in the generator's terminal catch.
2. Finding 3 — extend the expected-model list to all 11 derived Codex IDs.
3. Finding 6 — replace exact-list assertion with subset membership; keep exclusion/pricing/272k pins.

---

Assessment by mach6

[m-aebrer]

Progress Update

Fixed review findings 1, 2, 3, 4, 6, 7, 8:

• finding 1: generator sets non-zero exit code on failure, so the build can no longer compile a stale models.generated.ts.
• findings 2/3/4/7: Codex derivation extracted to packages/ai/scripts/codex-models.ts. Expected-ID guard covers all 11 Codex models; bare-family suppression counts only Codex-eligible variants (dated snapshots excluded); missing/low catalog limits now throw instead of shrinking context. New codex-models-derivation.test.ts tests synthetic catalogs.
• finding 6: registry test asserts expected-subset membership, so new upstream models no longer break CI.
• finding 8: gating test requires the expensive opt-in on every context-overflow block.
• providers.md updated.

Build, npm run check, full npm test pass.

Commit: ed3968e

---

Progress tracked by mach6

[m-aebrer]

Unverified Review Candidates — Pending Assessment

Review round: 2
Reviewed commit: ed3968e

These are unverified candidates. Severity reflects reviewer confidence; do not treat any item as a merge blocker until the assessment comment is posted.

Critical

(none)

Important

• Finding 1 (error-auditor, 94) — codex-models.ts copies prices unvalidated; the OpenAI loader substitutes 0 for absent prices, so a new frontier model with missing pricing becomes a zero-cost Codex model and the parity test still passes.
• Finding 2 (error-auditor, 96) — oauth.ts / coding-agent utilities.ts: with DREB_LIVE_API=1, an unreadable/malformed auth.json is treated as empty, so OAuth suites skip silently (fallback predates PR).
• Finding 3 (error-auditor, 97) — oauth.ts resolveApiKey() catches refresh failures indiscriminately → opted-in suites skip instead of failing.
• Finding 4 (code-reviewer, 90) — maxTokens < 128k check throws for a valid lower output limit, blocking the entire build when a legit new model has e.g. 64k output.
• Finding 5 (code-reviewer, 85) — Derived models are hard-coded reasoning: true, input: ["text","image"] rather than copied from the source entry.

Suggestions

• Finding 6 (simplifier, 85) — Duplicated "collect ungated conditions" logic in live-api-gating.test.ts; extract a helper.

Strengths

• Round-1 blockers verified fixed: generator sets non-zero exit code; expected-ID guard covers all 11 models; registry test uses subset membership.
• Completeness-checker: every acceptance criterion met; no residual DREB_SKIP_LIVE_API.
• Test-reviewer: derivation tests on synthetic catalogs are thorough; no gaps ≥80.

Agents run: code-reviewer, error-auditor, test-reviewer, completeness-checker, simplifier

---

Reviewed by mach6

[m-aebrer]

Review Assessment

#539 (comment)

Classifications

Finding	Classification	Reasoning
1	useful follow-up	Factual: cost: { ...m.cost } unvalidated; loader substitutes 0; parity test passes on zeros. Scope: "pricing equals catalog" holds literally; all 11 current models priced. Practical: only on a future unpriced upstream entry; cheap guard mirroring the limits throw.
2	deferred	Factual: confirmed. Scope: fallback verbatim on merge base; PR only narrows reach to opt-in runs. Practical: needs opt-in + corrupt auth.json; skips visible in vitest summary.
3	deferred	Factual: confirmed. Scope: pre-existing catch, untouched. Practical: needs opt-in + failed refresh; skips visible.
4	nitpick	Factual: throws on any sub-128k output. Scope: this is the intended round-1 loud-failure fix. Practical: loud build failure with actual numbers forces a human decision; silent alternative is worse.
5	nitpick	Factual: hardcoded, matches pre-PR behavior; all 11 models are reasoning+image in catalog. Practical: speculative.
6	nitpick	~4 shared lines across two distinct assertions.

Action Plan

No merge blockers.

---

Assessment by mach6