Skip to content

fix(api): validate GPT-6 Astra reasoning capabilities - #449

Closed
oraclexing wants to merge 3 commits into
steipete:mainfrom
oraclexing:fix/astra-api-capabilities
Closed

fix(api): validate GPT-6 Astra reasoning capabilities#449
oraclexing wants to merge 3 commits into
steipete:mainfrom
oraclexing:fix/astra-api-capabilities

Conversation

@oraclexing

@oraclexing oraclexing commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Explicit Astra API reasoning options currently fail Oracle's GPT-5.6-only validation. Register the exact gpt-6-astra API model and validate its effective reasoning effort against a separate capability set: low, medium, high, xhigh, and max are accepted; none is rejected before inference, including when supplied by a model override. An explicit supported effort can replace an invalid configured default. GPT-5.6 retains its existing none support.

Responses standard/pro mode retains the existing route checks and Pro lifecycle defaults. Astra uses web_search and a conservative 272K default input budget at the standard-rate boundary; the documentation distinguishes that budget from maximum context and explains cost-estimate limits.

Browser boundary

This remains API-focused and does not incorporate #448's browser implementation. Until that browser support lands, explicit Astra selection fails before browser launch. Existing current and ignore strategies remain usable without claiming the active ChatGPT model is Astra. No installed MCP deployment or default model is changed.

Current validation

Updated onto main fe2c8d9905624c71a406eb006bc1ab7cc3735744 (0.19.0) at head 9434f68dd8a1d10b50ad713621beec6eec09f796. The merge resolves the changelog conflict while preserving the released main entries and placing this feature under Unreleased.

  • pnpm run check and pnpm run build: passed on Windows / Node 24.11.1.
  • pnpm vitest run --maxWorkers 4: 2,240 passed, 53 skipped; 176 test files passed, 18 skipped.
  • API request coverage exercises supported effort, Pro/max, invalid explicit/configured effort, override precedence, proxy rejection, and Sol regression behavior. MCP tests retain saved-configuration and explicit browser-strategy coverage after the upstream SDK v2 integration.
  • Built CLI API dry-run resolves gpt-6-astra with Pro/max. Browser dry-runs succeed for current and ignore; unsupported select exits 1 before browser launch.
  • Built first-party route diagnostic reports api.openai.com, missing OPENAI_API_KEY, and no ready inference route. No real API request was sent.

These are local tests and dry-runs. They do not prove live Astra inference, Pro completion, or current ChatGPT account availability.

Remaining proof gate

Real Responses/Pro completion remains outstanding because no API credential is available in the validation environment. Browser subscription runs are not API proof. Maintainers or a credentialed contributor still need to supply real inference evidence and fresh/existing-configuration runtime coverage before treating that gate as satisfied.

Official sources were successfully retrieved again on September 8, 2026: the Astra model page confirms the supported efforts, $10/$50 per-million input/output token rates, and higher pricing above 272K input tokens; the model guide confirms no none effort and continued pro-mode support. This resolves the documentation-access uncertainty from the previous review, not the live-inference gap.

@clawsweeper

clawsweeper Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 5, 2026
@clawsweeper

clawsweeper Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed September 7, 2026, 7:57 PM ET / 23:57 UTC.

ClawSweeper review

What this changes

Adds GPT-6 Astra API configuration and reasoning validation, updates help and documentation, and rejects unsupported browser selection while preserving current/ignore strategies.

Merge readiness

Blocked before merge - 3 items remain

Keep open: the API support remains useful and is not implemented on main. The earlier browser-strategy finding is resolved, and official documentation supports the model contract, but real Responses/Pro completion remains unproven. Repository policy also prohibits automatic closure.

Priority: P2
Reviewed head: 9434f68dd8a1d10b50ad713621beec6eec09f796

Review scores

Measure Result What it means
Overall readiness 🦪 silver shellfish (2/6) The implementation is focused and the earlier defect is resolved, but the central runtime behavior lacks real proof.
Proof confidence 🦪 silver shellfish (2/6) Needs stronger real behavior proof before merge: The captured Windows/Node tests and CLI dry-runs exercise configuration and validation, but explicitly send no request through the changed API runner to OpenAI. They therefore do not establish Astra Responses acceptance or Pro completion, and the prior fresh/existing-configuration proof request remains unmet. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Needs proof Needs stronger real behavior proof before merge: The captured Windows/Node tests and CLI dry-runs exercise configuration and validation, but explicitly send no request through the changed API runner to OpenAI. They therefore do not establish Astra Responses acceptance or Pro completion, and the prior fresh/existing-configuration proof request remains unmet. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed 8 items Repository policy and scope: Read the complete root AGENTS.md; no nested AGENTS.md files were found and .agents/maintainer-notes is absent. Applied the opt-in live-test and user-visible changelog guidance. The supplied repository profile prohibits automatic closure.
Main and release still lack the requested support: Main restricts explicit reasoning options to GPT-5.6 models. The inspected v0.19.0 model configuration has no Astra entry; the branch adds distinct capability validation rather than duplicating a supported configuration path.
Effective-effort validation and dispatch: The runner validates explicit options and resolved model defaults before client dispatch. Request construction preserves explicit-effort precedence, and existing Responses routing and Pro lifecycle handling are reused.
Findings None None.
Security None None.

How this fits together

Oracle turns CLI or MCP requests into model consultations. API requests pass through model configuration and reasoning validation before provider dispatch; browser requests use a separate model-selection path.

flowchart TD
  A[CLI or MCP request] --> B{API or browser}
  B -->|API| C[Model configuration and overrides]
  C --> D[Validate reasoning options]
  D --> E[Responses request and completion]
  B -->|Browser| F{Selection strategy}
  F -->|Select Astra| G[Explain unsupported selection]
  F -->|Current or ignore| H[Keep active ChatGPT model]
Loading

Before merge

  • Add real behavior proof - Needs stronger real behavior proof before merge: The captured Windows/Node tests and CLI dry-runs exercise configuration and validation, but explicitly send no request through the changed API runner to OpenAI. They therefore do not establish Astra Responses acceptance or Pro completion, and the prior fresh/existing-configuration proof request remains unmet. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • Resolve merge risk (P1) - Real Astra request acceptance, Pro completion, and fresh/existing-configuration runtime behavior remain unverified.
  • Complete next step (P2) - Provide after-fix real Astra Responses/Pro completion and unsupported-effort rejection evidence, covering fresh and existing configuration. Terminal screenshots or recordings are welcome; copied output and logs also count. Redact API keys, private endpoints, IP addresses, and other private information. Update the PR body to trigger re-review; if that does not happen, ask a maintainer to comment @clawsweeper re-review.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test growth production +80/-13; tests +221/-0 Production growth implements model registration and validation, with most added lines devoted to focused regression coverage.

Merge-risk options

Maintainer options:

  1. Decide the mitigation before merge
    Keep Astra capability checks within the existing API pipeline, preserve browser strategy preferences, and establish real completion and configuration compatibility before landing.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Technical review

Best possible solution:

Keep Astra capability checks within the existing API pipeline, preserve browser strategy preferences, and establish real completion and configuration compatibility before landing.

Do we have a high-confidence way to reproduce the issue?

Not applicable as an established-behavior bug: this adds model-specific support. Source inspection confirms main rejects explicit Astra reasoning options; no live inference was attempted.

Is this the best way to solve the issue?

Yes, the implementation extends the existing model registry and validation path without adding a competing provider adapter; its runtime claims still need real evidence.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against fe2c8d990562.

Labels

Label justifications:

  • P2: This is a bounded model-support improvement without evidence of an urgent regression in established workflows.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The captured Windows/Node tests and CLI dry-runs exercise configuration and validation, but explicitly send no request through the changed API runner to OpenAI. They therefore do not establish Astra Responses acceptance or Pro completion, and the prior fresh/existing-configuration proof request remains unmet. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

What I checked:

  • Repository policy and scope: Read the complete root AGENTS.md; no nested AGENTS.md files were found and .agents/maintainer-notes is absent. Applied the opt-in live-test and user-visible changelog guidance. The supplied repository profile prohibits automatic closure. (AGENTS.md, 9434f68dd8a1)
  • Main and release still lack the requested support: Main restricts explicit reasoning options to GPT-5.6 models. The inspected v0.19.0 model configuration has no Astra entry; the branch adds distinct capability validation rather than duplicating a supported configuration path. (src/oracle/run.ts:114, fe2c8d990562)
  • Effective-effort validation and dispatch: The runner validates explicit options and resolved model defaults before client dispatch. Request construction preserves explicit-effort precedence, and existing Responses routing and Pro lifecycle handling are reused. (src/oracle/run.ts:294, 9434f68dd8a1)
  • Earlier browser finding resolved: The Astra guard returns immediately for current/ignore. MCP resolves the explicit or saved strategy before invoking it, and the added tests cover both sources. The previous review's request for real completion and configuration evidence remains outstanding. (src/cli/browserConfig.ts:341, 9434f68dd8a1)
  • Official provider contract verified: Opened the official Astra model documentation and model guide. They support the five accepted effort levels, exclusion of none, Pro mode, web search, and the documented pricing boundary. This contract is directly relevant to the model configuration and validation added by the patch.
  • Captured proof remains incomplete: The complete supplied body at sourceRevision 2350045f789e56afbb32cebe76f2153851793298cd0bb5e92be69cf975aeec37 reports Windows/Node tests and built CLI dry-runs, explicitly states no API request was sent, and identifies real Responses/Pro completion plus fresh/existing-configuration runtime coverage as outstanding. No target code was executed during this review. (9434f68dd8a1)

Likely related people:

  • Paul Bohm: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Lu Wang: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Add redacted after-fix Responses Pro completion and unsupported-effort rejection evidence, including fresh and existing configuration coverage.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)
  • reviewed 2026-09-05T12:37:55.264Z sha 3b2b947 :: needs real behavior proof before merge. :: [P1] [P1] Preserve current/ignore browser strategies when rejecting Astra
  • reviewed 2026-09-05T13:04:59.198Z sha 0ecfc00 :: needs real behavior proof before merge. :: none

@clawsweeper clawsweeper Bot removed the merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. label Sep 7, 2026
steipete added a commit to FNDEVVE/oracle that referenced this pull request Sep 8, 2026
Preserve browser-only Pro aliases through CLI engine discovery, retain model-specific API capabilities, and verify localized Latest and effort controls. Combine the compatible work from steipete#448, steipete#449, and steipete#465; release notes are deferred to the final notes PR.
@steipete

steipete commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Landed through #465, which consolidates this change with the related model work and keeps your contributor credit in the squash commit and changelog. Thank you.

steipete pushed a commit that referenced this pull request Sep 8, 2026
Consolidate API capability validation and Latest browser selection from #449 and #448. Validate model-specific reasoning efforts, reject invalid saved defaults, recognize localized picker controls, and preserve browser-only aliases during engine discovery. Keep the default model unchanged.

Real standard/Pro API calls, Latest/High and Latest/Pro browser requests, and a controlled Sol-to-Latest transition passed. Thanks @kiyo-e for the browser work incorporated into this PR.

Co-authored-by: FND <fndevve@proton.me>
Co-authored-by: oraclexing <xing_z@hotmail.com>
Co-authored-by: nummy <6960749+malvarezcastillo@users.noreply.github.com>
@steipete steipete closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants