Repository navigation
[codex] v0.11.1 chess benchmark hotfix - #23
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1b807f376f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let quality_failures = self.quality_gate_failures(&plan, &runtimes); | ||
| if !quality_failures.is_empty() | ||
| && quality_repair_attempts < MAX_QUALITY_REPAIR_ATTEMPTS | ||
| && should_auto_repair_quality(&plan, tokens_used) |
There was a problem hiding this comment.
Gate auto-repair on non-duplicated token totals
The new should_auto_repair_quality(&plan, tokens_used) check uses the scheduler’s aggregate tokens_used, but that counter is not a pure provider-total: SubtaskProgress sets rt.tokens_used = tokens_so_far during execution and handle_completion later adds the final worker_tokens again, so runs with progress updates can be double-counted. With this change, a chess attempt that actually uses under 8k provider tokens can appear above 8k and incorrectly skip automatic quality repair, turning recoverable quality failures into hard task failures; the ceiling check should be based on non-duplicated token usage accounting.
Useful? React with 👍 / 👎.
Summary
Validation