bench a provider when the provider is what failed a CoS agent run - #3627
Merged
Conversation
An agy TUI agent blocked on Google's "We're finishing verifying your account eligibility" banner failed fast (good) but left the provider marked available, so the next dequeued task picked Antigravity again and died identically — a queue of tasks could burn down in under a minute. Worse, the banner was classified actionable, which BLOCKED each task over a condition whose own message says it clears itself. Two changes: - The eligibility signal is now non-actionable, so resolveFailedTaskDecision takes the retry path instead of blocking the task, and carries origin:'provider' (it is provider chrome, not text an agent could print). - agentFinalization benches the provider on ANY provider-origin failure, not just usage/rate limits. The provenance gate (#2642) is what makes this safe: a loose keyword match on a repainted TUI transcript stays 'output-scan' and never benches. The cooldown policy moves to server/lib/providerCooldown.js, shared with promptRunner.js — the two paths previously disagreed on how long the same category was worth (auth-error: 15m via the prompt cascade, never via the agent path).
The bench predicate kept a `|| category === 'usage-limit' || category === 'rate-limit'` fallback alongside the new `origin === 'provider'` check — preserving the old behavior, but contradicting the rule the surrounding comment states. Both categories have loose pattern alternatives (a bare "rate limit" / "quota exceeded"), so an agent's own failing test printing that phrase would bench a healthy provider for 5 minutes off nothing but its transcript. Provenance is now the whole gate. Genuine limits already carry origin:'provider' via their structuredMarker, and every finalizeAgent caller analyzes through analyzeAgentFailure or detectImmediateFallbackSignal — both of which stamp an origin on every branch — so no real provider failure stops benching. The bench test now covers output-scan rate-limit and usage-limit, which is what let this through: it only exercised auth-error. Verified by planting the old predicate back and watching both new cases fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agyTUI agents were failing in bursts with the Antigravity banner "We're finishing verifying your account eligibility. This usually takes a moment. Please try again shortly."Detection for that banner already existed (shipped 2026-07-27) and correctly failed the run in ~3s instead of waiting out the 180s idle reaper. Two things downstream of it were wrong:
actionable, soresolveFailedTaskDecisionblocked the task outright — over a condition whose own message says it clears itself.agentFinalizationonly marked a provider unavailable forusage-limit/rate-limit, so Antigravity stayed "available" and the next dequeued task picked it again and died identically. A queue of tasks could burn down in under a minute with every task parked.Confirmed against two real runs on this install (
agent-70d755ef,agent-ca367708): bothsuccess: false,category: auth-error,requiresFallback: true, ~3s duration, provider never benched.Changes
server/lib/providerCooldown.js(new) — the cooldown policy (COOLDOWN_MS_BY_CATEGORY, the request-specific never-bench rule,resolveProviderBench) extracted out ofpromptRunner.js, which was the only owner. The prompt cascade and the agent path previously disagreed about the same input:auth-errorbenched 15m via the prompt path and never via the agent path.agentFinalization.js— benches on any provider-origin failure, resolving the window through the shared policy. The gate isorigin === 'provider'(the provenance flag from Require provider/runner provenance before excluding environmental failures from task learning #2642), so a loose keyword match on a repainted TUI transcript — which is a whole session of text the agent itself wrote — can never bench a provider.errorDetection.js— the eligibility signal isactionable: false(task retries instead of blocking) andorigin: 'provider'(it is provider chrome). Signals stay actionable-by-default; only one that says it self-clears opts out. NobenchMson the payload — the vendored toolkit stays a classifier and the host owns remediation.finalizeHelpers.js— hoisted theimmediateFallbackAnalysisshort-circuit above a ~16KB spool read whose result was discarded. This path fires more often now.Behavior preserved:
markUsageLimitkeeps its dedicated marker (it parses its own "resets 5pm" window); request-specific failures (bad model id, content refusal, off-shape response) still never bench.Test plan
npm test --prefix server— 1245 files / 25664 tests pass.server/lib/providerCooldown.test.jscovers the policy directly (skip/usage-limit/per-category/default/missing-category).server/services/agentFinalization.providerBench.test.jsdrivesfinalizeAgentand asserts what gets benched. It imports the real detector and the real cooldown table so the assertions can't drift from what ships.agytranscript on this install (the banner renders as two lines with a leading space from the erase-to-start sequence — the fixture inerrorDetection.test.jsis that shape, not idealized text).output-scancases fail, then they pass again once restored.Review
Reviewed by
agy, which caught that the first cut kept a|| category === 'usage-limit' || category === 'rate-limit'fallback next to the new provenance check — preserving old behavior but contradicting the rule the code stated, and leaving a healthy provider benchable off an agent's own test output printing "rate limit". Fixed in d8c2f40, with the test gap that hid it closed.