Skip to content

bench a provider when the provider is what failed a CoS agent run - #3627

Merged
atomantic merged 2 commits into
mainfrom
cos/task-mskggd8k/agent-36f09853
Aug 8, 2026
Merged

bench a provider when the provider is what failed a CoS agent run#3627
atomantic merged 2 commits into
mainfrom
cos/task-mskggd8k/agent-36f09853

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

agy TUI agents were failing in bursts with the Antigravity banner "We're finishing verifying your account eligibility. This usually takes a moment. Please try again shortly."

Detection for that banner already existed (shipped 2026-07-27) and correctly failed the run in ~3s instead of waiting out the 180s idle reaper. Two things downstream of it were wrong:

  1. The banner was classified actionable, so resolveFailedTaskDecision blocked the task outright — over a condition whose own message says it clears itself.
  2. Nothing benched the provider. agentFinalization only marked a provider unavailable for usage-limit/rate-limit, so Antigravity stayed "available" and the next dequeued task picked it again and died identically. A queue of tasks could burn down in under a minute with every task parked.

Confirmed against two real runs on this install (agent-70d755ef, agent-ca367708): both success: false, category: auth-error, requiresFallback: true, ~3s duration, provider never benched.

Changes

  • server/lib/providerCooldown.js (new) — the cooldown policy (COOLDOWN_MS_BY_CATEGORY, the request-specific never-bench rule, resolveProviderBench) extracted out of promptRunner.js, which was the only owner. The prompt cascade and the agent path previously disagreed about the same input: auth-error benched 15m via the prompt path and never via the agent path.
  • agentFinalization.js — benches on any provider-origin failure, resolving the window through the shared policy. The gate is origin === 'provider' (the provenance flag from Require provider/runner provenance before excluding environmental failures from task learning #2642), so a loose keyword match on a repainted TUI transcript — which is a whole session of text the agent itself wrote — can never bench a provider.
  • errorDetection.js — the eligibility signal is actionable: false (task retries instead of blocking) and origin: 'provider' (it is provider chrome). Signals stay actionable-by-default; only one that says it self-clears opts out. No benchMs on the payload — the vendored toolkit stays a classifier and the host owns remediation.
  • finalizeHelpers.js — hoisted the immediateFallbackAnalysis short-circuit above a ~16KB spool read whose result was discarded. This path fires more often now.

Behavior preserved: markUsageLimit keeps its dedicated marker (it parses its own "resets 5pm" window); request-specific failures (bad model id, content refusal, off-shape response) still never bench.

Test plan

  • npm test --prefix server1245 files / 25664 tests pass.
  • New server/lib/providerCooldown.test.js covers the policy directly (skip/usage-limit/per-category/default/missing-category).
  • New server/services/agentFinalization.providerBench.test.js drives finalizeAgent and asserts what gets benched. It imports the real detector and the real cooldown table so the assertions can't drift from what ships.
  • Detection verified against the actual ANSI-stripped bytes from a real failing agy transcript on this install (the banner renders as two lines with a leading space from the erase-to-start sequence — the fixture in errorDetection.test.js is that shape, not idealized text).
  • The provenance gate is pinned by a bypass probe: planting the old category-based predicate back makes the two output-scan cases fail, then they pass again once restored.

Review

Reviewed by agy, which caught that the first cut kept a || category === 'usage-limit' || category === 'rate-limit' fallback next to the new provenance check — preserving old behavior but contradicting the rule the code stated, and leaving a healthy provider benchable off an agent's own test output printing "rate limit". Fixed in d8c2f40, with the test gap that hid it closed.

An agy TUI agent blocked on Google's "We're finishing verifying your
account eligibility" banner failed fast (good) but left the provider
marked available, so the next dequeued task picked Antigravity again and
died identically — a queue of tasks could burn down in under a minute.
Worse, the banner was classified actionable, which BLOCKED each task over
a condition whose own message says it clears itself.

Two changes:

- The eligibility signal is now non-actionable, so resolveFailedTaskDecision
  takes the retry path instead of blocking the task, and carries
  origin:'provider' (it is provider chrome, not text an agent could print).
- agentFinalization benches the provider on ANY provider-origin failure,
  not just usage/rate limits. The provenance gate (#2642) is what makes
  this safe: a loose keyword match on a repainted TUI transcript stays
  'output-scan' and never benches.

The cooldown policy moves to server/lib/providerCooldown.js, shared with
promptRunner.js — the two paths previously disagreed on how long the same
category was worth (auth-error: 15m via the prompt cascade, never via the
agent path).
The bench predicate kept a `|| category === 'usage-limit' || category ===
'rate-limit'` fallback alongside the new `origin === 'provider'` check —
preserving the old behavior, but contradicting the rule the surrounding
comment states. Both categories have loose pattern alternatives (a bare
"rate limit" / "quota exceeded"), so an agent's own failing test printing
that phrase would bench a healthy provider for 5 minutes off nothing but
its transcript.

Provenance is now the whole gate. Genuine limits already carry
origin:'provider' via their structuredMarker, and every finalizeAgent
caller analyzes through analyzeAgentFailure or detectImmediateFallbackSignal
— both of which stamp an origin on every branch — so no real provider
failure stops benching.

The bench test now covers output-scan rate-limit and usage-limit, which is
what let this through: it only exercised auth-error. Verified by planting
the old predicate back and watching both new cases fail.
@atomantic
atomantic merged commit cc2e995 into main Aug 8, 2026
6 checks passed
@atomantic
atomantic deleted the cos/task-mskggd8k/agent-36f09853 branch August 8, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant