Repository navigation
feat(ai): fall back to Gemini when a provider runs out of budget - #25
Merged
Merged
Conversation
Not OpenAI, which was the ask: its key is exhausted too. Both keys in the vault are dead — Anthropic capped until 2026-09-01, OpenAI at `429 insufficient_quota, no credits remaining` — so falling back there would have swapped one dead vendor for another. Gemini answers, and is what this uses. `FallbackModel` sits behind the existing `TextModel` interface, so the composer, the profile drafter and the site extractor are untouched and the deterministic quality gates still run over whatever comes back. One package still talks to a model; only which service answers changes. The whole design rests on one distinction: a provider being out of budget is worth retrying elsewhere, and almost nothing else is. A refusal is a judgement the next vendor will likely repeat — shopping it around until someone agrees is precisely the behaviour this product should not have. A malformed request is our bug and fails identically downstream. A 500 is transient and belongs to whatever retries the job. Detecting exhaustion needs message matching, not just status, because the vendors disagree: Anthropic reports a spend cap as a **400** invalid_request_error, OpenAI reports exhausted credits as a 429, and Gemini uses 429 RESOURCE_EXHAUSTED for both quota and plain rate limiting. Two things the Gemini adapter has to get right, both found by calling it: - Thinking tokens count against `maxOutputTokens`. A 13-token prompt spent 147 of them before writing 5 tokens of answer, so sizing the budget to the caller's `maxTokens` returns an empty string — the model thinks up to the cap and never reaches the reply. This is the same failure that made the capped Anthropic call look like a silent success. - The default model is an alias. `gemini-2.0-flash` is already retired and answers 404 "no longer available"; a pinned version is a scheduled outage. Thinking is also reported as output tokens rather than hidden, because an accounting line that reads low while the bill reads high is how a cost surprise happens. Ordered anthropic -> gemini, built from whichever keys are set, so a capped key that regains access returns to being primary with no redeploy. A fallback should be temporary rather than a permanent quiet substitution. 18 new tests, 498 total. Verified live: the real capped Anthropic key 400s, the chain hands over, and Gemini returns correct JSON. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 13, 2026
ralyodio
added a commit
that referenced
this pull request
Aug 13, 2026
The chain added in #25 was anthropic -> gemini. This puts OpenAI between them, because it is the closer substitute: the prompts and the deterministic gates were built around Claude, and a same-shaped model is likelier to produce a draft that still passes them. Gemini stays as the last link rather than the only one. `OpenAIModel` is plain HTTP for the reason `GeminiModel` gives — one POST with a documented body is less to keep working than another SDK. Three things in it are not obvious and each is a bug avoided: - `max_completion_tokens`, never `max_tokens`. The reasoning models reject the latter outright, and a fallback that 400s on every call is worse than no fallback at all. - The same reasoning-token headroom Gemini needed. Sizing the cap to the caller's `maxTokens` lets the model reason up to the limit and return an empty string, which reads downstream as a model with nothing to say. - A refusal arrives as HTTP 200 with `content: null` and `refusal` set. Read without checking it, that is a successful blank draft. The default is `gpt-5.5` rather than a dated snapshot. There is no moving `-latest` alias for the reasoning line to use the way Gemini has one, so the generation is bumped by hand and `OPENAI_MODEL` overrides it. Verified against the live API: the id resolves, and the key in the vault answers 429 `insufficient_quota` / `credit_balance_exhausted` — which `isBudgetExhausted` already classifies as budget, so the chain steps past it to Gemini rather than failing the request. That body is now a test. Also documents `GEMINI_API_KEY`, which #25 never added to `.env.example`, and corrects the 503 that told operators to set `ANTHROPIC_API_KEY` as though it were the only key that enables drafting. 510 pass, 0 fail; format and typecheck clean. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(ai): fall back to Gemini when a provider runs out of budget
Not OpenAI, which was the ask: its key is exhausted too. Both keys in
the vault are dead — Anthropic capped until 2026-09-01, OpenAI at
429 insufficient_quota, no credits remaining— so falling back therewould have swapped one dead vendor for another. Gemini answers, and is
what this uses.
FallbackModelsits behind the existingTextModelinterface, so thecomposer, the profile drafter and the site extractor are untouched and
the deterministic quality gates still run over whatever comes back. One
package still talks to a model; only which service answers changes.
The whole design rests on one distinction: a provider being out of
budget is worth retrying elsewhere, and almost nothing else is. A
refusal is a judgement the next vendor will likely repeat — shopping it
around until someone agrees is precisely the behaviour this product
should not have. A malformed request is our bug and fails identically
downstream. A 500 is transient and belongs to whatever retries the job.
Detecting exhaustion needs message matching, not just status, because
the vendors disagree: Anthropic reports a spend cap as a 400
invalid_request_error, OpenAI reports exhausted credits as a 429, and
Gemini uses 429 RESOURCE_EXHAUSTED for both quota and plain rate
limiting.
Two things the Gemini adapter has to get right, both found by calling it:
maxOutputTokens. A 13-token promptspent 147 of them before writing 5 tokens of answer, so sizing the
budget to the caller's
maxTokensreturns an empty string — the modelthinks up to the cap and never reaches the reply. This is the same
failure that made the capped Anthropic call look like a silent success.
gemini-2.0-flashis already retiredand answers 404 "no longer available"; a pinned version is a scheduled
outage.
Thinking is also reported as output tokens rather than hidden, because
an accounting line that reads low while the bill reads high is how a
cost surprise happens.
Ordered anthropic -> gemini, built from whichever keys are set, so a
capped key that regains access returns to being primary with no
redeploy. A fallback should be temporary rather than a permanent quiet
substitution.
18 new tests, 498 total. Verified live: the real capped Anthropic key
400s, the chain hands over, and Gemini returns correct JSON.