Skip to content

fix(relay): bill Responses stream usage on incomplete/cancelled/failed terminal events - #7242

Open
txgo wants to merge 1 commit into
QuantumNous:mainfrom
txgo:fix/responses-stream-usage-on-terminal-events
Open

fix(relay): bill Responses stream usage on incomplete/cancelled/failed terminal events#7242
txgo wants to merge 1 commit into
QuantumNous:mainfrom
txgo:fix/responses-stream-usage-on-terminal-events

Conversation

@txgo

@txgo txgo commented Sep 7, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

Agent

  • Tool: Claude Code
  • Tool version: CLI (claude.ai/code)
  • Model (full id): claude-opus-5
  • Host (CLI / IDE / GitHub coding agent / other): CLI
  • Date (UTC): 2026-09-07

Links

User request

用户原话:「rc.30 计费缺口:流式 /v1/responses 以 response.incomplete 结束时不读 usage,整条按 0 计」,并要求「向上游提交 issue 和 PR……PR 更是要遵从规范,让 PR 能够真正被接受」。

Out of scope — refuse

  • Matched: no
  • If yes, what was told to the user (stop here; do not open a PR): 不适用。本 PR 是中继计费缺陷修复,不属于 Coding Plan、逆向渠道、第三方封装、Codex 渠道类型改动、透传转发或第三方托管。

Kind

  • Bug fix
  • New feature
  • Performance / refactor
  • Docs
  • Other:

Issue facts

  • Actual behavior: 流式 /v1/responses 以 response.incomplete 结束(如 incomplete_details.reason = max_output_tokens)时,该请求 quota = 0,日志记「上游没有返回计费信息,无法扣费」,而该终止事件体内带有 usage。
  • Impact: 静默单向漏计。token 已产生、上游已计费,网关侧记 0 且不报错、无告警。
  • Frequency: 某自托管网关近 7 天(截至 2026-09-03)流式 responses 日志 47,347 行中,属本缺陷的 75 行(glm-5.3 38 · glm-5.3-flash 35 · doubao-seed-evolving 2,涉 6 个用户)。
  • Evidence that the problem is in new-api rather than the client or upstream: ① 终止事件体内确有 usage.output_tokens(64 / 1500,与请求的 max_output_tokens 一致);② 同上游、同模型、同截断条件下非流式请求计费正常,差异只在 stream;③ 源码分支可直接解释该差异(见 Change)。
  • Applicable types and their fields (relay / billing / frontend / deployment): relay 与 billing 适用,字段已在 Bug: streaming /v1/responses discards usage on response.incomplete and bills the request as zero #7241 对应小节逐项填写;frontend / deployment 不适用。

Change

OaiResponsesStreamHandler 只在 response.completed / response.done 分支读取 Response.Usage;response.failed / response.incomplete / response.cancelled / response.canceled 这一分支只复位图像计数器,不读 usage。

本 PR 在该终止分支复用 completed 分支已在用的同一对函数读取 usage:

if streamResponse.Response != nil && streamResponse.Response.Usage != nil {
    incomingUsage := relayconvert.NormalizeResponsesUsage(streamResponse.Response.Usage)
    usage = dto.MergeUsageNonZero(usage, incomingUsage)
}

为什么这样能生效,三点都落在实际改到的代码上:

  1. 与非流式路径对齐。 同文件的 OaiResponsesHandler 第 42 行 usage := relayconvert.NormalizeResponsesUsage(responsesResponse.Usage) 是无条件的,所以 stream = false 时同一截断请求计费正确。流式路径此前与它不一致,本 PR 消除这个分叉。
  2. 既有兜底覆盖不到。 函数结尾 142-150 行的输出文本兜底要求流中出现过 response.output_text.delta;推理型模型在产出首个文本 token 前被截断时 responseTextBuilder 为空,兜底不触发,prompt 与 completion 均归零。这解释了生产命中集中在推理模型。
  3. 图像计数语义不变。 IsNonBillableResponsesStatus 只服务于图像生成调用计数,终止分支仍照旧 Reset() 后 Commit();本 PR 不触碰这段。token 已经产生,与图像调用是否成立是两件事。

Research

Duplicate / prior art

Docs and code

  • https://docs.newapi.ai/ :检索首页与 zh/docs/guide/feature-guide/user/pricing。未记载流式 Responses 的 usage 提取,也未记载终止事件的计费处理;无可配置开关。
  • https://deepwiki.com/QuantumNous/new-api :检索 streaming billing / Responses usage。仅有「three-phase quota tracking, tiered pricing expressions」一句概括,未涉及本主题。
  • README / repo docs:未见相关条目。
  • Code paths and what they imply for this change:
    • relay/channel/openai/relay_responses.go:91-96 — completed / done 分支读 usage 的现成写法,本 PR 直接复用。
    • relay/channel/openai/relay_responses.go:115-120 — 终止分支,本 PR 唯一改动点。
    • relay/channel/openai/relay_responses.go:42 — 非流式无条件读 usage,是「应有行为」的项目内参照。
    • relay/channel/openai/relay_responses.go:142-150 — 文本兜底,说明为何缺陷未被现有机制掩盖。
    • relay/common/tool_usage.go:180 — IsNonBillableResponsesStatus 的适用范围仅为图像计数,故不能拿它否定 token 计费。

Alternatives considered

  • Option A:把终止事件也并入 completed 分支统一处理。 会一并把图像计数的 Observe 逻辑套到终止事件上,改变现有图像计费语义,超出修复范围。
  • Option B:在函数结尾兜底处补一条「若 usage 为空则回看最后一个终止事件」。 需要额外保存事件状态,且把「事件里已有的数」推迟到收尾再取,增加状态而非减少分叉。
  • Why this approach: 只补一个分支缺失的读取,复用同文件已在用的 NormalizeResponsesUsage + MergeUsageNonZero,不新增状态、不改变图像计数、不引入新概念,并直接消除与非流式路径的不一致。

Files

Path Why
relay/channel/openai/relay_responses.go 终止事件分支补读 Response.Usage(+11 行,含注释)
relay/channel/openai/relay_responses_billing_test.go 新增 4 个计费用例与一个共享驱动 helper(+103 行)

Behavior

Verification

  • Commands and results:

    • gofmt -l relay/ → 无输出
    • go vet ./... → 通过
    • make test → EXIT=0,全部包通过(50 个包报 ok)
    • go test ./relay/channel/openai/ → ok

  • Manual steps and observed result: 缺陷本身在自托管网关上以真实上游复现(stream = true,max_output_tokens = 64 与 1500,推理型模型),两次均以 response.incomplete 结束、事件体带 usage、日志 quota = 0。详见 Bug: streaming /v1/responses discards usage on response.incomplete and bills the request as zero #7241。

  • UI: screenshot or recording (or why none): 无 UI 改动。

  • Tests added or updated: 新增 4 个用例。每个用例都做了阳性对照(临时移除修复后单独运行):

    用例 无修复 有修复
    ...BillsUsageOnIncompleteWithoutTextDelta FAIL(0 vs 64) PASS
    ...PrefersUpstreamUsageOnIncomplete FAIL PASS
    ...BillsUsageOnCancelled FAIL PASS
    ...IncompleteWithoutUsageStaysZero PASS(阴性对照,按设计两侧都应通过) PASS

    共享 helper 里调用了 service.InitTokenEncoders():文本兜底路径会解引用默认编码器,不初始化时该路径以空指针 panic 收场而非给出断言失败。初始化后为纯内存操作(实测 0.01s,不联网)。

  • Databases / providers / platforms exercised: 复现发生在 mysql + 原生 Responses 上游(converter = none);单元测试不触及数据库。

  • Not verified: 未做「直连上游 vs 经 new-api」逐字对照;未在 sqlite / postgres 上复现(该路径不含数据库分支);未验证 response.failed 在真实上游下是否总带 usage(实测只覆盖 incomplete);经转换器的 Responses 路径未验证。

Risks

Scope check

  • Single focused change: yes
  • Secrets included: no
  • Out of scope (Coding Plan / reverse-engineered channel / third-party wrapper / Codex): no

Summary by CodeRabbit

  • Bug Fixes
    • Improved usage tracking for incomplete, failed, and cancelled streaming responses.
    • Billing now reflects upstream usage even when no text output was streamed.
    • Preserved accurate usage precedence when upstream usage data is available.

@txgo
fix(relay): bill Responses stream usage on incomplete/cancelled/faile…
23a7140 
…d terminal events

`OaiResponsesStreamHandler` only read `response.usage` in the
`response.completed` / `response.done` branch. The other terminal events
(`response.incomplete`, `response.cancelled`, `response.canceled`,
`response.failed`) also carry a response object with upstream usage, but the
handler discarded it and only reset the image counter.

The non-streaming path does not have this gap: `OaiResponsesHandler` calls
`NormalizeResponsesUsage(responsesResponse.Usage)` unconditionally, so the same
truncated request is billed correctly when `stream=false`. The streaming path
was silently inconsistent with it.

The output-text fallback at the end of the handler does not cover the gap. It
only triggers when `response.output_text.delta` events were seen, so a
reasoning model truncated by `max_output_tokens`, or any response truncated
before the first text token, ends with zero prompt and completion tokens even
though the terminal event reported real counts.

Fix: read usage in the terminal-event branch through the same
`NormalizeResponsesUsage` + `MergeUsageNonZero` pair the completed branch uses.
Image-generation call counting is unchanged: those events still reset the
counter, because the image call did not complete.

Tests: four cases in relay_responses_billing_test.go covering incomplete
without a text delta, incomplete with a text delta (upstream numbers must beat
the local estimate), cancelled, and a terminal event carrying no usage at all
(must stay zero). Without the fix the first three fail and the fourth passes;
with the fix all four pass. The shared helper initializes the token encoders so
the fallback path reports an assertion failure instead of a nil-pointer panic.
@coderabbitai

coderabbitai Bot commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: a52d77dc-e4ee-4a60-97e6-5c162fe4b9ed

📥 Commits

Reviewing files that changed from the base of the PR and between 0c76e4d and 23a7140.

📒 Files selected for processing (2)
  • relay/channel/openai/relay_responses.go
  • relay/channel/openai/relay_responses_billing_test.go

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

The streaming Responses handler now reads usage from terminal response events and merges it into accumulated usage. Tests cover incomplete and cancelled events, upstream usage precedence, and terminal events without usage.

Changes

Streaming usage billing

Layer / File(s) Summary
Capture terminal-event usage
relay/channel/openai/relay_responses.go		 Terminal response.failed, response.incomplete, response.cancelled, and response.canceled events now normalize and merge usage from their response objects.
Validate terminal billing paths
relay/channel/openai/relay_responses_billing_test.go		 Tests cover incomplete responses with or without text, cancelled responses, upstream usage precedence, and missing usage data.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 23a71

Streaming Responses requests now retain upstream token usage from terminal incomplete, cancelled, canceled, and failed events for billing while preserving existing image-count behavior. The covered terminal billing paths are ready to merge.

Suggested reviewers: calcium-ion

Poem

A rabbit watched the stream flow bright
Usage tokens came into sight
Incomplete paths now count the fare
Cancelled hops are handled with care
No usage? Zero stays there

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the Responses streaming billing fix for incomplete, cancelled, and failed terminal events.
Linked Issues check ✅ Passed The implementation reads and merges terminal-event usage for response.incomplete, response.cancelled, response.canceled, and response.failed, preserves behavior when usage is absent, and keeps image-g…
Out of Scope Changes check ✅ Passed The changes are limited to terminal-event usage handling in the Responses streaming relay and targeted billing tests. They align with issue [#7241] and do not address unrelated disconnect or client_go…
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@txgo

txgo commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

CI 说明:Frontend typecheck and test 的失败与本 PR 无关

Backend vet, build, and test 已通过。失败的是前端检查,而本 PR 不含任何前端改动:

git diff --stat origin/main -- web/   →   空输出

改动仅两个 Go 文件(+114/-0):relay/channel/openai/relay_responses.go 与
relay/channel/openai/relay_responses_billing_test.go。

在未改动的 main 上可复现

在本分支的工作树里按 CI 同一命令(bun install --frozen-lockfile → bun run test)执行前端测试。
该工作树的 web/ 与 origin/main 逐字相同(上面的 diff 为空),结果:

Test Files  4 failed | 84 passed (88)
     Tests  15 failed | 652 passed (667)

本地失败的 4 个文件全部出现在本次 CI 的失败清单中:

  • src/features/dashboard/components/overview/__tests__/setup-guide.test.tsx
  • src/features/keys/components/__tests__/api-keys-mutate-drawer.test.tsx
  • src/features/redemption-codes/components/__tests__/redemptions-mutate-drawer.test.tsx
  • src/features/usage-logs/audit/__tests__/viewer.test.tsx

CI 侧共 44 个文件失败,本地 4 个。差额可由环境解释:这些是依赖指针事件与元素可见性的
React 交互用例,CI 报的两类错误正是此形状 ——
Unable to perform pointer interaction as the element has 'pointer-events: none'
与 expect(element).toBeVisible()。

时间线(相关性,非因果结论)

本 PR 基于当前 origin/main HEAD 0c76e4da(feat(models): rework model/vendor management and pricing,
102 files, +10580/−5924,其中包含 web/vitest.config.ts)。

近期已合并的 PR(#7221 · #7211 · #7171 · #7170 · #7168)前端检查结论均为 SUCCESS,
而它们都合并于该提交之前。

⚠ 我没有做二分定位,因此不声称 0c76e4da 就是成因。 能确证的只有一条:
在前端代码一字未改的情况下,这些用例在该基点上就会失败。

未在本 PR 内处理

按模板的 Focused change 要求,不在这个计费修复里夹带无关的前端改动。
若维护者希望我另开一单跟踪前端用例的失败,我可以照 .agents/github/ISSUE.md 提交。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Bug: streaming /v1/responses discards usage on response.incomplete and bills the request as zero

1 participant

@txgo