fix(relay): preserve multimodal tool outputs in Responses-to-Chat conversion

[zvv1999]

Agent

• Tool: OpenAI Codex（AI 辅助编写与验证，非项目核心开发者）
• Tool version: 桌面宿主版本未读取；本机 CLI 包 0.141.0，CLI 因缺少平台可选依赖未用于执行本次任务。
• Model (full id): 宿主标识 GPT-6；完整模型 ID 未向任务公开。
• Host (CLI / IDE / GitHub coding agent / other): Codex Desktop / macOS
• Date (UTC): 2026-09-06

Links

• 未关联本仓库 Issue；本 PR 附有独立、确定性的源码复现与回归测试。
• Related: fix(openai): preserve multimodal client tool outputs Wei-Shaw/sub2api#6246 （参考其保留多模态工具结果的原则；未直接移植代码）
• Related: fix(relay): auto-recovery for images, context overflow, and encrypted reasoning items #7019 （不同路径的错误恢复）

User request

你能不能用Sub2API的修复 给New API 提一下PR

Out of scope — refuse

• Matched: no
• 此处修复标准 Responses → Chat 请求转换器的 function_call_output；不是 Codex 渠道、第三方托管故障或透传请求修复。

Kind

• Bug fix
• New feature
• Performance / refactor
• Docs
• Other:

Issue facts

以下事实来自本次针对 upstream 49ec4696682530781a036eab1ac195f0b04706c0 的离线复现，不将其他项目的生产事故当成本仓库证据。

• Actual behavior: responseToolOutputToChatContent 对非字符串 output 整体 Marshal 成字符串。合法的 input_image 数组因此成为 tool 文本中的 Base64，input_file 同样失去结构。
• Impact: 转换后的请求不再包含对应的视觉/文件内容；大 Base64 可能作为文本占用上下文。未在此 PR 声称实测了具体模型的计费或 token 数。
• Frequency: 每次转换该类构造请求都可确定性复现。
• Evidence: 相同新增测试在原始上游实现失败；修复后通过。失败输出显示两条 tool content 为包含完整 image data URI 的 JSON 字符串。
• Applicable types: RelayKit 的 openai_responses_to_openai_chat_completions 请求转换。数据库、部署、UI、定价逻辑不涉及。未运行 HTTP 服务或调用外部模型。

Change

仅识别由 input_text、input_image、input_file 组成的非空工具结果数组。纯文本数组转换为工具文本；含媒体的数组保持内容顺序，转换为带原始 call_id 说明的后续 user 多模态消息，tool 回复保留 call_id 和占位说明。所有连续工具回复完成后才追加媒体，避免打断并行调用配对。

图片 URL/data URI、detail、文件数据和引用保持原值，不下载、重编码、压缩或施加字节限制。Chat 无法直接表达的 image file_id 和 file_url 明确报错。普通对象、字符串、空数组及未知类型数组保留原有行为。

Research

Duplicate / prior art

• Search queries: repo:QuantumNous/new-api is:pr multimodal；repo:QuantumNous/new-api "function_call_output" "image"。
• fix(relay): auto-recovery for images, context overflow, and encrypted reasoning items #7019 在错误后删除/压缩历史图片并重试，不修复该转换器的字符串化。feat: 新增 Responses → ChatCompletions 协议降级兼容配置 #5209 为协议降级配置提案。Sub2API 支付金额错误导致充值失败 #6246 处理 Responses→Responses，不能直接复制至 Chat；本实现针对 Chat tool 仅接受文本的约束。

Docs and code

• https://docs.newapi.ai/ ：已打开官方入口，包含 API 文档与统一网关介绍。
• https://deepwiki.com/QuantumNous/new-api ：已打开；索引时间早于当前提交，仅作目录导航，结论以代码和测试为准。
• README / repo docs: 根 README 的统一网关定位；AGENTS.md 要求独立验证 RelayKit。
• Code paths: relaykit/relayconvert/internal/oai_responses/to_oai_chat_req.go 的 responsesRequestMessagesToChat 与 responseToolOutputToChatContent；原实现将媒体结果写成字符串。

Alternatives considered

• Option A: 直接把 Responses 多模态数组写进 Chat tool content；不符合目标协议的文本工具回复约束。
• Option B: 删除、压缩图片或在错误后重试；无法解决内容类型错误。
• Why this approach: 保留结构化内容及工具关联，同时维持并行工具回复顺序。跨角色承载的语义折衷明确披露，供维护者审查。

Files

Path	Why
relaykit/relayconvert/internal/oai_responses/to_oai_chat_req.go	识别工具内容数组并转换为 Chat 可表达的消息
relaykit/relayconvert/internal/oai_responses/to_oai_chat_req_test.go	在既有测试文件补充并行、多图、文件、detail 和兼容边界

Behavior

• Before: 图片/文件的整个内容数组进入 tool 文本，结构化输入丢失。
• After: tool call_id 保留，媒体按原顺序写为结构化 user 内容，带明确的工具数据归属说明。
• Explicit non-goals / leftover work: 不改原生 Responses 透传、custom tools、腾讯特定适配、压缩/重试/计量策略；不添加远程文件解析。

Verification

• Commands and results (all from repository root):
  • 新增测试配原始生产文件：GOWORK=off go -C relaykit test ./relayconvert/internal/oai_responses -run TestResponsesTool -count=1，退出 1，可复现媒体被字符串化。
  • 恢复修复后同一命令通过。
  • GOWORK=off go -C relaykit test ./... 通过。
  • GOWORK=off go -C relaykit build ./... 通过。
  • GOWORK=off go -C relaykit vet ./... 通过。
  • gofmt 与 git diff --check 通过。
• Manual steps and observed result: 回归测试序列化实际转换结果，断言 image_url.url、detail、文件数据及两条 tool 回复先于媒体消息，输入原文未变。
• UI: 不涉及。
• Tests added or updated: 同一既有文件中的两组回归测试，覆盖多图、并行回复、结尾输出、纯文本、普通对象、空数组、未知类型、文件引用及不支持的引用形式。
• Databases / providers / platforms exercised: macOS 本地 Go；无数据库、无真实 provider 调用。
• Not verified: 根应用构建、HTTP 端到端、外部 provider 的视觉识别和计费。

Risks

• Failure modes: 目标模型仍须支持图片/文件；本转换不能赋予文本模型视觉能力。媒体从 tool 角色转为有明确标记的 user 角色，是 Chat 协议下的折衷，标记不构成安全隔离保证。
• Billing / quota / auth impact: 无相关代码变更；结构化媒体可能改变上游的正常计量，未做真实计费验证。
• Follow-ups: 维护者可使用支持视觉的 Chat provider 做端到端验收。

Scope check

• Single focused change: yes
• Secrets included: no
• Out of scope (Coding Plan / reverse-engineered channel / third-party wrapper / Codex): no

Summary by CodeRabbit

• New Features

  • Tool responses containing supported images or files are now converted into a separate user message while preserving their associated assistant tool calls.
  • Image URLs, detail settings, and file data are retained during conversion.
  • Parallel tool responses maintain their ordering and payloads.

• Bug Fixes

  • Invalid media formats and missing required identifiers now produce clear conversion errors.
  • Base64 file data is preserved without being incorrectly embedded in text content.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 2a18f341-7cb6-45e0-a402-2e2c1413ab12

📥 Commits

Reviewing files that changed from the base of the PR and between 49ec469 and f479cb2.

📒 Files selected for processing (2)

• relaykit/relayconvert/internal/oai_responses/to_oai_chat_req.go
• relaykit/relayconvert/internal/oai_responses/to_oai_chat_req_test.go

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

Walkthrough

Responses input conversion now handles multimodal function_call_output items. Tool replies remain ordered, supported images and files become adjacent user messages, and unsupported media formats return errors. Tests cover parallel replies, payload preservation, content boundaries, and input immutability.

Changes

Responses tool media conversion

Layer / File(s)	Summary
Tool output routing relaykit/relayconvert/internal/oai_responses/to_oai_chat_req.go	Function call outputs use dedicated handling. Tool replies preserve parallel ordering, while media content is deferred to adjacent user messages.
Multimodal output validation relaykit/relayconvert/internal/oai_responses/to_oai_chat_req.go, relaykit/relayconvert/internal/oai_responses/to_oai_chat_req_test.go	Text, images, and files follow separate conversion rules. Unsupported file_id images and file_url files return errors. Tests verify roles, identifiers, media payloads, boundaries, and unchanged input data.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to f479c

Responses tool outputs containing supported images or files now retain their multimodal structure during Chat conversion, while unsupported references fail explicitly. Covered ordering, payload, compatibility, and immutability behavior leaves no concrete merge-blocking risk.

Sequence Diagram(s)

sequenceDiagram
  participant ResponsesInput
  participant Converter
  participant ChatRequest
  ResponsesInput->>Converter: provide function_call_output items
  Converter->>ChatRequest: append ordered tool messages
  Converter->>ChatRequest: append supported media as user messages
  Converter-->>ResponsesInput: return converted request or validation error

Loading

Suggested reviewers: calcium-ion

Poem

A rabbit sorts tool calls in a row
Images and files follow below
Text hops neatly into place
Errors guard the media space
Parallel replies keep their pace

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: preserving multimodal tool outputs during Responses-to-Chat conversion.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}