Three CN Lite tasks have prompt/manifest inconsistencies (tasks 23, 127, and 386)

[eissac]

Summary

Three Chinese Workspace-Bench Lite tasks contain inconsistencies between their prompts, manifests, and the files available in the full workspace image:

• opendatabox-workspace-bench-23
• opendatabox-workspace-bench-127
• opendatabox-workspace-bench-386

I checked each task's CN Lite metadata and packaged inputs, the complete filename index of filesys_cn.zip, and the repository's workspace construction logic.

1. Task 23: the current-inventory workbook is missing from the task manifest

The prompt asks the agent to correlate the stocktaking report, current inventory list, and inbound/outbound logs. However, the task's data_manifest and file_dep_graph omit:

当前库存物品总清单_2024-12.xlsx

The workbook does exist in the full CN workspace image at:

LogisticsManager_Workdir/后勤/库存/物品清单/当前库存物品总清单_2024-12.xlsx

The original runner normally copies the complete raw persona workspace before overlaying manifest files, so the file may remain visible in the original full-workspace evaluation. Nevertheless, the task package is not self-contained, the declared dependency graph is incomplete, and manifest-only conversions omit a source explicitly required by the prompt.

Suggested fix

Add the existing workbook to:

• data_manifest
• file_dep_graph
• Any generated setup/input manifest or solver-visible input list

No workspace image rebuild should be necessary because the workbook is already present in filesys_cn.zip.

2. Task 127: prompt says 10 Python files, but only 8 exist

The prompt says:

测试项目文件夹的python目录下有10个python文件

However, all other sources consistently show 8 files:

• The Lite data_manifest contains 8 Python files.
• The file_dep_graph contains 8 Python source nodes.
• The packaged task data contains 8 Python files.
• Research_Workdir/桌面/项目/测试项目/python/ in the full CN image contains 8 Python files.
• One rubric explicitly refers to all 8 Python files.

The files are:

1. data_process.py
2. database.py
3. image.py
4. machine_learning.py
5. parsing.py
6. utils.py
7. visualization.py
8. web_network.py

Suggested fix

Change the task text from 10 Python files to 8. No image, manifest, dependency-graph, or rubric change is otherwise required.

3. Task 386: the prompt requires transcription of four absent .m4a recordings

The prompt describes four .m4a recordings and explicitly asks the system to perform:

音频→文字→结构化决策表

The complete CN workspace image contains no .m4a files. It contains only:

• 会议录音元数据.json
• D1_上午场_转写稿.txt
• D2_下午场_转写稿.txt
• The five spreadsheet inputs used by the task

The metadata describes four recording sessions and marks ASR as completed for all four. It references four transcript paths, but only two transcript TXT files are actually supplied: D1 morning and D2 afternoon. The D1 afternoon and D2 morning transcript files are also absent.

The existing rubrics evaluate conclusions derived from the supplied transcripts, recording metadata, NPS survey, DAU logs, CRM data, and competitor report. They do not evaluate audio decoding or ASR execution.

Suggested fix

Revise the prompt to describe the supplied recording metadata and the available completed ASR transcripts, and remove the unsupported requirement to transcribe four recordings. To avoid implying that all four transcripts exist, wording such as the following would be accurate:

Analyze the supplied recording metadata and the available completed ASR transcripts.

Alternatively, add the four original .m4a files and the two missing transcript files if audio transcription is intended to remain part of the task.

Expected outcome

Please align these task prompts and manifests with the actual workspace inputs so that each task is self-contained and does not claim unavailable input files or incorrect file counts.

[wu-uk]

English

Thank you for the detailed audit. An opt-in local patch is available in PR #28: #28. To preserve fair comparison with existing leaderboard submissions, we are not changing the currently published Hugging Face snapshot at this time. Users who need the corrected task contracts can explicitly run the patch command below. We also plan to release a new Workspace-Bench version with refreshed task metadata and assets soon.

The three affected tasks have been corrected as follows:

• Task 23
  • Added 当前库存物品总清单_2024-12.xlsx to the task package, data_manifest, and file_dep_graph.
  • Added target_path entries so the corrected task-local inputs are materialized into the standard workspace at runtime.
  • Rechecked the purchase-order source and corrected two inconsistent rubric values to the values derivable from the supplied input: 29 pending-approval orders and 7 approved orders.
• Task 127
  • Corrected the prompt from 10 Python files to 8, matching the manifest, dependency graph, packaged inputs, and workspace directory.
• Task 386
  • Removed the unsupported requirement to transcribe four .m4a recordings.
  • The revised prompt now states that the available materials are recording metadata and two completed ASR transcripts (D1 morning and D2 afternoon), and explicitly prohibits inventing content for unavailable sessions.

To keep the fixes reproducible before the upstream Hugging Face snapshot is updated, the repository now includes an idempotent task-patch mechanism:

• evaluation/src/task_patches.py
• evaluation/scripts/apply_task_patches.py
• evaluation/task_patches/lite_cn/

The downloader does not apply these optional patches automatically. After downloading the unchanged snapshot, apply them explicitly with:

cd evaluation
python3 scripts/apply_task_patches.py --kind lite --language cn

The task-patch tests and the full evaluation test suite pass.

中文

感谢详细核查。可选的本地 patch 已提交至 PR #28：https://github.com/OpenDataBox/Workspace-Bench/pull/28。为了与现有 leaderboard 结果公平对比，我们暂不修改当前已发布的 Hugging Face 快照；如需使用修正后的任务合同，可显式运行下方 patch 命令。我们也计划很快发布包含更新后任务元数据和资源的新版本 Workspace-Bench。

这 3 个受影响任务已按以下方式修复：

• Task 23
  • 已将 当前库存物品总清单_2024-12.xlsx 加入任务包、data_manifest 和 file_dep_graph。
  • 已补充 target_path，确保修复后的任务级输入会在运行时被 materialize 到标准 workspace。
  • 已重新核对采购单源文件，并将两条与输入不一致的 rubric 修正为可由输入推导的实际值：待审批 29 张、已批准 7 张。
• Task 127
  • 已将题面中的 Python 文件数从 10 改为 8，与 manifest、依赖图、打包输入和 workspace 目录保持一致。
• Task 386
  • 已移除“对 4 段 .m4a 录音执行转写”的无输入支持要求。
  • 新题面明确当前可用材料为录音元数据和两份已完成 ASR 转写稿（D1 上午场、D2 下午场），并明确禁止虚构未提供会话的内容。

为在 Hugging Face 上游快照更新前保持修复可复现，仓库新增了可重复执行的 task patch 机制：

• evaluation/src/task_patches.py
• evaluation/scripts/apply_task_patches.py
• evaluation/task_patches/lite_cn/

下载器不会自动应用这些可选 patch。下载未修改的快照后，如有需要可显式执行：

cd evaluation
python3 scripts/apply_task_patches.py --kind lite --language cn

Task patch 测试和完整 evaluation 测试集均已通过。