[MLPerf 6.1][GRPO] Define Qwen3.5 OpenHands agent harness contract

[al-rigazzi]

Summary

• Define a benchmark-specific OpenHands CodeActAgent harness contract for qwen35_397b_grpo.
• Allow a submitter-selected immutable OpenHands repository and commit, with narrow documented deviations, while preserving model inputs, tool behavior, turn accounting, termination semantics, and scoring.
• Make security_risk optional for execute_bash and str_replace_editor, and require disclosure of the harness repository, commit, and all reference deviations.

Motivation

The current rule pins the agent harness to a single OpenHands repository and commit. Integrating the harness with different training, inference, serving, or environment frameworks may require narrow implementation changes for compatibility, reliability and failure handling, resource isolation, or security. An exact commit requirement makes behaviorally equivalent implementations noncompliant even when they preserve benchmark-relevant behavior. This change instead fixes the model-visible inputs, tool-call contract, action and result semantics, turn accounting, termination behavior, and scoring contract, while allowing a submitter-selected immutable implementation and requiring every deviation from the reference to be disclosed.

We propose making security_risk optional because it is auxiliary, model-generated metadata that is not required for tool execution or task scoring in this benchmark's noninteractive configuration. The reference OpenHands tool schemas declare security_risk required for execute_bash and str_replace_editor, but the reference runtime does not enforce its presence: it checks command for execute_bash, checks command and path for str_replace_editor, and sets security_risk only when supplied. In the benchmark's noninteractive configuration, confirmation_mode defaults to false, risk handling is conditional on confirmation mode, and task reward depends only on whether the task is resolved. Its surrounding syntax accounts for a substantial fraction of observed rollout–trainer logprob outliers, which can cause entire successful trajectories to be discarded. Making the annotation optional removes the requirement to generate those problematic contexts while preserving task semantics, tool capabilities, execution controls, and training filters. Preliminary multi-seed results support the mitigation.

Validation

• Documentation-only policy change.
• git diff --check origin/master...HEAD passes.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅