How should source-only deepagents-evals CLI bugs be reported?

[Lucien (dakjdakd)]

I found a reproducible exit-code bug in the deepagents-evals run CLI on monorepo main at 0c756c4f8. libs/evals/tests/evals/pytest_reporter.py::pytest_sessionfinish() intentionally resets pytest exit status 1 to 0 after failed tests so aggregation can finish. libs/evals/deepagents_evals/cli.py::_cmd_run() checks only that subprocess status, so a run whose report contains {"failed": 1} exits 0. Automated callers therefore see success for a failed evaluation. trials and aggregate already inspect report failure counts.

A network-free CLI reproduction:

from pathlib import Path
from subprocess import CompletedProcess
from tempfile import TemporaryDirectory
from unittest.mock import patch

from deepagents_evals import cli

with TemporaryDirectory() as tmp:
    report = Path(tmp) / "report.json"

    def fake_run(cmd, **kwargs):
        Path(cmd[cmd.index("--evals-report-file") + 1]).write_text('{"failed": 1}')
        return CompletedProcess(cmd, 0)  # the reporter masks pytest exit 1

    with patch.object(cli.subprocess, "run", side_effect=fake_run):
        print(cli.main(["run", "--model", "fake:model", "--report", str(report)]))

Observed on main: 0; expected: 1 when the report records a failed evaluation. A separate check through the real reporter hooks produced pytest_exit=0 and report_failed=1. Related #3221 introduced the unified CLI, and #2932 addresses the no-tests-run exit, not this path.

I would report this with the Bug Report template, but its required checklist says the bug must reproduce with the latest released version. The published deepagents-evals==0.0.1 package is a placeholder; this CLI exists only in monorepo source, so I cannot truthfully check that item. What is the preferred route for a source-only evals bug? If this discussion can serve as the approved contribution link, I can provide a focused fix and regression tests once a maintainer approves the work and assigns me (@dakjdakd). Otherwise I can file an Issue through the route you recommend.