diff --git a/.agents/skills/README.md b/.agents/skills/README.md index d35991bd8..5754006c0 100644 --- a/.agents/skills/README.md +++ b/.agents/skills/README.md @@ -34,6 +34,7 @@ This table lists the maintainer skills in this set and what each covers. | [`contribute-docs`](contribute-docs/SKILL.md) | Change documentation or examples in step with public behavior. | | [`review-doc-style`](review-doc-style/SKILL.md) | Review documentation and public text for NVIDIA technical-writing style. | | [`validate-change`](validate-change/SKILL.md) | Choose and run the right validation matrix for a change. | +| [`qualify-harbor`](qualify-harbor/SKILL.md) | Guide shared-BYOM or individual-provider setup, then run one shared Terminal-Bench task across the adapter catalog with isolated installs and retained evidence. | | [`python-tests`](python-tests/SKILL.md) | Write Python tests for NeMo Fabric. | | [`maintain-ci`](maintain-ci/SKILL.md) | Maintain GitHub Actions workflows with pinned actions and local validation. | | [`maintain-packaging`](maintain-packaging/SKILL.md) | Maintain package metadata, native artifacts, lockfiles, and release surfaces. | diff --git a/.agents/skills/maintain-packaging/SKILL.md b/.agents/skills/maintain-packaging/SKILL.md index af3ca53ee..58cbea14b 100644 --- a/.agents/skills/maintain-packaging/SKILL.md +++ b/.agents/skills/maintain-packaging/SKILL.md @@ -56,6 +56,13 @@ in parity. The task uses `fabric-agent-ctl` from the Python SDK. Run `just adapter-catalog`, `just check-adapter-catalog`, and `uv build --wheel --sdist --out-dir dist sdk/python/nemo-fabric-adapter-catalog` when maintaining the bundle. `just set-version` refreshes it after stamping Python and TypeScript versions. Python tests and wheel builds reject stale bundles; wheels and source distributions must build independently of the adapter source tree. Keep these maintainer details out of the PyPI-facing package README. +### Pi SDK Compatibility + +Keep Pi’s optional peer constraints, development pins, consumer install command, +and harness-install diagnostic aligned across all four SDK packages, including +`pi-codemode`. Verify imports in a fresh consumer install from packed artifacts; +the workspace lockfile can hide incompatible transitive SDK resolutions. + ### Select Dependencies Treat every direct dependency as a long-lived API, supply-chain, and licensing diff --git a/.agents/skills/qualify-harbor/SKILL.md b/.agents/skills/qualify-harbor/SKILL.md new file mode 100644 index 000000000..e50da9397 --- /dev/null +++ b/.agents/skills/qualify-harbor/SKILL.md @@ -0,0 +1,36 @@ +--- +name: qualify-harbor +description: Guide interactive model and credential setup, then qualify NeMo Fabric across its adapter catalog or a selected subset through Harbor and real harnesses. Use for reproducible integration checks and all-harness failure triage. +license: Apache-2.0 +--- + +# Qualify NVIDIA NeMo Fabric With Harbor + +Use the runnable gate in this directory to test Harbor → NeMo Fabric → harness → inference → result → verifier. Harbor owns containers, task lifecycle, and scoring. The gate runs exactly one shared Terminal-Bench task across the catalog, builds isolated package installs on that task environment, and retains a complete adapter denominator. It does not install or manage inference servers. + +## Start With A Short Setup Conversation + +Use the user's existing choices and authorization. When starting a new run without a configured plan, ask how they want to supply models: + +- **One shared BYOM endpoint** for every selected adapter that supports its API and model. +- **Individual provider credentials**, grouping adapters that can share the same provider/key reference. + +Offer a mix when the user wants coverage beyond the shared endpoint’s compatible subset; otherwise record those exceptions as skipped. If the user already chose a mode or supplied a plan, continue from it; do not restart the questionnaire. Ask only for missing information, in small steps, using the available question tool or a concise chat question. + +Read [interactive setup](references/setup.md) for the conversation, credential handling, compatibility decisions, and coverage preview. The agent owns plan construction and adapter-specific configuration; users should not have to edit JSON rows or discover provider flags themselves. + +Before builds or model calls, show which adapters are ready, need setup, or will be skipped, with model/provider and credential **variable names only**. Resolve outstanding scope or cost decisions on that concrete preview, reusing any authorization already given. A shared BYOM choice never implies universal compatibility. Kiro requires its native provider and `KIRO_API_KEY`; BYOM is unsupported and missing credentials mean skipped execution, retained in the catalog denominator. + +## Execute The Saved Plan + +Read [the runbook](references/runbook.md) for reproducible commands, preparation profiles, and failure triage. The conversational setup produces the same noninteractive plan used by these commands. + +1. Identify the exact package version or source commit and compatible Harbor package version. Use a disposable checkout of that version with a fresh catalog (`just check-adapter-catalog`). Do not use a developer's ambient editable environment as reproducible installation evidence. +2. Establish inference prerequisites before running. A local checkpoint must already be served through a compatible HTTP API. Verify the served model ID, container reachability, protocol, authentication, and available resources. Use the same served model for compatible harnesses; record provider restrictions rather than assuming universal compatibility. Do not stop existing services, take allocated GPUs, mount credential directories, or publish new network listeners. +3. Select one task directory, not a benchmark dataset. The runbook pins TB2 `kv-store-grpc` as a comprehensive baseline. Generate and inspect the plan; its task hash covers the instruction, environment, and unchanged upstream verifier. It includes every catalog adapter, exact package/descriptor metadata, settings, estimated provider cost, and capability coverage. Resolve prerequisites by configuring individual rows and dedicated task images. Registered workflows require explicit `workflow_target_id` and optional `workflow_settings` with the SDK bridge; validate catalog binding, both descriptor fingerprints, settings transport, and installed target provenance. Keep blocked, unsupported, and excluded rows in the plan with reasons. +4. Prepare isolated host and task installs, then run the selected matrix. Use `--only` for a developer subset; its report cannot establish a full-catalog pass. The default is one trial at a time, one attempt, bounded runtime/build deadlines, and no automatic retries. Preserve each failure before reproducing it in a new output directory. +5. Use compatible host/task Python versions, selecting `task_python` per row when needed. Keep Fabric packages noneditable; allow upstream-supported pinned Hermes source installs with verified clean Git provenance. Do not require the task base image’s Python version for every harness. +6. Inspect `report.json`, `summary.md`, installation logs, and trial artifacts. The pass criterion is Harbor → Fabric → harness → inference → result integration: lifecycle, selection/fingerprint, package identity, result/accounting transport, requested telemetry, and cleanup. A completed verifier's task reward is informational; reward zero can pass integration. Harness usage availability is also informational: retain unknown counters and scope limitations without inventing totals. Accounting that changes in transport remains a failure. Reproduce findings and classify them before attributing a regression. Fix gate defects narrowly; report unrelated product defects separately. +7. Share the sanitized report and exact evidence locations with the reviewer. A full pass requires every catalog entry to pass integration. Missing credentials, incompatible providers, install failures, missing or invalid verifier evidence, timeout, cancellation, and skipped coverage remain nonpasses. Model task quality and documented harness capabilities do not fail an otherwise successful round trip. + +One shared TB2 task is a scored integration check, not a full benchmark or proof of all-feature equivalence. MCP, skills, ATIF/export, and subagent paths require their own evidence where supported. The report labels untested paths. Use the existing Harbor integration suites for deterministic lifecycle, error, credential, MCP, skills, and telemetry coverage; record any live capability tests separately. Never infer a capability pass from one successful task. diff --git a/.agents/skills/qualify-harbor/assets/Dockerfile b/.agents/skills/qualify-harbor/assets/Dockerfile new file mode 100644 index 000000000..9d112c83a --- /dev/null +++ b/.agents/skills/qualify-harbor/assets/Dockerfile @@ -0,0 +1,29 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +ARG TASK_IMAGE=python:3.12-slim-bookworm +FROM ${TASK_IMAGE} +COPY --from=node:24.16.0-bookworm-slim /usr/local /usr/local +RUN apt-get update && apt-get install -y --no-install-recommends build-essential curl git pkg-config libssl-dev unzip ripgrep \ + && rm -rf /var/lib/apt/lists/* +ENV CARGO_BUILD_JOBS=4 +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o /tmp/rustup.sh \ + && sh /tmp/rustup.sh -y --profile minimal --default-toolchain 1.94.0 && rm /tmp/rustup.sh +ENV PATH=/root/.cargo/bin:/opt/harness/node_modules/.bin:$PATH +COPY source /src +ARG FABRIC_VERSION +ARG INSTALL_MODE +RUN mkdir /wheels && if [ "$INSTALL_MODE" = source ]; then \ + pip wheel --no-deps --wheel-dir /wheels /src/adapter-contract/python \ + /src/sdk/python/nemo-fabric-runtime /src/sdk/python/nemo-fabric \ + /src/sdk/python/nemo-fabric-adapter-catalog \ + /src/adapters/python/*/; \ + else pip download --only-binary=:all: --no-deps --dest /wheels \ + "nemo-fabric==$FABRIC_VERSION" "nemo-fabric-runtime==$FABRIC_VERSION" \ + "nemo-fabric-adapter-catalog==$FABRIC_VERSION" \ + "nemo-fabric-adapter-contract==$FABRIC_VERSION" "nemo-fabric-adapters-common==$FABRIC_VERSION"; fi +RUN pip install /wheels/nemo_fabric-*.whl /wheels/nemo_fabric_runtime-*.whl \ + /wheels/nemo_fabric_adapter_catalog-*.whl /wheels/nemo_fabric_adapter_contract-*.whl \ + /wheels/nemo_fabric_adapters_common-*.whl +COPY identity.py /qualification/identity.py +WORKDIR /app diff --git a/.agents/skills/qualify-harbor/references/runbook.md b/.agents/skills/qualify-harbor/references/runbook.md new file mode 100644 index 000000000..cb8566417 --- /dev/null +++ b/.agents/skills/qualify-harbor/references/runbook.md @@ -0,0 +1,251 @@ + + +# Harbor Qualification Runbook + +When invoked as an agent skill, start with [interactive setup](setup.md): choose a shared BYOM endpoint or individual provider credentials, resolve compatible routes, and preview coverage. The commands below are the replayable execution reference. The agent generates and validates the plan; a user does not need to hand-edit the per-adapter examples. + +## Prerequisites And Scope + +Run from the repository root on Linux x86_64 with Git, Python 3.12 or newer, `uv`, Docker, and Docker Compose. The image builder preserves the selected task Docker environment and adds Node.js 24.16.0, Rust 1.94.0, and packaged NVIDIA NeMo Fabric and harness dependencies. The baseline task image uses Python 3.13 on Debian Bookworm; that is not a Fabric or Harbor version pin. Select any compatible installed interpreter for each component. Custom tasks must provide a compatible Debian environment with pip and apt-get. It can download an isolated host Python interpreter through `uv`. Use a dedicated output directory on storage visible to the Docker daemon; Snap Docker needs a `TMPDIR` visible outside its private `/tmp`, as described in the [Harbor examples](../../../../examples/harbor/README.md#docker-installed-with-snap). Remote Docker daemons need the same absolute paths as the client. No host credential directories are mounted. + +Choose the exact NVIDIA NeMo Fabric package version and matching checkout/catalog, plus an exact Harbor version compatible with that NeMo Fabric line. The native Harbor `fabric` agent is the default; `--bridge sdk` explicitly tests `nemo_fabric.integrations.harbor:FabricAgent` instead. Record the distinction when comparing runs. Native Harbor and the SDK bridge are independently versioned implementations. + +`--mode registry` installs the exact published version. `--mode source` builds noneditable wheels from the recorded committed source, installs the same runtime wheel on host and task, and builds TypeScript tarballs from the checked-in lockfiles. Source mode qualifies source integration, not registry publication. Its local tarballs can satisfy package dependencies that are not published yet. Run registry mode against the selected packages before treating registry installability as proven. Build logs and identities retain actual resolved harness versions; unpinned upstream ranges can resolve differently on another date. + +For a published prerelease, pass `--catalog-json` with the catalog resource extracted from that exact catalog wheel. Published prerelease packages may use stamped metadata while the committed catalog can retain the stable base version. Keep the reference checkout at the matching source revision: the gate still checks its complete adapter denominator and descriptor bytes against the selected packages, then verifies the installed catalog's full hash. It preserves Python and npm versions independently (for example, `0.6.0rc1` and `0.6.0-rc.1`). To obtain the resource without importing any harness, run: + +```bash +uv tool run --from pip pip download --no-deps --only-binary=:all: \ + --dest "$WORK/catalog" "nemo-fabric-adapter-catalog==$PACKAGE_VERSION" +python3 - "$WORK/catalog" <<'PY' +from pathlib import Path +from zipfile import ZipFile +import sys +directory = Path(sys.argv[1]) +with ZipFile(next(directory.glob('*.whl'))) as wheel: + (directory / 'catalog.json').write_bytes( + wheel.read('nemo_fabric_adapter_catalog/catalog.json')) +PY +``` + +Use `--mode registry --version "$PACKAGE_VERSION" --catalog-json "$WORK/catalog/catalog.json"` with the other plan arguments below. Use a fresh catalog directory per package version. + +The denominator is every descriptor in the checked-in metadata catalog, including both NOOA variants and remote-agent. Workflow-only adapters remain unsupported until an explicit registered target is selected with the SDK bridge; their catalog target IDs remain listed in the report. The separate BenchAgent entry exercises standalone NOOA. Future catalog entries appear automatically and need a reviewed installation recipe/provider profile. They are not silently removed when no harness or credential is available. The builder derives Python extras and TypeScript optional peers from package manifests; a few harnesses deliberately installed outside adapter dependencies have explicit tested recipes in `prepare.py`. Review those recipes when the supported harness pins change. + +The OpenHands recipe uses the adapter's documented compatible stack: +`openhands-sdk==1.50.0`, `openhands-tools==1.50.0`, `litellm==1.100.1`, and +`mcp==1.28.1`. Newer incompatible LiteLLM/MCP combinations can surface as an +SDK service-unavailable error before any provider request. Inspect retained +stderr and installed versions before attributing that error to inference. +The image runs `pip check`. Task dependencies such as gRPC/protobuf resolve +normally; their observed versions remain in the evidence without becoming +extra pins or integration pass criteria. + +## Serve A Local Checkpoint Or Select A Provider + +A filesystem checkpoint path is not an inference API. Start with an existing operator-managed NIM/vLLM or other compatible server for that checkpoint, using an allocated GPU and the server's documented serving command. This gate deliberately does not load checkpoints, install serving infrastructure, allocate GPUs, or modify existing model services. Validate these prerequisites: + +- The endpoint advertises the exact served model ID and supports tool calling. Responses, Chat Completions, and Anthropic Messages are different protocols. Check the API description and exercise the selected harness; an advertised route alone is not a passing integration test. +- The task container can reach the endpoint. Its `127.0.0.1` is the container, not the host. Keep a local server bound to loopback by default and arrange an operator-approved container route/proxy if needed. Do not expose a model service publicly to make the gate work. +- HTTPS is the default for remote endpoints. For an explicitly trusted private HTTP endpoint, name the exact host with `--trusted-http-host`; it is also forwarded to harnesses that advertise `insecure_http_model_hosts`. This transmits prompts and any credential without TLS to that host. +- Export the selected credential variable through your usual secret source. Never put the value in the plan, command arguments, Docker build arguments, settings, URLs, or committed files. For a confirmed unauthenticated endpoint, a nonsecret `MODEL_API_KEY=unused` satisfies clients that require a key. It is not an authentication test. + +Claude needs Anthropic Messages, Codex normally uses Responses, and most other local coding harnesses use Chat Completions. `--endpoint-kind openai-and-anthropic` permits the same endpoint to be attempted through both families. Kiro requires its native provider, authentication and advertised model. Droid uses provider `factory`; its supported native BYOK profile can route to a custom endpoint without setting normalized model `base_url`. + +For Kiro, keep the row prerequisite-blocked (skipped execution) unless the +native CLI, `KIRO_API_KEY`, and a Kiro-advertised model are available. Kiro does +not support BYOM through this adapter: do not use the local Nemotron endpoint, +a dummy `MODEL_API_KEY`, or an experimental protocol bridge. To enable a native +trial, select `model_provider=kiro`, the advertised model ID, `base_url=null`, +and `api_key_env=KIRO_API_KEY`, then mark the prepared row runnable. The runner +checks credential presence before task installation or launch; only the secret +variable name belongs in the plan. Missing credentials retain a blocked row, +not an integration failure or a full-catalog pass. + +For Remote Agent HTTP integration, explicitly set its row to `action=run`, clear +normalized `base_url`, and configure `harness_settings` with `base_url` and +`api_type=openai-completions` for a compatible endpoint. Keep the actual served +model ID. This exercises the declared HTTP adapter contract; autonomous remote +workspace/tool execution remains untested. Task reward zero does not negate a +successful round trip. + +For Droid BYOK, the prepared image includes CLI `droid@0.233.0` and the adapter's +pinned SDK. Extend its isolated task image with native +`/root/.factory/settings.json`, following the [Droid BYOK guide](../../../../docs/integrations/harness/droid.mdx#optional-use-a-custom-byok-model). +Set `cloudSessionSync=false` and define the real served model in `customModels` +using provider `generic-chat-completion-api`, the endpoint `baseUrl`, and an +`apiKey` environment reference. Select `custom:` in the plan, +`model_provider=factory`, `base_url=null`, and `api_key_env=FACTORY_API_KEY`. +For an explicitly unauthenticated local endpoint only, the reference can also +be `${FACTORY_API_KEY}` with dummy value `EMPTY`; this is not authentication +coverage. Keep that profile confined to the disposable task image, record its +Dockerfile/settings and final image identity, then mark the row runnable. Do +not mount ambient Factory configuration or credentials. + +Hermes uses its upstream-supported Python 3.14 runtime and source/assets layout; +its broader package metadata also supports older updaters, not older runtimes. +Use a prepared task image with an isolated 3.14 environment, its frozen core + +`mcp` dependency graph, and the exact supported source installation. Do not +bypass its wheel/sdist build guard. Set the row's `task_python` to that +environment's executable. Harbor may use a different compatible host Python; +the runner and identity/descriptor probes use `task_python`, while the upstream +task verifier keeps its original interpreter and instructions. No exact match +between host and task Python versions is required. + +Fabric runtime/catalog/adapter packages must remain noneditable. A pinned +Hermes source installation is supported even when its metadata says editable: +record `{"hermes-agent":""}` in +`/opt/qualification-harness-sources.json`. The identity collector records its +actual Git revision, source URL and dirty state; the gate requires a matching +clean source revision before and after the trial. The complete dependency and +interpreter identities remain evidence. Read-only image permissions are useful +but are not a sandbox against the task's root user. + +The Codex profile uses `danger-full-access` and `deny_all` **inside the gate-created Harbor task container**. Harbor supplies the isolation boundary; nested `bwrap` sandboxing requires user namespaces that many Docker hosts prohibit. Do not reuse this profile on a host workspace or attach host credentials, Docker sockets, or unrelated files to the task. The gate never enables privileged Docker mode or changes host namespace policy. + +Estimate paid-provider cost before running, using the selected model's current prices, expected prompt/output tokens per trial, and the number of runnable entries. Supply the estimate and assumptions explicitly. The estimate is recorded evidence, not an enforced spending cap. The deadline bounds runtime, not token billing. Never retry silently or infer a zero cost from unavailable accounting. + +## Plan, Prepare, And Run + +Run **one shared task across the harnesses**, never the full TB2 dataset. The baseline is [`kv-store-grpc`](https://github.com/harbor-framework/terminal-bench-2/tree/2fd12b88aafdd04a52c298e3940bcb189f9766d6/kv-store-grpc): dependency installation, protocol definition, generated code, server implementation, and a background service checked with real RPCs. Fetch the pinned task outside the NeMo Fabric checkout: + +```bash +WORK=/path/to/private/qualification +mkdir -m 700 "$WORK" +git clone --no-checkout https://github.com/harbor-framework/terminal-bench-2.git "$WORK/tb2" +git -C "$WORK/tb2" sparse-checkout set kv-store-grpc +git -C "$WORK/tb2" checkout 2fd12b88aafdd04a52c298e3940bcb189f9766d6 +``` + +The gate requires `--task` pointing to that single directory. It records a hash of the task inputs and verifier, rejects a dataset root, and rejects changed files after planning. Preparation builds the task's original Dockerfile before adding NeMo Fabric and each harness. Harbor builds from the prepared image rather than pulling the task-only `docker_image`. The instruction, task assets, resources, and upstream verifier stay intact; the oracle solution is excluded from staging. A wrapper collects the agent's final package and selection evidence before running the upstream verifier, whose own dependency installation is outside the agent's execution. Fabric package/wheel identity, installed Python harness distributions, npm harness packages, interpreter/source identity, catalog and descriptor provenance remain protected. Other Python package changes are recorded, allowing task setup to install or replace its own dependencies. If such changes break execution, lifecycle/result checks still fail. The original base and final image IDs are retained. This integration overlay is not an unmodified leaderboard submission. + +Set the version, compatible Harbor pin, served model ID, and container-reachable endpoint for your environment. The following example uses an already running local NIM endpoint and a nonsecret placeholder key: + +```bash +QUALIFY=.agents/skills/qualify-harbor/scripts +export MODEL_API_KEY=unused + +python3 "$QUALIFY/qualify.py" plan \ + --output "$WORK/plan.json" \ + --task "$WORK/tb2/kv-store-grpc" \ + --mode source --version 0.6.0 \ + --harbor-version 0.24.1.dev202610250408 \ + --model nvidia/nemotron-3-ultra-550b-a55b \ + --base-url http://model-gateway:8000/v1 \ + --trusted-http-host model-gateway \ + --endpoint-kind openai-and-anthropic \ + --backend 'Operator-managed NIM; record the exact image/version' \ + --estimated-cost-usd 0 \ + --cost-basis 'Existing local inference; no metered provider charge; hardware excluded' \ + --timeout 900 + +# Inspect and configure rows before preparing; retain the full denominator. +python3 "$QUALIFY/prepare.py" --plan "$WORK/plan.json" --work "$WORK/environment" + +"$WORK/environment/host/bin/python" "$QUALIFY/qualify.py" run \ + --plan "$WORK/plan.json" --environment "$WORK/environment/environment.json" \ + --output "$WORK/full-run" +``` + +Preparation and execution each require new directories and never overwrite previous evidence. Pass `--host-python /absolute/path/to/python` to `prepare.py` to use an existing Python 3.12+ interpreter rather than downloading Python 3.12. A base installation failure writes `environment.json` and logs before exiting; run the gate with `python3` in that case to produce a fully blocked report. `prepare.py` continues past individual harness installation failures, which become failed matrix rows. Retain the environment directory while inspecting or reproducing a run. Images are tagged with the source revision; record their immutable IDs in the report. No image is pushed. + +For a keyed HTTPS provider, export `MODEL_API_KEY` securely, use its model slug and `--base-url https://...`, supply a nonzero cost estimate with token/pricing assumptions, and set the appropriate endpoint kind. Each plan row can override `model`, `model_provider`, `base_url`, `api_key_env`, `harness_settings`, `telemetry`, and `image`. A custom task image must contain the matching runtime/catalog/adapter, its harness, `/qualification/identity.py`, and `/opt/qualification-identity.json` produced by that script. Fabric packages must be noneditable; supported pinned Hermes source installs follow the provenance rules above. Set a prerequisite row's `action` to `run` only after providing those dependencies. Do not pass a generic local model to a proprietary provider under a fabricated model alias. + +For an OpenAI-compatible endpoint with a vendor-prefixed served ID, set that row’s `model_provider` to `openai` without changing `model`. This requires the SDK bridge containing [PR #522](https://github.com/NVIDIA/NeMo-Fabric/pull/522); generate the plan with `--bridge sdk`. Native Harbor’s equivalent option is deferred to a separate upstream change. The selected provider is retained in both admission input and the report. Older SDKs remain usable when the override is omitted. + +For adapter-declared model settings, use the SDK bridge and set that plan row's +`model_settings` object. It projects to `fabric_model_settings` and remains +subject to the adapter model schema. For example, an explicitly unpriced +Mini-SWE model uses `{"cost_tracking":"disabled"}` with a positive `max_turns` +on the same row. This disables monetary budgeting and keeps cost unavailable; +it does not report a zero cost. `max_turns` projects to `fabric_max_turns`. +The bridge and runner must both contain the model-settings transport from #522. + +For a registered workflow, generate the plan with `--bridge sdk` and configure +that adapter's row with `workflow_target_id` and optional `workflow_settings`. +For example, the generic NOOA adapter can select its registered coding target: + +```json +{ + "workflow_target_id": "nvidia.nooa.coding-agent", + "workflow_settings": {}, + "action": "run", + "reason": "" +} +``` + +Keep the row's existing adapter ID, catalog record, model, credentials reference, +and all other catalog rows. Do not substitute the separate BenchAgent adapter. +The target must exist in the exact catalog snapshot, bind to the selected adapter, +and have an unchanged descriptor. Settings must be JSON without inline +credentials; the SDK validates them against the target's schema before setup. +This requires matching host/task SDKs with registered-workflow Harbor support. +Native Harbor workflow transport is not qualified by this path. + +The command and installed-image preflight transport the same target/settings. +Task evidence retains the target's factory and installed descriptor provenance; +the admission fingerprint covers both normalized adapter and target descriptors. +A factory, settings schema, selection or origin mismatch fails integration. +Workflow selection does not claim complete usage, MCP, skills or subagent +coverage. Unknown accounting remains unknown and is checked for exact transport. + +A developer subset retains every other row as excluded and sets `scope=developer_subset`: + +```bash +"$WORK/environment/host/bin/python" "$QUALIFY/qualify.py" run \ + --plan "$WORK/plan.json" --environment "$WORK/environment/environment.json" \ + --output "$WORK/pi-reproduction" --only nvidia.fabric.pi +``` + +Exit zero means the requested subset passed, or every entry passed in a full run. A subset never sets `full_catalog_passed=true`. Nonzero means failure, blocked coverage, cancellation, or invalid prerequisites. + +The gate tests integration, not model task quality. A valid reward of zero does +not fail a completed Harbor/Fabric/harness round trip. The report keeps task +reward and `checks.task_success` as informational evidence. Missing, malformed, +or nonfinite reward evidence still fails verifier completion; Harbor exceptions, +failed invocations, timeouts, and cleanup failures remain failures. + +Harness accounting availability is a separate capability observation. Missing +complete counters remain unknown, with a recorded limitation; they do not fail +integration when Fabric transports the same normalized unknown values into +Harbor. `checks.usage_available` is informational. A mismatch in result identity +or accounting transport still fails, regardless of reward or usage availability. +Reports record these pass criteria explicitly and show reward alongside the +integration outcome. A subset cannot establish full-catalog coverage. + +## Evidence And Triage + +`report.json` is the sanitized machine summary; `summary.md` is the human matrix. Both are rewritten atomically after each row, with pending rows retained if interrupted. `plan.json` retains the full descriptor snapshot. Each attempted row includes a reproducible command, timestamps, image identity, host/task package and harness versions, runtime wheel hash, observed runtime/invocation/adapter identity, selected workflow target/settings when present, reward, usage, capability limitations, and cleanup status. Catalog hashes cover raw metadata; Harbor admission hashes cover the core-normalized adapter descriptor and selected target descriptor. The gate records both and compares the uploaded admission hash against SDK metadata inspection, never directly against the raw catalog hash. Raw task artifacts stay in the private run directory. The gate redacts known credential values from text output, but binary artifacts and unknown secrets are not shareable by assumption; inspect before exporting them. + +Source-built task images remove the build tree after packaging. The native core can discover its compile-time source directory when that directory still exists; identical descriptor bytes do not guarantee the intended execution origin. A metadata-only task-container preflight verifies installed Python package data or the explicitly selected npm package before any model call. The verifier records the actual task's descriptor provenance again. A bundled/source-tree origin or drift between those observations fails admission even when versions and fingerprints match. + +Use the failure categories to choose the next investigation: + +| Category | Start With | +| --- | --- | +| Prerequisite/install | `environment.json`, `task-build.log`, `base-build.log`, `host-install.log`, per-image `install.log`; package availability, Python/Node compatibility, harness pins | +| Auth/provider/model | Endpoint protocol, exact served ID, credential environment reference, harness error code, and task's model config | +| Adapter admission/fingerprint | Selected catalog SHA, uploaded `fabric-run-*.json`, installed descriptor, package versions, task identity | +| Transport/runner | Harbor trial exception, `agent/fabric-result-*.json` or runner-error envelope, runner stdout/stderr | +| Harbor lifecycle/verifier | Trial `result.json`, `exception_info`, verifier output, reward file; valid reward zero is an informational task outcome, missing/invalid reward is a failure | +| Usage | Invocation usage and Harbor accounting; missing complete counters are a harness limitation, transport mismatches fail; unknown is not zero | +| Telemetry/export | Relay validation, canonical ATIF schema, promoted `trajectory.json`, artifacts; baseline runs do not test telemetry | +| Timeout/cancellation/cleanup | Recorded deadline/exit code, partial logs, owned Compose resources and `cleanup.log` | +| Infrastructure | Connectivity, image fetch/build, storage exhaustion, server/rate-limit errors; reproduce before declaring a product regression | + +Harbor normally deletes task containers. After completion or interruption, the gate removes only Compose containers/networks whose project label starts with this run's randomly allocated task prefix, plus metadata-probe containers carrying its exact qualification label. It reads resource labels without collecting container environments. It never prunes Docker globally or stops a model service. SIGINT is forwarded to its own process group, followed by bounded TERM/KILL escalation. Abrupt host loss or SIGKILL cannot guarantee cleanup; retain the run's task prefix and inspect its exact labels before manual cleanup. Images and build evidence are deliberately retained for reproduction. The gate does not retry trials; harness-native retries can still occur within the recorded deadline and remain visible in the logs. + +For deterministic gate and integration checks, use the repository environment: + +```bash +uv run --no-sync pytest tests/scripts/test_harbor_qualification.py \ + tests/integrations/test_harbor_runner.py tests/integrations/test_harbor_mcp.py \ + tests/integrations/test_harbor_skills.py tests/integrations/test_harbor_telemetry.py \ + tests/python/test_harbor_accounting.py tests/python/test_harbor_credentials.py \ + tests/python/test_adapter_capabilities.py +``` + +The baseline leaves MCP, skills, subagents, and telemetry untested where declared, or unsupported/not declared where metadata does not advertise them. For optional Relay qualification, configure a supported row with `telemetry=relay` and a compatible image containing Relay and any required harness extension; the gate validates export and canonical ATIF. Use the [SWE-Bench examples](../../../../examples/harbor/swebench/README.md) for live MCP/skills/telemetry scenarios and attach their exact commands and artifacts separately. Subagent functionality needs harness-specific evidence; neither one task nor ATIF schema validity establishes it. Historical flaky CI tests are context, not present-run exemptions. diff --git a/.agents/skills/qualify-harbor/references/setup.md b/.agents/skills/qualify-harbor/references/setup.md new file mode 100644 index 000000000..011f4bd2f --- /dev/null +++ b/.agents/skills/qualify-harbor/references/setup.md @@ -0,0 +1,73 @@ + + +# Interactive Harbor Setup + +Use this conversation when starting a new qualification run. A supplied plan or explicit provider choice takes precedence; reuse known answers and existing authorization. Planning alone does not build images, spend provider credit, or launch a task. + +## Choose How Models Are Supplied + +Start with: **“Use one shared BYOM endpoint for compatible adapters, or individual provider credentials?”** Recommend the shared endpoint when the user already has one served. Explain that native-provider exceptions can be added afterward. Do not begin with command flags, 18 adapter forms, or a request for secret values. + +Establish the adapter scope from the request. Offer all compatible adapters or a named subset if scope is missing. Keep every catalog row in the eventual plan, including adapters outside the selected subset. Derive version/source, execution host and task from existing context where possible; propose the current committed checkout and the runbook baseline when appropriate instead of asking users to supply Git hashes. Ask for a host or version decision when it cannot be inferred safely. + +### One Shared BYOM Endpoint + +Collect only missing connection details: reachable base URL, exact served model ID, supported API protocols, and credential environment-variable name. For example, ask for the URL and model first, then resolve authentication and protocol uncertainty. Reuse a known NIM/provider configuration instead of asking again. + +A Chat Completions endpoint does not establish Responses or Anthropic Messages support. Check the pinned adapter documentation/model schema and the endpoint's documented capabilities or authorized preflight evidence. Keep uncertain protocols as “needs verification.” Do not send paid test prompts merely to complete the questionnaire. + +If the user requested only compatible BYOM adapters, skip native-only/protocol-incompatible routes without requesting their keys again. Offer optional native exceptions in the preview only when broader coverage is wanted. + +Some custom-model routes need metadata beyond a URL and key, such as protocol, token budgets or accepted input formats. First verify that the pinned adapter and public Harbor bridge can express the required fields, then ask only for missing shared metadata. If adapter docs, schema or bridge transport disagree, report a blocked contract mismatch rather than inventing plan fields, fabricated model aliases or zero prices. Pi gateway models are one case to check; a successful known-model profile does not prove arbitrary model registration. + +Use the same endpoint, full model ID and credential reference wherever the adapter supports them. Select its real transport/provider, such as an explicit `openai` override for a vendor-prefixed model ID, rather than rewriting the served ID. Prefer the SDK bridge when the requested profiles require its provider, model-settings or workflow options; use the requested bridge when compatible and explain any required change before executing. + +### Individual Provider Credentials + +Ask which providers/models the user wants for the selected adapters. Group compatible adapters by provider so one credential reference can serve several adapters; permit per-adapter overrides. Show available credential **names and presence only** on the execution host, and ask for unresolved model/provider choices. Do not assume that a native account subscription provides a usable API key or that credentials on the local machine exist on a remote execution host. + +For missing credentials, offer secure provisioning through the user's normal secret source or skipping the affected adapters. If skipping is already authorized, record the reason and continue. Do not repeatedly prompt for skipped providers. Label user-reported presence separately from an actual execution-host check. A populated environment variable proves availability, not validity or account access. In this mode, the global model/endpoint arguments are only a template for plan generation: configure every selected row explicitly so the seed provider/model cannot leak into unrelated adapters. + +## Keep Secrets Outside The Conversation + +Ask for variable names such as `MODEL_API_KEY`, `ANTHROPIC_API_KEY` or `KIRO_API_KEY`, never key values. Check only whether the named variable is nonempty in the environment that will launch Harbor, without displaying its value. Explain that Harbor forwards selected variables into disposable task containers; avoid forwarding unrelated credentials. + +Use the existing managed environment transport. Plans, previews, commands shown to the user, settings, image builds, committed files and reports contain references only. Do not ask users to paste or upload keys. If a value arrives unsolicited, do not repeat it in the plan or summary. + +For a confirmed unauthenticated local endpoint, explain the nonsecret placeholder required by clients and record that this is not authentication coverage. Do not replace a missing real provider credential with a dummy key or infer unauthenticated access from a missing variable. + +## Resolve Adapter Routes For The User + +Use the exact catalog and installed-version contracts, with the profiles in [the runbook](runbook.md). The table below identifies decisions, not a static allowlist; new adapters still come from the complete catalog. + +| Route | Setup behavior | +| --- | --- | +| Direct compatible API | Apply the shared endpoint/key reference or the chosen provider profile. Verify the required protocol; in this gate Claude needs Anthropic Messages and Codex normally uses Responses. | +| Kiro | Native Kiro provider and advertised model only; `KIRO_API_KEY` is required. In BYOM mode, offer native credentials as an optional exception or skip. Never create an experimental BYOM bridge. | +| Droid custom BYOK | Prepare the supported native custom-model profile/image described in the runbook. Its profile can use the shared endpoint, but normalized `base_url` is not the route. Preserve its credential-reference contract and record the profile/image identity. | +| Remote Agent | Prepare the compatible HTTP profile in the runbook. Explain that HTTP-contract coverage does not demonstrate autonomous remote workspace/tool execution. | +| Registered workflow | Select a catalog target appropriate to the task, such as NOOA CodingAgent for a coding task, and validate its settings. Do not substitute a different adapter to make the row pass. Ask only when the target choice is genuinely ambiguous. | +| Runtime or budgeting prerequisites | Prepare a supported interpreter/source image where needed, such as Hermes. For an unpriced Mini-SWE model, make the cost-tracking opt-out and positive turn limit explicit in the preview; do not silently report zero cost or remove the bound. | + +Compatibility, credentials and installation readiness are separate. A valid provider key does not satisfy a missing harness/profile; a supported BYOM profile does not prove that the endpoint implements the required protocol. Prepare supported profiles rather than treating missing setup as unsupported capability. A complete supported recipe makes a row eligible for preparation; an installed, checked image/profile establishes execution readiness. Do not label an unbuilt image or an unverified endpoint ready to run. + +## Preview Coverage, Then Save The Plan + +Summarize the selected package/source version, bridge, execution host, single task, provider cost estimate/basis and execution bounds. Include a compact table with one row per catalog adapter: + +| Adapter | Model / route | Credential reference | Readiness / reason | +| --- | --- | --- | --- | +| A selected Chat Completions adapter | Shared model endpoint | `MODEL_API_KEY`: availability checked | Ready to run; endpoint/profile checks complete | +| Kiro without native credentials | Native provider required; BYOM unsupported | `KIRO_API_KEY`: missing | Skip; credential prerequisite | +| An adapter outside a requested subset | Outside selected scope | Not forwarded | Excluded | + +The example is a layout, not a coverage result. Use “needs verification/setup” while prerequisites remain unresolved, “ready to prepare” for a complete recipe, and “ready to run” only after actual preparation and checks. Separate these from “skipped.” Ask only for unresolved choices, such as adding a native provider exception, provisioning a missing key, accepting a bounded unpriced-model policy, or proceeding with a subset. Reuse prior run/spending authorization; do not add another approval step when the concrete plan is already authorized. An unanswered required choice remains pending, not implicitly accepted. + +A conversational preview can be incomplete. If the exact source/package identity, real task path/hash, model selections or required planning inputs are missing, show the preview and ask for the next missing input; do not invent placeholders or claim an executable plan was validated. Respect plan-only/no-call requests during this setup. + +Once those inputs are available, generate `plan.json` using `qualify.py plan` and apply the chosen per-row profiles on the user's behalf. Preserve catalog records and rows, target bindings, the original task/verifier/resources, credential names, reasons, cost basis and bounds. Validate the final plan with the gate's existing `validate_plan` before preparation. Do not add new plan fields just to record this conversation or store secret values. Show the saved plan path and the resulting readiness/coverage counts. Human-facing “skip” retains the gate’s existing actions: `blocked` for missing prerequisites, `unsupported` for an unavailable route, and `excluded` outside the requested subset. Use `run` for selected rows with a complete preparation recipe; do not introduce a `skipped` plan action or launch them before preparation succeeds. + +Once prerequisites and authorization are settled, follow the runbook to prepare and execute. A missing native key produces skipped coverage, not a Fabric integration defect; retain the gate's prerequisite-blocked outcome and do not claim a full-catalog pass. Keep model task reward and complete usage availability informational under the existing integration criterion. Replaying a configured plan should not restart onboarding. diff --git a/.agents/skills/qualify-harbor/scripts/identity.py b/.agents/skills/qualify-harbor/scripts/identity.py new file mode 100644 index 000000000..44e0bd220 --- /dev/null +++ b/.agents/skills/qualify-harbor/scripts/identity.py @@ -0,0 +1,160 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Write package provenance without collecting environment variables or credentials.""" + +from __future__ import annotations + +import importlib.metadata +from importlib.resources import files +import hashlib +import json +import platform +from pathlib import Path +import shutil +import subprocess +import sys +from urllib.parse import unquote, urlsplit + + +def selection(payload: dict) -> dict: + from nemo_fabric import Fabric, FabricConfig + from nemo_fabric.capabilities import _adapter_descriptor_sha256 + + plan = Fabric().plan( + FabricConfig.from_mapping(payload["config"]), + base_dir=payload["config_base_dir"], + ) + resolved = plan.to_mapping() + descriptor = resolved["adapter_descriptor"] + selected = { + "adapter_id": descriptor["descriptor"]["adapter_id"], + "python_prefix": sys.prefix, + "provenance": descriptor["provenance"], + "runner": descriptor["descriptor"]["runner"], + "admission_sha256": _adapter_descriptor_sha256(plan), + } + target = resolved.get("adapter_target_descriptor") + if target is not None: + selected["workflow"] = { + "target_id": target["descriptor"]["id"], + "entrypoint": target["descriptor"]["spec"]["entrypoint"], + "provenance": target["provenance"], + } + return selected + + +def identity() -> dict: + import nemo_fabric + + packages = {} + editable = [] + editable_sources = {} + archives = {} + for distribution in importlib.metadata.distributions(): + name = distribution.metadata["Name"] + packages[name] = distribution.version + direct = json.loads(distribution.read_text("direct_url.json") or "{}") + if direct.get("dir_info", {}).get("editable"): + editable.append(name) + url = urlsplit(direct.get("url", "")) + if url.scheme == "file": + source = Path(unquote(url.path)) + revision = subprocess.run( + ["git", "-C", str(source), "rev-parse", "HEAD"], + capture_output=True, + text=True, + check=False, + timeout=30, + ) + status = subprocess.run( + ["git", "-C", str(source), "status", "--porcelain"], + capture_output=True, + text=True, + check=False, + timeout=30, + ) + editable_sources[name] = { + "url": direct["url"], + "revision": revision.stdout.strip() + if revision.returncode == 0 + else None, + "dirty": status.returncode != 0 or bool(status.stdout.strip()), + } + if name.startswith("nemo-fabric"): + archives[name] = ( + direct.get("archive_info", {}).get("hashes", {}).get("sha256") + ) + npm = ( + subprocess.run( + ["npm", "ls", "--prefix", "/opt/harness", "--json", "--depth=0"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + if shutil.which("npm") + else None + ) + # npm may include local paths and diagnostic prose; retain versions only. + dependencies = json.loads(npm.stdout or "{}").get("dependencies", {}) if npm else {} + global_npm = ( + subprocess.run( + ["npm", "ls", "--global", "--json", "--depth=0"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + if npm + else None + ) + global_dependencies = ( + json.loads(global_npm.stdout or "{}").get("dependencies", {}) + if global_npm + else {} + ) + return { + "python": platform.python_version(), + "platform": platform.platform(), + "executable": str(Path(sys.executable).resolve()), + "runtime_origin": nemo_fabric.__file__, + "packages": dict(sorted(packages.items())), + "editable": sorted(editable), + "editable_sources": editable_sources, + "wheel_sha256": archives, + "catalog_sha256": hashlib.sha256( + json.dumps( + json.loads( + files("nemo_fabric_adapter_catalog") + .joinpath("catalog.json") + .read_text() + ), + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + ).encode() + ).hexdigest(), + "npm": {name: info.get("version") for name, info in dependencies.items()}, + "npm_global": { + name: info.get("version") for name, info in global_dependencies.items() + }, + "harness_sources": json.loads( + Path("/opt/qualification-harness-sources.json").read_text() + ) + if Path("/opt/qualification-harness-sources.json").exists() + else {}, + } + + +if __name__ == "__main__": + if len(sys.argv) == 3 and sys.argv[1] == "--selection": + payload = ( + json.load(sys.stdin) + if sys.argv[2] == "-" + else json.loads(Path(sys.argv[2]).read_text()) + ) + result = selection(payload) + else: + result = identity() + print(json.dumps(result, indent=2, sort_keys=True)) diff --git a/.agents/skills/qualify-harbor/scripts/prepare.py b/.agents/skills/qualify-harbor/scripts/prepare.py new file mode 100644 index 000000000..fa5ed7d49 --- /dev/null +++ b/.agents/skills/qualify-harbor/scripts/prepare.py @@ -0,0 +1,288 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Build isolated host/task installs for the Harbor qualification plan.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import shutil +import signal +import subprocess +import tarfile +from pathlib import Path + +from qualify import ROOT, execute, load, save, stop, validate_plan + + +def base_environment(plan: dict, work: Path, host_python: str) -> dict: + context = work / "base" + context.mkdir() + archive = context / "source.tar" + with archive.open("wb") as stream: + subprocess.run( + ["git", "archive", plan["source_sha"]], + cwd=ROOT, + stdout=stream, + check=True, + timeout=120, + ) + source = context / "source" + source.mkdir() + with tarfile.open(archive) as bundle: + bundle.extractall(source, filter="data") + archive.unlink() + shutil.copyfile(Path(__file__).parent / "identity.py", context / "identity.py") + shutil.copyfile( + Path(__file__).parents[1] / "assets/Dockerfile", context / "Dockerfile" + ) + run_id = hashlib.sha256(str(work.resolve()).encode()).hexdigest()[:12] + tag = f"fabric-qualification-base:{plan['source_sha'][:12]}-{plan['mode']}-{run_id}" + task_base = tag + "-task" + if execute( + [ + "docker", + "build", + "--progress=plain", + "-t", + task_base, + str(Path(plan["task"]["path"]) / "environment"), + ], + work / "task-build.log", + 900, + ): + raise RuntimeError("task environment build failed; inspect task-build.log") + code = execute( + [ + "docker", + "build", + "--progress=plain", + "--build-arg", + f"FABRIC_VERSION={plan['version']}", + "--build-arg", + f"INSTALL_MODE={plan['mode']}", + "--build-arg", + f"TASK_IMAGE={task_base}", + "-t", + tag, + str(context), + ], + work / "base-build.log", + 3600, + ) + if code: + raise RuntimeError("base installation failed; inspect base-build.log") + wheels = work / "wheels" + container = subprocess.check_output( + ["docker", "create", tag], text=True, timeout=30 + ).strip() + try: + subprocess.run( + ["docker", "cp", f"{container}:/wheels", str(wheels)], + check=True, + timeout=120, + ) + finally: + subprocess.run( + ["docker", "rm", container], check=True, capture_output=True, timeout=30 + ) + venv = work / "host" + uv = shutil.which("uv") or str(Path.home() / ".local/bin/uv") + for command in ( + [uv, "venv", "--python", host_python, str(venv)], + [str(venv / "bin/python"), "-m", "ensurepip"], + [ + str(venv / "bin/python"), + "-m", + "pip", + "install", + "--disable-pip-version-check", + "--retries", + "1", + "--timeout", + "30", + "--find-links", + str(wheels), + *[ + str(path) + for path in sorted(wheels.glob("*.whl")) + if path.name.split(f"-{plan['version']}-")[0] + in { + "nemo_fabric", + "nemo_fabric_runtime", + "nemo_fabric_adapter_catalog", + "nemo_fabric_adapter_contract", + "nemo_fabric_adapters_common", + } + ], + f"harbor=={plan['harbor_version']}", + ], + ): + if execute(command, work / "host-install.log", 900): + raise RuntimeError("host installation failed; inspect host-install.log") + return { + "base_image": tag, + "task_base_image": task_base, + "task_base_image_id": subprocess.check_output( + ["docker", "image", "inspect", "--format", "{{.Id}}", task_base], + text=True, + timeout=30, + ).strip(), + "python": str(venv / "bin/python"), + "wheels": str(wheels), + "wheel_sha256": { + path.name: hashlib.sha256(path.read_bytes()).hexdigest() + for path in wheels.glob("*.whl") + }, + } + + +def adapter_dockerfile(plan: dict, row: dict, base: str) -> str: + record = row["catalog"] + leaf = Path(record["source"]).parent.name + name = record["package"]["name"] + version = record["package"]["version"] + lines = [f"FROM {base}"] + if record["ecosystem"] == "python": + manifest = load_manifest( + ROOT / Path(record["source"]).parent / "pyproject.toml" + ) + extra = ( + "[harness]" + if "harness" in manifest["project"].get("optional-dependencies", {}) + else "" + ) + requirement = ( + f"{name}{extra} @ file:///wheels/{name.replace('-', '_')}-{version}-py3-none-any.whl" + if plan["mode"] == "source" + else f"{name}{extra}=={version}" + ) + lines.append(f'RUN pip install --find-links /wheels "{requirement}"') + else: + directory = f"/src/adapters/typescript/{leaf}" + if plan["mode"] == "source": + lines.extend( + [ + "RUN npm ci --prefix /src/adapter-contract/typescript --ignore-scripts", + f"RUN npm ci --prefix /src/adapters/typescript --workspace {name} --include-workspace-root --ignore-scripts", + "RUN npm run build --prefix /src/adapter-contract/typescript", + "RUN npm run build --prefix /src/adapters/typescript --workspace nemo-fabric-adapters-common", + f"RUN npm run build --prefix /src/adapters/typescript --workspace {name}", + "RUN mkdir /packs && npm pack --prefix /src/adapter-contract/typescript /src/adapter-contract/typescript --pack-destination /packs --ignore-scripts", + "RUN npm pack /src/adapters/typescript/common --pack-destination /packs --ignore-scripts", + f"RUN npm pack {directory} --pack-destination /packs --ignore-scripts", + "RUN mkdir -p /opt/harness && npm install --prefix /opt/harness /packs/*.tgz --omit=dev", + ] + ) + else: + lines.append( + f"RUN mkdir -p /opt/harness && npm install --prefix /opt/harness {name}@{version} --omit=dev" + ) + manifest = load(ROOT / Path(record["source"]).parent / "package.json") + peers = manifest.get("peerDependencies", {}) + tested = manifest.get("devDependencies", {}) + if peers: + specs = [f"{peer}@{tested.get(peer, pin)}" for peer, pin in peers.items()] + lines.append("RUN npm install --prefix /opt/harness " + " ".join(specs)) + # These harnesses deliberately are not dependencies of their adapter packages. + installs = { + "nvidia.fabric.droid": "npm install --global droid@0.233.0", + "nvidia.fabric.openhands": ( + 'pip install "openhands-sdk==1.50.0" "openhands-tools==1.50.0" ' + '"litellm==1.100.1" "mcp==1.28.1" && pip check' + ), + "nvidia.fabric.openclaw": "npm install --global openclaw@2026.9.4 --allow-scripts=openclaw", + "nvidia.fabric.cline": "npm install --prefix /opt/harness @cline/sdk@0.0.83", + "nvidia.fabric.strands": "npm install --global @strands-agents/cli@0.2.1", + "nvidia.fabric.opencode": "npm install --global bun@1.4.2", + } + if row["adapter_id"] in installs: + lines.append("RUN " + installs[row["adapter_id"]]) + lines.extend( + [ + "RUN rm -rf /src /packs", + "COPY identity.py /qualification/identity.py", + "RUN python /qualification/identity.py > /opt/qualification-identity.json", + 'LABEL org.nvidia.fabric.qualification="true"', + f'LABEL org.nvidia.fabric.source="{plan["source_sha"]}"', + "WORKDIR /app", + ] + ) + return "\n".join(lines) + "\n" + + +def load_manifest(path: Path) -> dict: + import tomllib + + return tomllib.loads(path.read_text()) + + +def main() -> None: + os.umask(0o077) + signal.signal(signal.SIGTERM, stop) + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--plan", required=True, type=Path) + parser.add_argument("--work", required=True, type=Path) + parser.add_argument( + "--host-python", + default="3.12", + help="Host interpreter version or existing executable; must support Harbor (Python >=3.12)", + ) + args = parser.parse_args() + plan = load(args.plan) + validate_plan(plan) + work = args.work.resolve() + work.mkdir(mode=0o700) # Never overwrite another run or environment. + environment = {"plan": plan, "adapters": {}} + save(work / "environment.json", environment) + try: + environment.update(base_environment(plan, work, args.host_python)) + except (OSError, RuntimeError, subprocess.SubprocessError) as error: + environment["error"] = str(error) + save(work / "environment.json", environment) + raise SystemExit(1) from error + save(work / "environment.json", environment) + for index, row in enumerate(plan["adapters"]): + if row["action"] != "run": + continue + directory = work / str(index) + directory.mkdir() + tag = ( + row.get("image") + or f"fabric-qualification:{environment['base_image'].split(':', 1)[1]}-{index}" + ) + if not row.get("image"): + shutil.copyfile( + Path(__file__).with_name("identity.py"), directory / "identity.py" + ) + (directory / "Dockerfile").write_text( + adapter_dockerfile(plan, row, environment["base_image"]) + ) + code = execute( + ["docker", "build", "--progress=plain", "-t", tag, str(directory)], + directory / "install.log", + 1800, + ) + else: + code = execute( + ["docker", "image", "inspect", tag], directory / "install.log", 30 + ) + environment["adapters"][row["adapter_id"]] = { + "image": tag, + "status": "failed" if code else "ready", + "install_exit_code": code, + "log": str(directory / "install.log"), + } + save(work / "environment.json", environment) + if code == 130: + raise SystemExit(130) + print( + f"{row['adapter_id']}: {'install failed' if code else 'ready'}", flush=True + ) + + +if __name__ == "__main__": + main() diff --git a/.agents/skills/qualify-harbor/scripts/qualify.py b/.agents/skills/qualify-harbor/scripts/qualify.py new file mode 100644 index 000000000..a5007d67f --- /dev/null +++ b/.agents/skills/qualify-harbor/scripts/qualify.py @@ -0,0 +1,1569 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Plan and run a bounded Harbor -> Fabric -> harness integration qualification.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import os +import re +import shlex +import shutil +import signal +import subprocess +import sys +import threading +import tomllib +import uuid +from datetime import datetime, timezone +from pathlib import Path +from urllib.parse import urlsplit + +ROOT = Path(__file__).resolve().parents[4] +CATALOG = Path( + "sdk/python/nemo-fabric-adapter-catalog/src/nemo_fabric_adapter_catalog/catalog.json" +) + + +def load(path: Path) -> dict: + return json.loads(path.read_text(encoding="utf-8")) + + +def save(path: Path, value: dict) -> None: + temporary = path.with_suffix(".tmp") + temporary.write_text( + json.dumps(value, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + temporary.replace(path) + + +def now() -> str: + return datetime.now(timezone.utc).isoformat() + + +def fingerprint(value: dict) -> str: + return hashlib.sha256( + json.dumps( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=False + ).encode() + ).hexdigest() + + +def sanitize(text: str, credential_names: tuple[str, ...] = ()) -> str: + for name, value in os.environ.items(): + if ( + value + and len(value) >= 4 + and ( + name in credential_names + or name + in { + "DOCKER_AUTH_CONFIG", + "AWS_ACCESS_KEY_ID", + "AZURE_STORAGE_CONNECTION_STRING", + } + or re.search( + r"(?:^|_)(?:KEY|TOKEN|SECRET|PASSWORD|CREDENTIALS?)$", name, re.I + ) + ) + ): + text = text.replace(value, "[REDACTED]") + text = re.sub( + r"(?i)(authorization[\"']?\s*[:=]\s*[\"']?(?:bearer\s+)?)[^\s\"',}]+", + r"\1[REDACTED]", + text, + ) + return re.sub(r"(https?://)[^/\s:@]+:[^/\s@]+@", r"\1[REDACTED]@", text) + + +def sanitize_data(value: object, credential_names: tuple[str, ...]) -> object: + """Redact string values without changing JSON types or structure.""" + if isinstance(value, str): + return sanitize(value, credential_names) + if isinstance(value, dict): + return { + key: sanitize_data(item, credential_names) for key, item in value.items() + } + if isinstance(value, list): + return [sanitize_data(item, credential_names) for item in value] + return value + + +def sanitize_artifact(text: str, suffix: str, credential_names: tuple[str, ...]) -> str: + """Preserve machine-readable trial evidence when scrubbing credential values.""" + if suffix in {".jsonl", ".ndjson"}: + return "\n".join( + sanitize_artifact(line, ".json", credential_names).removesuffix("\n") + for line in text.split("\n") + ) + if suffix == ".json": + try: + value = json.loads(text) + except ValueError: + return sanitize(text, credential_names) + cleaned = sanitize_data(value, credential_names) + if cleaned == value: + return text + return json.dumps(cleaned, ensure_ascii=False) + "\n" + return sanitize(text, credential_names) + + +def process_env(key_env: str | None = None) -> dict[str, str]: + names = ( + "PATH", + "HOME", + "TMPDIR", + "LANG", + "LC_ALL", + "SSL_CERT_FILE", + "SSL_CERT_DIR", + "DOCKER_HOST", + "DOCKER_CONTEXT", + ) + env = {name: os.environ[name] for name in names if name in os.environ} + env.update(PYTHONNOUSERSITE="1", UV_NO_CONFIG="1", PIP_CONFIG_FILE=os.devnull) + if key_env: + env[key_env] = os.environ[key_env] + return env + + +def execute( + command: list[str], + log: Path, + timeout: int, + *, + key_env: str | None = None, + cwd: Path | None = None, +) -> int: + """Bound the entire process group; redact before writing persistent output.""" + credential_names = (key_env,) if key_env else () + with log.open("a", encoding="utf-8") as stream: + stream.write(sanitize(shlex.join(command), credential_names) + "\n") + stream.flush() + process = subprocess.Popen( + command, + cwd=cwd, + env=process_env(key_env), + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + errors="replace", + start_new_session=True, + ) + + def drain() -> None: + assert process.stdout is not None + for line in process.stdout: + stream.write(sanitize(line, credential_names)) + stream.flush() + + reader = threading.Thread(target=drain, daemon=True) + reader.start() + try: + code = process.wait(timeout=timeout) + except (subprocess.TimeoutExpired, KeyboardInterrupt) as error: + code = 130 if isinstance(error, KeyboardInterrupt) else 124 + for sig, grace in ( + (signal.SIGINT, 30), + (signal.SIGTERM, 10), + (signal.SIGKILL, 5), + ): + try: + os.killpg(process.pid, sig) + except ProcessLookupError: + break + try: + process.wait(timeout=grace) + break + except subprocess.TimeoutExpired: + continue + reader.join(timeout=5) + return code + + +def plan_catalog(plan: dict) -> dict: + return plan.get("catalog_snapshot") or load(ROOT / CATALOG) + + +def task_identity(path: Path) -> dict: + """Identify exactly one Harbor task, including its unmodified verifier.""" + path = path.resolve() + for relative in ( + "task.toml", + "instruction.md", + "environment/Dockerfile", + "tests/test.sh", + ): + if not (path / relative).is_file(): + raise ValueError(f"single Harbor task missing {relative}: {path}") + config = tomllib.loads((path / "task.toml").read_text()) + files = {} + for item in sorted(path.rglob("*")): + relative = item.relative_to(path) + if relative.parts[0] == "solution" or "__pycache__" in relative.parts: + continue + if item.is_symlink(): + raise ValueError(f"task symlinks are unsupported: {relative}") + if item.is_file(): + files[str(relative)] = hashlib.sha256(item.read_bytes()).hexdigest() + return { + "path": str(path), + "name": config.get("task", {}).get("name", path.name), + "sha256": fingerprint(files), + } + + +def stage_task( + plan: dict, destination: Path, image: str, task_python: str = "python3" +) -> None: + """Keep task assets and tests; use the prepared task plus harness image.""" + shutil.copytree( + plan["task"]["path"], + destination, + ignore=shutil.ignore_patterns("solution", "__pycache__"), + ) + # Harbor must build our overlay rather than pulling the benchmark-only image. + config_path = destination / "task.toml" + config = config_path.read_text() + config = re.sub(r"(?m)^docker_image\s*=.*$", "", config) + config_path.write_text(config) + (destination / "environment/Dockerfile").write_text(f"FROM {image}\n") + # Inspect the agent's final install before the verifier installs its own deps. + # Run the upstream verifier unchanged and preserve its exit status. + verifier = destination / "tests/test.sh" + verifier.rename(destination / "tests/upstream-test.sh") + python = shlex.quote(task_python) + verifier.write_text( + "#!/bin/bash\n" + f"{python} /qualification/identity.py > /logs/verifier/qualification-identity.json\n" + f"{python} /qualification/identity.py --selection /tmp/fabric-run-*.json > /logs/verifier/qualification-selection.json\n" + "bash /tests/upstream-test.sh\n" + ) + + +def valid_install_sources(identity: dict) -> bool: + if not identity.get("editable"): + return True + source = identity.get("editable_sources", {}).get("hermes-agent", {}) + revision = source.get("revision") + return ( + identity["editable"] == ["hermes-agent"] + and isinstance(revision, str) + and re.fullmatch(r"[0-9a-f]{40}", revision) is not None + and revision == identity.get("harness_sources", {}).get("hermes-agent") + and source.get("dirty") is False + ) + + +def unchanged_install(expected: dict, observed: dict) -> bool: + """Protect Fabric/harness identity; record other task dependency changes only.""" + if not valid_install_sources(observed): + return False + for key, value in expected.items(): + if key == "packages": + if any( + observed.get(key, {}).get(name) != version + for name, version in value.items() + if name.startswith("nemo-fabric") + or name + in { + "deepagents", + "hermes-agent", + "mini-swe-agent", + "nooa", + "nooa-bench", + "nooa-cli", + "openhands-sdk", + "openhands-tools", + } + ): + return False + elif observed.get(key) != value: + return False + return True + + +def validate_plan(plan: dict) -> None: + if plan["format_version"] != 1 or plan["mode"] not in {"source", "registry"}: + raise ValueError("unsupported qualification plan") + if ( + plan["concurrency"] != 1 + or plan["attempts"] != 1 + or not 30 <= plan["runtime_timeout_seconds"] <= 1800 + ): + raise ValueError( + "qualification requires concurrency=1, attempts=1 and a 30..1800 second deadline" + ) + if task_identity(Path(plan["task"]["path"])) != plan["task"]: + raise ValueError("task changed; regenerate the qualification plan") + reference = load(ROOT / CATALOG) + catalog = plan_catalog(plan) + if plan["mode"] == "source" and catalog != reference: + raise ValueError("source mode requires the checkout's unmodified catalog") + expected = set(reference["adapters"]) + ids = [row["adapter_id"] for row in plan["adapters"]] + if ( + set(catalog["adapters"]) != expected + or len(ids) != len(set(ids)) + or set(ids) != expected + ): + raise ValueError( + "plan must contain every catalog adapter exactly once; regenerate it" + ) + if plan["version"] != catalog["catalog_version"]: + raise ValueError( + "catalog must match the exact package version under test; use --catalog-json for a published prerelease catalog" + ) + if ( + plan["source_sha"] + != subprocess.check_output( + ["git", "rev-parse", "HEAD"], cwd=ROOT, text=True + ).strip() + ): + raise ValueError("source HEAD changed; regenerate the qualification plan") + subprocess.run( + [ + "git", + "diff", + "--exit-code", + "HEAD", + "--", + "crates", + "sdk", + "adapters", + "adapter-contract", + "Cargo.toml", + "Cargo.lock", + ], + cwd=ROOT, + check=True, + stdout=subprocess.DEVNULL, + ) + subprocess.run( + [ + sys.executable, + str(ROOT / "scripts/ci/generate_adapter_catalog.py"), + "--check", + ], + cwd=ROOT, + check=True, + env=process_env(), + ) + for row in plan["adapters"]: + if row["model"].startswith(("/", "./", "../", "~/")): + raise ValueError( + "serve a local checkpoint first, then use its advertised model ID and HTTP endpoint" + ) + record = catalog["adapters"][row["adapter_id"]] + source = reference["adapters"][row["adapter_id"]] + if ( + row["catalog"] != record + or fingerprint(load(ROOT / record["source"])) != record["sha256"] + or any( + record[key] != source[key] + for key in ("source", "ecosystem", "descriptor") + ) + or record["package"]["name"] != source["package"]["name"] + ): + raise ValueError( + "catalog/source descriptor drift; regenerate catalog and plan" + ) + if row["action"] not in {"run", "blocked", "unsupported", "excluded"}: + raise ValueError("invalid plan action") + if row["action"] != "run" and not row.get("reason"): + raise ValueError("non-run entries require a reason") + provider = row.get("model_provider") + if provider is not None and ( + not isinstance(provider, str) + or not provider + or provider != provider.strip() + ): + raise ValueError( + "model_provider must be non-empty without surrounding whitespace" + ) + key = row.get("api_key_env") + task_python = row.get("task_python") + if task_python is not None and ( + not isinstance(task_python, str) or not task_python.strip() + ): + raise ValueError("task_python must name a compatible task interpreter") + if key and ( + not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", key) + or not re.search(r"(?:API_KEY|TOKEN|SECRET|PASSWORD)$", key) + ): + raise ValueError( + "use a Harbor-redactable credential variable such as MODEL_API_KEY" + ) + + def validate_settings(value: object) -> None: + if isinstance(value, dict): + for name, child in value.items(): + if not isinstance(name, str): + raise ValueError("settings JSON objects require string keys") + if re.search( + r"(?:api.?key|auth.?token|password|secret)$", name, re.I + ): + raise ValueError( + "credentials belong in an environment variable, not harness settings" + ) + validate_settings(child) + elif isinstance(value, list): + for child in value: + validate_settings(child) + + validate_settings(row["harness_settings"]) + target_id = row.get("workflow_target_id") + workflow_settings = row.get("workflow_settings") + if target_id is None: + if workflow_settings is not None: + raise ValueError("workflow_settings requires workflow_target_id") + if row["action"] == "run" and record["descriptor"].get("target_types") == [ + "workflow" + ]: + raise ValueError("workflow-only adapters require workflow_target_id") + else: + if ( + plan["bridge"] != "sdk" + or not isinstance(target_id, str) + or not target_id + or target_id != target_id.strip() + ): + raise ValueError( + "workflow_target_id requires a non-empty ID and the SDK bridge" + ) + target = catalog["targets"].get(target_id) + target_source = reference["targets"].get(target_id) + if ( + target is None + or target_source is None + or any( + target[key] != target_source[key] + for key in ("source", "ecosystem", "descriptor") + ) + or target["package"]["name"] != target_source["package"]["name"] + or fingerprint(load(ROOT / target["source"])) != target["sha256"] + or target["descriptor"]["type"] != "workflow" + or target["descriptor"]["adapter_id"] != row["adapter_id"] + or "workflow" not in record["descriptor"].get("target_types", []) + ): + raise ValueError( + "workflow target metadata must match its catalog and adapter" + ) + if workflow_settings is not None: + if not isinstance(workflow_settings, dict) or any( + not isinstance(key, str) for key in workflow_settings + ): + raise ValueError("workflow_settings must be a JSON object") + try: + json.dumps(workflow_settings, allow_nan=False) + except (TypeError, ValueError, OverflowError, RecursionError): + raise ValueError( + "workflow_settings must contain JSON values" + ) from None + validate_settings(workflow_settings) + model_settings = row.get("model_settings") + if model_settings is not None: + if plan["bridge"] != "sdk" or not isinstance(model_settings, dict): + raise ValueError("model_settings requires an object and the SDK bridge") + validate_settings(model_settings) + max_turns = row.get("max_turns") + if max_turns is not None and ( + isinstance(max_turns, bool) + or not isinstance(max_turns, int) + or max_turns <= 0 + ): + raise ValueError("max_turns must be a positive integer") + url = row.get("base_url") + if url: + parsed = urlsplit(url) + if ( + parsed.scheme not in {"http", "https"} + or not parsed.hostname + or parsed.username + or parsed.password + or parsed.query + or parsed.fragment + ): + raise ValueError( + "endpoint must be an HTTP(S) URL without credentials, query, or fragment" + ) + if parsed.scheme == "http" and parsed.hostname not in { + "localhost", + "127.0.0.1", + "::1", + *plan["trusted_http_hosts"], + }: + raise ValueError( + "non-loopback HTTP requires an explicit --trusted-http-host" + ) + cost = plan["estimated_cost_usd"] + if ( + not isinstance(cost, (int, float)) + or not math.isfinite(cost) + or cost < 0 + or not plan["cost_basis"] + ): + raise ValueError("record a finite nonnegative cost estimate and its basis") + + +def make_plan(args: argparse.Namespace) -> dict: + catalog = load(getattr(args, "catalog_json", None) or ROOT / CATALOG) + rows = [] + prerequisites = { + "nvidia.fabric.hermes": "Requires a supported Hermes runtime/source image; select its compatible task_python and record pinned, clean source provenance.", + "nvidia.fabric.remote-agent": "Requires an explicitly configured compatible HTTP endpoint/profile; workspace/tool execution is separate coverage.", + "nvidia.fabric.kiro": "Requires Kiro CLI, KIRO_API_KEY, and a Kiro-advertised provider/model. Skip when KIRO_API_KEY is absent; BYOM is unsupported.", + "nvidia.fabric.droid": "Requires Factory authentication or a prepared native Droid BYOK profile/image; custom endpoints use native settings, not model base_url.", + } + for adapter_id, record in sorted(catalog["adapters"].items()): + reason = prerequisites.get(adapter_id, "") + unsupported = record["descriptor"].get("target_types") == ["workflow"] + targets = [ + target_id + for target_id, target in catalog["targets"].items() + if target["descriptor"]["adapter_id"] == adapter_id + ] + if unsupported: + reason = "This adapter requires a registered workflow target. Select workflow_target_id and optional workflow_settings with the SDK bridge before marking the row runnable." + if ( + adapter_id == "nvidia.fabric.claude" + and args.endpoint_kind == "chat-completions" + ): + reason = "Claude requires an Anthropic Messages-compatible endpoint; this plan selects Chat Completions." + accepts = record["descriptor"].get("config", {}).get("accepts", []) + settings = {} + if adapter_id == "nvidia.fabric.codex": + # Harbor supplies the disposable container boundary; nested bwrap + # requires user namespaces that many Docker hosts prohibit. + settings = {"sandbox": "danger-full-access", "approval_mode": "deny_all"} + if adapter_id == "nvidia.fabric.claude": + settings = {"permission_mode": "bypassPermissions"} + if adapter_id == "nvidia.fabric.langchain.deepagents": + settings = {"deepagents": {"backend": {"type": "local_shell"}}} + if adapter_id == "nvidia.fabric.qwen": + settings = {"permission_mode": "yolo"} + properties = ( + record["descriptor"].get("settings_schema", {}).get("properties", {}) + ) + if "insecure_http_model_hosts" in properties and args.trusted_http_host: + settings["insecure_http_model_hosts"] = args.trusted_http_host + rows.append( + { + "adapter_id": adapter_id, + "catalog": record, + "action": "unsupported" + if unsupported + else "blocked" + if reason + else "run", + "reason": reason, + "registered_targets": targets, + "model": args.model, + "base_url": args.base_url, + "api_key_env": "KIRO_API_KEY" + if adapter_id == "nvidia.fabric.kiro" + else args.api_key_env, + "harness_settings": settings, + "enabled_tools": [ + "ls", + "read_file", + "write_file", + "edit_file", + "glob", + "grep", + "execute", + ] + if adapter_id == "nvidia.fabric.langchain.deepagents" + else None, + "telemetry": "none", + "capabilities": { + name: "untested" + if supported + else "not_declared" + if name == "subagents" + else "unsupported" + for name, supported in { + "mcp": "mcp" in accepts, + "skills": "skills" in accepts, + "telemetry_atif": any( + "atif" in provider.get("outputs", []) + for provider in record["descriptor"] + .get("telemetry", {}) + .get("providers", {}) + .values() + ), + "subagents": "subagents" + in json.dumps(record["descriptor"].get("settings_schema", {})), + }.items() + }, + } + ) + return { + "format_version": 1, + "task": task_identity(args.task), + "mode": args.mode, + "version": args.version, + "catalog_snapshot": catalog, + "catalog_sha256": fingerprint(catalog), + "source_sha": subprocess.check_output( + ["git", "rev-parse", "HEAD"], cwd=ROOT, text=True + ).strip(), + "harbor_version": args.harbor_version, + "bridge": args.bridge, + "endpoint_kind": args.endpoint_kind, + "backend": getattr(args, "backend", "operator-managed inference"), + "trusted_http_hosts": args.trusted_http_host, + "estimated_cost_usd": args.estimated_cost_usd, + "cost_basis": args.cost_basis, + "concurrency": 1, + "attempts": 1, + "runtime_timeout_seconds": args.timeout, + "created_at": now(), + "adapters": rows, + } + + +def harbor_command( + plan: dict, row: dict, python: str, task: Path, jobs: Path, name: str +) -> list[str]: + command = [ + python, + "-m", + "harbor.cli.main", + "run", + "--path", + str(task), + "--agent", + "fabric" + if plan["bridge"] == "native" + else "nemo_fabric.integrations.harbor:FabricAgent", + "--model", + row["model"], + "--job-name", + name, + "--jobs-dir", + str(jobs), + "--n-concurrent", + "1", + "--n-attempts", + "1", + ] + options = { + "fabric_adapter_id": row["adapter_id"], + "fabric_workspace": "/app", + "fabric_config_base_dir": "/app", + "fabric_harness_settings": row["harness_settings"], + "fabric_runtime_timeout_seconds": plan["runtime_timeout_seconds"], + "fabric_timeout_sec": plan["runtime_timeout_seconds"] + 60, + "fabric_telemetry": row["telemetry"], + } + if row.get("task_python") is not None: + options["fabric_python"] = row["task_python"] + if row.get("model_provider") is not None: + options["fabric_model_provider"] = row["model_provider"] + if row.get("model_settings") is not None: + options["fabric_model_settings"] = row["model_settings"] + if row.get("workflow_target_id") is not None: + options["fabric_workflow_target_id"] = row["workflow_target_id"] + options["fabric_workflow_settings"] = row.get("workflow_settings") or {} + if row.get("max_turns") is not None: + options["fabric_max_turns"] = row["max_turns"] + if row.get("base_url"): + options["fabric_model_base_url"] = row["base_url"] + if row.get("api_key_env"): + key = row["api_key_env"] + options["fabric_model_api_key_env"] = key + command.extend(["--ae", f"{key}=${{{key}}}"]) + if row.get("enabled_tools") is not None: + options["fabric_enabled_tools"] = row["enabled_tools"] + if row["catalog"]["ecosystem"] == "typescript": + record = row["catalog"] + options["fabric_discovery_paths"] = [ + f"/opt/harness/node_modules/{record['package']['name']}/{Path(record['source']).name}" + ] + if row["adapter_id"] == "nvidia.fabric.claude": + options["fabric_environment_env"] = {"IS_SANDBOX": "1"} + for key, value in options.items(): + command.extend( + [ + "--ak", + f"{key}={json.dumps(value) if isinstance(value, (dict, list)) else value}", + ] + ) + return command + + +def one(directory: Path, pattern: str) -> Path: + paths = list(directory.glob(pattern)) + if len(paths) != 1: + raise ValueError(f"expected one {pattern}, found {len(paths)}") + return paths[0] + + +def classify(text: str) -> str: + for category, pattern in ( + ( + "adapter_admission", + r"fingerprint|descriptor.*(drift|mismatch)|capabilit|unsupported.*(provider|model)|model_schema|settings_schema|config.*invalid", + ), + ( + "auth", + r"unauthorized|authentication|api.key.*(missing|invalid|required)|\b(?:401|403)\b", + ), + ( + "install", + r"no matching distribution|modulenotfound|no module named|harness_unavailable|cannot find (module|package)|enoent", + ), + ( + "provider_model", + r"model.*(not found|unknown|not supported)|\b404\b|messages.compatible|provider", + ), + ("timeout", r"timeout|timed out|deadline"), + ( + "infrastructure", + r"no space left|connection refused|docker.*(error|failed)|temporary failure|rate.limit|\b(?:429|502|503)\b", + ), + ): + if re.search(pattern, text, re.I): + return category + return "transport_runner" + + +def normalized_accounting(path: Path) -> dict: + from harbor.models.agent.context import AgentContext + from nemo_fabric.integrations.harbor.fabric_agent import ( + populate_context_from_result, + ) + + context = AgentContext() + populate_context_from_result(context, path) + return { + key: getattr(context, key) + for key in ("n_input_tokens", "n_output_tokens", "n_cache_tokens", "cost_usd") + } + + +def admission_fingerprint( + spec: dict, descriptor: dict, target_descriptor: dict | None = None +) -> str | None: + from nemo_fabric import FabricConfig, inspect_adapter + + options = ( + {"target_descriptor": target_descriptor} + if target_descriptor is not None + else {} + ) + return inspect_adapter( + FabricConfig.from_mapping(spec["config"]), descriptor, **options + ).descriptor_sha256 + + +def valid_selection(selected: dict, row: dict, target: dict | None = None) -> bool: + origins = selected.get("provenance") or [] + if selected.get("adapter_id") != row["adapter_id"] or not origins: + return False + workflow = selected.get("workflow") + if row.get("workflow_target_id") is None: + if workflow is not None: + return False + elif ( + target is None + or not isinstance(workflow, dict) + or workflow.get("target_id") != row["workflow_target_id"] + or workflow.get("entrypoint") != target["descriptor"]["spec"]["entrypoint"] + or not workflow.get("provenance") + or not all( + origin.get("source") == "installed_package" + and origin.get("path", "").startswith( + selected.get("python_prefix", "") + "/share/nemo-fabric/adapters/" + ) + and Path(origin["path"]).name == Path(target["source"]).name + for origin in workflow["provenance"] + ) + ): + return False + filename = Path(row["catalog"]["source"]).name + if row["catalog"]["ecosystem"] == "typescript": + expected = ( + f"/opt/harness/node_modules/{row['catalog']['package']['name']}/{filename}" + ) + return all( + origin.get("source") == "explicit_local" and origin.get("path") == expected + for origin in origins + ) + prefix = selected.get("python_prefix", "") + "/share/nemo-fabric/adapters/" + return all( + origin.get("source") == "installed_package" + and origin.get("path", "").startswith(prefix) + and Path(origin["path"]).name == filename + for origin in origins + ) + + +def selection_payload(plan: dict, row: dict) -> dict: + from nemo_fabric.integrations.harbor.fabric_agent import build_harbor_config + + paths = ( + [ + f"/opt/harness/node_modules/{row['catalog']['package']['name']}/{Path(row['catalog']['source']).name}" + ] + if row["catalog"]["ecosystem"] == "typescript" + else [] + ) + provider_options = ( + {"model_provider_name": row["model_provider"]} + if row.get("model_provider") is not None + else {} + ) + if row.get("model_settings") is not None: + provider_options["model_settings"] = row["model_settings"] + if row.get("workflow_target_id") is not None: + provider_options["workflow_target_id"] = row["workflow_target_id"] + provider_options["workflow_settings"] = row.get("workflow_settings") or {} + config = build_harbor_config( + adapter_id=row["adapter_id"], + workspace="/app", + harness_settings=row["harness_settings"], + model_name=row["model"], + model_base_url=row.get("base_url"), + model_api_key_env=row.get("api_key_env"), + timeout_seconds=plan["runtime_timeout_seconds"], + max_turns=row.get("max_turns"), + enabled_tools=row.get("enabled_tools"), + telemetry=row["telemetry"], + discovery_paths=tuple(paths), + **provider_options, + ) + return {"config": config.to_mapping(), "config_base_dir": "/app"} + + +def inspect_results( + job: Path, + row: dict, + expected_identity: dict, + expected_selection: dict, + target: dict | None = None, +) -> dict: + """Keep transport, reward, selection, usage and telemetry as separate evidence.""" + summary = {"checks": {}, "failures": [], "limitations": [], "evidence": []} + + def fail(category: str, reason: str) -> None: + summary["failures"].append({"category": category, "reason": sanitize(reason)}) + + # Verifier evidence remains meaningful even when result transport fails. + try: + reward_path = one(job, "*/verifier/reward.txt") + reward = float(reward_path.read_text().strip()) + if not math.isfinite(reward): + raise ValueError("Task reward is not finite") + summary["reward"] = reward + summary["checks"]["verifier_completed"] = True + summary["checks"]["task_success"] = summary["reward"] == 1.0 + except (OSError, ValueError) as error: + summary["checks"]["verifier_completed"] = False + fail("harbor_verifier", str(error)) + try: + task_identity = load(one(job, "*/verifier/qualification-identity.json")) + summary["checks"]["task_install_identity"] = unchanged_install( + expected_identity, task_identity + ) + if not summary["checks"]["task_install_identity"]: + fail( + "adapter_admission", + "Fabric/harness identity differs from inspected image or has editable installs", + ) + except (OSError, ValueError, KeyError, TypeError) as error: + fail("adapter_admission", f"Task identity unavailable: {error}") + try: + observed_selection = load(one(job, "*/verifier/qualification-selection.json")) + summary["task_selection"] = observed_selection + summary["checks"]["task_descriptor_provenance"] = ( + valid_selection(observed_selection, row, target) + and observed_selection == expected_selection + ) + if not summary["checks"]["task_descriptor_provenance"]: + fail( + "adapter_admission", + "Task descriptor origin differs from the installed preflight selection", + ) + except (OSError, ValueError, KeyError, TypeError) as error: + fail("adapter_admission", f"Task selection unavailable: {error}") + try: + job_result = load(job / "result.json") + stats = job_result["stats"] + summary["checks"]["harbor_lifecycle"] = ( + stats["n_completed_trials"] == 1 and stats["n_errored_trials"] == 0 + ) + if not summary["checks"]["harbor_lifecycle"]: + fail( + "harbor_lifecycle", + "Harbor did not complete exactly one trial without exceptions", + ) + trial = load(one(job, "*/result.json")) + exception = trial.get("exception_info") + if exception: + message = str(exception.get("exception_message", exception)) + fail(classify(message), message[:2000]) + result_path = one(job, "*/agent/fabric-result-*.json") + summary["evidence"].append(str(result_path)) + document = load(result_path) + if "runner_error" in document: + fail(classify(json.dumps(document)), json.dumps(document["runner_error"])) + return summary + from nemo_fabric import RunResult + + result = RunResult.from_mapping(document).to_mapping() + summary["checks"]["result_transport"] = True + summary["observed"] = { + key: result.get(key) + for key in ( + "status", + "adapter_id", + "harness", + "runtime_id", + "invocation_id", + "request_id", + ) + } + summary["checks"]["selection"] = result.get("adapter_id") == row["adapter_id"] + if not summary["checks"]["selection"]: + fail("adapter_admission", "Observed adapter differs from selected adapter") + if result["status"] != "succeeded" or result.get("error"): + fail( + classify(json.dumps(result.get("error"))), + json.dumps(result.get("error")), + ) + spec = load(one(job, "*/agent/fabric-run-*.json")) + target_id = row.get("workflow_target_id") + workflow = spec["config"].get("workflow") + summary["checks"]["workflow_selection"] = ( + workflow is None + if target_id is None + else isinstance(workflow, dict) + and workflow.get("target_id") == target_id + and (workflow.get("settings") or {}) == (row.get("workflow_settings") or {}) + ) + if not summary["checks"]["workflow_selection"]: + fail( + "adapter_admission", + "Uploaded workflow differs from the selected target/settings", + ) + expected_admission = admission_fingerprint( + spec, + row["catalog"]["descriptor"], + target["descriptor"] if target is not None else None, + ) + summary["admission_sha256"] = { + "expected_normalized": expected_admission, + "uploaded": spec.get("adapter_descriptor_sha256"), + "catalog_raw": row["catalog"]["sha256"], + **({"target_catalog_raw": target["sha256"]} if target is not None else {}), + } + summary["checks"]["descriptor_fingerprint"] = ( + expected_admission is not None + and spec.get("adapter_descriptor_sha256") == expected_admission + ) + if not summary["checks"]["descriptor_fingerprint"]: + fail( + "adapter_admission", + "Uploaded fingerprint differs from the core-normalized catalog descriptor", + ) + context = trial["agent_result"] or {} + usage = result.get("usage") or {} + summary["usage"] = { + key: usage.get(key) + for key in ( + "input_tokens", + "output_tokens", + "cached_input_tokens", + "cost_usd", + ) + } + summary["harbor_usage"] = { + key: context.get(key) + for key in ( + "n_input_tokens", + "n_output_tokens", + "n_cache_tokens", + "cost_usd", + ) + } + expected_usage = normalized_accounting(result_path) + known = all( + isinstance(expected_usage[key], int) + and not isinstance(expected_usage[key], bool) + and expected_usage[key] >= 0 + for key in ("n_input_tokens", "n_output_tokens") + ) + summary["checks"]["usage_available"] = known + if not known: + summary["limitations"].append( + "Harness complete token usage is unavailable; unknown counters remain unknown" + ) + metadata = context.get("metadata", {}).get("fabric", {}) + summary["checks"]["harbor_result_transport"] = metadata.get( + "adapter_id" + ) == result.get("adapter_id") and metadata.get("invocation_id") == result.get( + "invocation_id" + ) + if not summary["checks"]["harbor_result_transport"]: + fail( + "transport_runner", + "Harbor context did not retain Fabric invocation identity", + ) + summary["checks"]["usage_transport"] = summary["harbor_usage"] == expected_usage + if not summary["checks"]["usage_transport"]: + fail( + "usage", + "Harbor accounting differs from Fabric's normalized invocation accounting", + ) + if ( + known + and expected_usage["n_input_tokens"] + expected_usage["n_output_tokens"] + == 0 + ): + summary["limitations"].append( + "Harness reported no positive inference usage; accounting does not establish model activity" + ) + if row["telemetry"] == "relay": + validation = load(one(job, "*/agent/telemetry-validation.json")) + trajectory = one(job, "*/agent/trajectory.json") + from harbor.models.trajectories.trajectory import Trajectory + + Trajectory.model_validate_json(trajectory.read_text()) + summary["checks"]["telemetry_atif"] = validation["status"] == "succeeded" + if not summary["checks"]["telemetry_atif"]: + fail("telemetry_export", "Relay export validation failed") + except (OSError, ValueError, KeyError, TypeError, RuntimeError) as error: + fail("transport_runner", str(error)) + return summary + + +def cleanup(prefix: str, log: Path) -> bool: + """Remove only Compose resources carrying this invocation's unique task prefix.""" + probes = subprocess.run( + [ + "docker", + "ps", + "-aq", + "--filter", + f"label=org.nvidia.fabric.qualification.run={prefix}", + ], + capture_output=True, + text=True, + timeout=30, + check=True, + ).stdout.split() + for resource in probes: + if execute(["docker", "rm", "-f", resource], log, 30): + return False + for kind, listing, deletion in ( + ("container", ["ps", "-aq"], ["rm", "-f"]), + ("network", ["network", "ls", "-q"], ["network", "rm"]), + ): + found = subprocess.run( + ["docker", *listing, "--filter", "label=com.docker.compose.project"], + capture_output=True, + text=True, + timeout=30, + check=True, + ).stdout.split() + for resource in found: + inspected = subprocess.check_output( + [ + "docker", + kind, + "inspect", + "--format", + "{{json .Config.Labels}}" + if kind == "container" + else "{{json .Labels}}", + resource, + ], + text=True, + timeout=30, + ) + labels = json.loads(inspected) + if (labels or {}).get("com.docker.compose.project", "").startswith(prefix): + if execute(["docker", *deletion, resource], log, 30): + return False + return True + + +def write_report(output: Path, report: dict) -> None: + rows = report["adapters"] + report["all_harnesses_passed"] = bool(rows) and all( + row["outcome"] == "passed" for row in rows + ) + report["full_catalog_passed"] = ( + report["scope"] == "full_catalog" + and report["all_harnesses_passed"] + and report.get("state", "completed") == "completed" + ) + report["updated_at"] = now() + report["pass_criteria"] = ( + "integration; task reward and harness usage availability are informational" + ) + credential_names = tuple( + name + for row in [*rows, *report["plan"].get("adapters", [])] + if (name := row.get("selected", row).get("api_key_env")) + ) + save(output / "report.json", sanitize_data(report, credential_names)) + lines = [ + f"Harbor qualification: {'PASS' if report['full_catalog_passed'] else 'NOT QUALIFIED'}", + f"Scope: {report['scope']}; denominator: {len(rows)} catalog adapters", + f"Mode: {report['plan']['mode']}; source: {report['plan']['source_sha']}", + f"Task: {report['plan']['task']['name']}; SHA256: {report['plan']['task']['sha256']}", + "", + "| Adapter | Integration | Task reward | Classification / limitations |", + "| --- | --- | --- | --- |", + ] + for row in rows: + reason = row.get("reason") or "; ".join( + f"{f['category']}: {f['reason']}" for f in row.get("failures", []) + ) + reason = "; ".join(filter(None, [reason, *row.get("limitations", [])])) + reward = row.get("reward", "n/a") + lines.append( + f"| {row['adapter_id']} | {row['outcome']} | {reward} | {reason.replace('|', '/').replace(chr(10), ' ')[:600]} |" + ) + lines.extend( + [ + "", + "Integration pass requires lifecycle, provenance, result/accounting transport and cleanup. Task reward and harness usage availability are informational; missing verifier evidence remains a failure.", + "Reproduce with the recorded commands and retained environment/plan. Raw task artifacts are private; inspect and redact before sharing.", + "MCP, skills, subagents and telemetry are untested unless explicitly exercised; one task does not establish benchmark or all-feature equivalence.", + ] + ) + temporary = output / "summary.tmp" + temporary.write_text(sanitize("\n".join(lines) + "\n", credential_names)) + temporary.replace(output / "summary.md") + + +def run(plan: dict, environment: dict, output: Path, selected: list[str]) -> int: + validate_plan(plan) + if environment["plan"] != plan: + raise ValueError("environment was prepared for a different plan") + unknown = set(selected) - {row["adapter_id"] for row in plan["adapters"]} + if unknown: + raise ValueError(f"unknown subset adapters: {sorted(unknown)}") + output.mkdir(mode=0o700) + prefix = "fq-" + uuid.uuid4().hex[:12] + report = { + "format_version": 1, + "started_at": now(), + "finished_at": None, + "state": "running", + "scope": "developer_subset" if selected else "full_catalog", + "plan": { + key: value + for key, value in plan.items() + if key not in {"adapters", "catalog_snapshot"} + }, + "tool_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + "host": {"hostname": os.uname().nodename, "platform": sys.platform}, + "adapters": [], + } + for row in plan["adapters"]: + report["adapters"].append( + { + "adapter_id": row["adapter_id"], + "outcome": "blocked", + "reason": "Not reached", + "capabilities": row["capabilities"], + "registered_targets": row.get("registered_targets", []), + "selected": { + "package": row["catalog"]["package"], + "descriptor_sha256": row["catalog"]["sha256"], + "model": row["model"], + "provider": row.get("model_provider") + or ( + row["model"].split("/", 1)[0] + if "/" in row["model"] + else "openai" + ), + "base_url": row.get("base_url"), + "api_key_env": row.get("api_key_env"), + "harness_settings": row["harness_settings"], + "model_settings": row.get("model_settings"), + "workflow_target_id": row.get("workflow_target_id"), + "workflow_settings": row.get("workflow_settings"), + "max_turns": row.get("max_turns"), + "task_python": row.get("task_python"), + "telemetry": row["telemetry"], + }, + } + ) + write_report(output, report) + save(output / "plan.json", plan) + host_python = environment.get("python") + if not host_python or not Path(host_python).is_file(): + for row, planned in zip(report["adapters"], plan["adapters"]): + if planned["action"] != "run": + row.update(outcome=planned["action"], reason=planned["reason"]) + continue + row["reason"] = ( + "Host installation unavailable; inspect environment.json and base/host installation logs" + ) + report.update(state="completed", finished_at=now()) + write_report(output, report) + return 1 + host_identity_path = output / "host-identity.json" + identity_script = Path(__file__).with_name("identity.py") + if execute([host_python, str(identity_script)], output / "host-identity.log", 45): + raise ValueError("host package identity inspection failed") + identity_text = subprocess.check_output( + [host_python, str(identity_script)], env=process_env(), text=True, timeout=45 + ) + host_identity = json.loads(identity_text) + save(host_identity_path, host_identity) + report["host"]["identity"] = host_identity + runtime_wheel = next( + name + for name in environment["wheel_sha256"] + if name.startswith("nemo_fabric_runtime-") + ) + expected_runtime_hash = environment["wheel_sha256"][runtime_wheel] + if ( + host_identity["editable"] + or host_identity["packages"].get("nemo-fabric-runtime") != plan["version"] + or host_identity["packages"].get("harbor") != plan["harbor_version"] + or host_identity["catalog_sha256"] != fingerprint(plan_catalog(plan)) + or host_identity["wheel_sha256"].get("nemo-fabric-runtime") + != expected_runtime_hash + ): + raise ValueError( + "host has editable installs or differs from planned Fabric/Harbor versions" + ) + for index, (row, result) in enumerate( + zip(plan["adapters"], report["adapters"], strict=True) + ): + if selected and row["adapter_id"] not in selected: + result.update( + outcome="excluded", + reason="Outside explicitly selected developer subset", + ) + write_report(output, report) + continue + if row["action"] != "run": + result.update( + outcome=row["action"], + reason=row["reason"], + failures=[ + { + "category": "provider_model" + if "provider" in row["reason"].lower() + else "prerequisite", + "reason": row["reason"], + } + ], + ) + write_report(output, report) + continue + if row.get("api_key_env") and not os.environ.get(row["api_key_env"]): + result.update( + reason=f"Missing credential environment variable {row['api_key_env']}", + failures=[ + {"category": "auth", "reason": "Required credential is unavailable"} + ], + ) + write_report(output, report) + continue + install = environment["adapters"].get(row["adapter_id"], {}) + if install.get("status") != "ready": + result.update( + outcome="failed", + reason="Installation failed or was not completed", + failures=[ + { + "category": "install", + "reason": install.get("log", "No prepared image"), + } + ], + ) + write_report(output, report) + continue + directory = output / str(index) + directory.mkdir() + task = directory / f"{prefix}-{index}" + image = install["image"] + stage_task(plan, task, image, row.get("task_python", "python3")) + jobs = directory / "jobs" + name = f"qualification-{index}" + command = harbor_command(plan, row, host_python, task, jobs, name) + result.update( + reason="Running", + started_at=now(), + command=shlex.join(command), + artifact_directory=str(directory), + ) + write_report(output, report) + try: + task_identity = json.loads( + subprocess.check_output( + [ + "docker", + "run", + "--rm", + "--network", + "none", + "--label", + f"org.nvidia.fabric.qualification.run={prefix}", + "--entrypoint", + "cat", + image, + "/opt/qualification-identity.json", + ], + text=True, + timeout=45, + ) + ) + save(directory / "image-identity.json", task_identity) + result["image_id"] = subprocess.check_output( + ["docker", "image", "inspect", "--format", "{{.Id}}", image], + text=True, + timeout=30, + ).strip() + result["checks"] = { + "host_task_runtime_version": task_identity["packages"].get( + "nemo-fabric-runtime" + ) + == plan["version"] + } + package = row["catalog"]["package"]["name"] + observed_packages = ( + task_identity["packages"] + if row["catalog"]["ecosystem"] == "python" + else task_identity["npm"] + ) + result["checks"]["task_adapter_version"] = ( + observed_packages.get(package) == row["catalog"]["package"]["version"] + ) + result["task_identity"] = task_identity + if ( + not result["checks"]["host_task_runtime_version"] + or not valid_install_sources(task_identity) + or not result["checks"]["task_adapter_version"] + or task_identity["catalog_sha256"] != fingerprint(plan_catalog(plan)) + or task_identity["wheel_sha256"].get("nemo-fabric-runtime") + != expected_runtime_hash + ): + raise ValueError( + "task runtime differs from host or has editable imports" + ) + probe = subprocess.run( + [ + "docker", + "run", + "--rm", + "--network", + "none", + "--label", + f"org.nvidia.fabric.qualification.run={prefix}", + "--entrypoint", + row.get("task_python", "python"), + "-i", + image, + "/qualification/identity.py", + "--selection", + "-", + ], + input=json.dumps(selection_payload(plan, row)), + capture_output=True, + text=True, + timeout=60, + env=process_env(), + check=False, + ) + (directory / "selection.log").write_text(sanitize(probe.stderr)) + if probe.returncode: + raise ValueError( + "adapter admission: installed descriptor preflight failed; inspect selection.log" + ) + selected_adapter = json.loads(probe.stdout) + result["preflight_selection"] = selected_adapter + target = plan_catalog(plan)["targets"].get(row.get("workflow_target_id")) + if not valid_selection(selected_adapter, row, target): + raise ValueError( + "adapter admission: source or unexpected descriptor origin selected" + ) + code = execute( + command, + directory / "harbor.log", + plan["runtime_timeout_seconds"] + 600, + key_env=row.get("api_key_env"), + cwd=directory, + ) + evidence = inspect_results( + jobs / name, row, task_identity, selected_adapter, target + ) + evidence["checks"] = {**result["checks"], **evidence["checks"]} + result.update(evidence) + if row["telemetry"] == "relay": + result["capabilities"]["telemetry_atif"] = ( + "passed" if result["checks"].get("telemetry_atif") else "failed" + ) + result["exit_code"] = code + if code: + log_text = (directory / "harbor.log").read_text()[-20000:] + result["failures"].append( + { + "category": "cancelled" + if code == 130 + else "timeout" + if code == 124 + else classify(log_text), + "reason": f"Harbor exit {code}; inspect harbor.log", + } + ) + result.update( + outcome="failed" if result["failures"] else "passed", reason="" + ) + except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as error: + result.update( + outcome="failed", + reason=sanitize(str(error)), + failures=[ + { + "category": "adapter_admission" + if "adapter admission" in str(error) + else "prerequisite", + "reason": sanitize(str(error)), + } + ], + ) + code = 1 + finally: + try: + result["cleanup"] = ( + "passed" if cleanup(prefix, directory / "cleanup.log") else "failed" + ) + except (OSError, subprocess.SubprocessError, ValueError): + result["cleanup"] = "failed" + if result["cleanup"] != "passed": + result["outcome"] = "failed" + result.setdefault("failures", []).append( + { + "category": "cleanup", + "reason": "Owned Compose resource cleanup failed", + } + ) + # Harbor redacts its configuration. Scrub known secrets from all + # retained text as well, including failing harness stdout. + for path in directory.rglob("*"): + if path.is_file() and not path.is_symlink(): + try: + text = path.read_text(encoding="utf-8") + except UnicodeError: + continue + cleaned = sanitize_artifact( + text, + path.suffix, + (row["api_key_env"],) if row.get("api_key_env") else (), + ) + if cleaned != text: + path.write_text(cleaned, encoding="utf-8") + result["finished_at"] = now() + write_report(output, report) + print(f"{row['adapter_id']}: {result['outcome']}", flush=True) + if code == 130: + report["state"] = "cancelled" + break + report.update( + state="cancelled" if report["state"] == "cancelled" else "completed", + finished_at=now(), + ) + write_report(output, report) + return ( + 0 + if report["full_catalog_passed"] + or ( + selected + and all( + row["outcome"] == "passed" + for row in report["adapters"] + if row["adapter_id"] in selected + ) + ) + else 1 + ) + + +def stop(_signum: int, _frame: object) -> None: + raise KeyboardInterrupt + + +def main() -> None: + signal.signal(signal.SIGTERM, stop) + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="command", required=True) + plan = sub.add_parser("plan") + plan.add_argument( + "--task", + type=Path, + required=True, + help="Exactly one Harbor task directory; never a dataset root", + ) + plan.add_argument("--output", type=Path, required=True) + plan.add_argument("--mode", choices=("source", "registry"), required=True) + plan.add_argument("--version", required=True) + plan.add_argument("--harbor-version", required=True) + plan.add_argument( + "--catalog-json", + type=Path, + help="Catalog resource extracted from the exact published catalog wheel; required when published package metadata differs from the checkout", + ) + plan.add_argument("--bridge", choices=("native", "sdk"), default="native") + plan.add_argument("--model", required=True) + plan.add_argument("--base-url") + plan.add_argument( + "--backend", + default="operator-managed inference", + help="Serving backend/version or provider deployment identity", + ) + plan.add_argument("--api-key-env", default="MODEL_API_KEY") + plan.add_argument( + "--endpoint-kind", + choices=("chat-completions", "anthropic-messages", "openai-and-anthropic"), + default="chat-completions", + ) + plan.add_argument("--trusted-http-host", action="append", default=[]) + plan.add_argument("--estimated-cost-usd", type=float, required=True) + plan.add_argument("--cost-basis", required=True) + plan.add_argument( + "--timeout", type=int, choices=range(30, 1801), metavar="30..1800", default=240 + ) + run_parser = sub.add_parser("run") + run_parser.add_argument("--plan", type=Path, required=True) + run_parser.add_argument("--environment", type=Path, required=True) + run_parser.add_argument("--output", type=Path, required=True) + run_parser.add_argument("--only", action="append", default=[]) + args = parser.parse_args() + os.umask(0o077) + if args.command == "plan": + value = make_plan(args) + validate_plan(value) + if args.output.exists(): + parser.error("plan output already exists") + save(args.output, value) + print( + f"Planned {len(value['adapters'])} adapters; estimated provider cost ${value['estimated_cost_usd']:.2f}" + ) + else: + raise SystemExit( + run( + load(args.plan), + load(args.environment), + args.output.resolve(), + args.only, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/.agents/skills/validate-change/SKILL.md b/.agents/skills/validate-change/SKILL.md index 18b182061..8ae866ff7 100644 --- a/.agents/skills/validate-change/SKILL.md +++ b/.agents/skills/validate-change/SKILL.md @@ -59,6 +59,7 @@ surfaces touched by a change. - **Adapter behavior changed** Run the focused adapter tests under `tests/adapters`, then `just test-python`. - **Harbor integration changed** + Use `qualify-harbor` for real inference and container checks across the adapter catalog or a developer subset. Retain its complete catalog report; deterministic tests alone do not establish an all-harness integration pass. Run `tests/integrations/test_harbor_runner.py` and `tests/python/test_adapter_capabilities.py`, then `just test-python`. For host admission changes, verify metadata inspection without harness SDKs diff --git a/adapters/python/deepagents/README.md b/adapters/python/deepagents/README.md index 038e4b4fe..0c11c98bc 100644 --- a/adapters/python/deepagents/README.md +++ b/adapters/python/deepagents/README.md @@ -218,12 +218,26 @@ provides only its fallback path; configure required variables explicitly and use absolute executable paths when needed. The `execute` tool remains subject to `tools.enabled` and `tools.blocked`, like all other Deep Agents built-ins. -File tools and shell commands use different absolute-path namespaces when -`virtual_mode=True`. For file tools, `/report.txt` resolves to -`/report.txt`; in `execute`, the same path refers to `/report.txt` on -the adapter host. Shell commands start in `environment.workspace`, so use -workspace-relative paths, such as `cat report.txt`, for files surfaced by file -tools. Do not pass a file tool's virtual absolute path directly to `execute`. +On POSIX, with `local_shell`, file tools accept absolute paths inside +`environment.workspace` as aliases for workspace files. For example, with +workspace `/app`, both file-tool paths `/app/report.txt` and `/report.txt` refer +to `/app/report.txt` on disk. Shell commands use host paths and start in the +workspace, so `cat report.txt` and `cat /app/report.txt` read that file. + +On POSIX, listings and searches return canonical physical paths inside the +workspace. These paths identify the same file in subsequent reads and edits, +even when a nested directory repeats the workspace prefix. For example, `/app/app/report.txt` +remains distinct from `/app/report.txt`; the relative file-tool path +`app/report.txt` also identifies the nested file. + +On Windows, listings and searches retain workspace-rooted virtual paths, such +as `/nested/report.txt`. Use these paths with file tools; Deep Agents rejects +drive-prefixed paths such as `C:/workspace/nested/report.txt` before backend +resolution. Shell commands continue to use native host paths. + +File-tool traversal and symlink-escape checks remain enabled. Paths outside the +workspace retain the existing virtual-root interpretation; they do not grant +access to host files. Python middleware, `FilesystemPermission` objects, Python tool objects, and precompiled `runnable` subagents are not exposed through `harness.settings`. @@ -252,7 +266,11 @@ service. The normalized result includes the final response, buffered messages and per-step events, LangGraph thread id, token usage (and cost when the provider reports it), and errors. Usage aggregates the current turn across the main agent -and any delegated subagents (streamed with `subgraphs=True`). Configuration and +and any delegated subagents (streamed with `subgraphs=True`). Input totals use +LangChain's inclusive convention, which already includes cached tokens; an +unavailable cache breakdown stays unknown. If an assistant message omits a token +counter, that invocation's corresponding total stays unknown rather than +reporting a partial sum. Configuration and preflight failures (a missing credential, an absent `deepagents` package, or an invalid MCP server) fail runtime start before an invocation is accepted. diff --git a/adapters/python/deepagents/src/nemo_fabric_adapters/deepagents/adapter.py b/adapters/python/deepagents/src/nemo_fabric_adapters/deepagents/adapter.py index 42b021c6d..c0a1a2cbc 100644 --- a/adapters/python/deepagents/src/nemo_fabric_adapters/deepagents/adapter.py +++ b/adapters/python/deepagents/src/nemo_fabric_adapters/deepagents/adapter.py @@ -269,6 +269,39 @@ def build_chat_model( return ChatOpenAI(**_supported_kwargs(ChatOpenAI, kwargs)), model_name, base_url +def _local_shell_backend(**kwargs: Any) -> Any: + from deepagents.backends import LocalShellBackend + + workspace = Path(kwargs["root_dir"]).absolute() + + class WorkspaceLocalShellBackend(LocalShellBackend): + """Accept the shell's workspace paths without weakening virtual file checks.""" + + def _to_virtual_path(self, path: Path) -> str: + # Listings and searches must return paths the resolver reads back + # unambiguously, even if a nested path repeats the workspace prefix. + virtual = super()._to_virtual_path(path) + # Deep Agents tools reject drive-prefixed paths. Windows virtual + # paths cannot collide with the drive-qualified physical aliases. + if self.cwd.drive: + return virtual + return str(self.cwd / virtual.lstrip("/")) + + def _resolve_path(self, key: str) -> Path: + path = Path(key) + if path.is_absolute(): + for root in (workspace, self.cwd): + try: + relative = path.relative_to(root) + except ValueError: + continue + # Native resolution still rejects traversal and symlink escapes. + return super()._resolve_path("/" + relative.as_posix()) + return super()._resolve_path(key) + + return WorkspaceLocalShellBackend(**kwargs) + + def resolve_backend( runtime_context: RuntimeContext, base_dir: str, @@ -290,9 +323,7 @@ def resolve_backend( root = Path(base_dir) / root if backend_config is not None: - from deepagents.backends import LocalShellBackend - - return LocalShellBackend( + return _local_shell_backend( root_dir=str(root), virtual_mode=True, env=dict(runtime_context.environment.env), @@ -838,7 +869,16 @@ async def _invoke( else None ), usage=( - AgentUsage(**tokens, cost_usd=cost) + # LangChain UsageMetadata input_tokens includes all input types, + # including cache. Preserve that convention for consumers such + # as Harbor without inventing a missing cache breakdown. + AgentUsage( + **tokens, + input_tokens_include_cache=( + True if "input_tokens" in tokens else None + ), + cost_usd=cost, + ) if tokens or cost is not None else None ), @@ -1436,6 +1476,7 @@ def _final_response(messages: list[dict[str, Any]]) -> Any: def _aggregate_usage(messages: list[dict[str, Any]]) -> dict[str, Any] | None: totals: dict[str, Any] = {} + unknown: set[str] = set() cost = 0.0 has_cost = False for message in messages: @@ -1452,6 +1493,10 @@ def _aggregate_usage(messages: list[dict[str, Any]]) -> dict[str, Any] | None: and 0 <= value <= (1 << 64) - 1 ): totals[target] = int(totals.get(target, 0)) + value + elif message.get("role") in ("ai", "assistant"): + # One missing model response makes this dimension incomplete; + # do not promote the other responses' partial sum to a total. + unknown.add(target) # Cost is not part of LangChain's UsageMetadata; surface it only when a # model/provider reports it on the usage or response metadata. candidate = ( @@ -1467,6 +1512,11 @@ def _aggregate_usage(messages: list[dict[str, Any]]) -> dict[str, Any] | None: has_cost = True if has_cost: totals["cost"] = cost + for target in unknown: + totals.pop(target, None) + for target in ("prompt_tokens", "completion_tokens", "total_tokens"): + if totals.get(target, 0) > (1 << 64) - 1: + totals.pop(target) return totals or None diff --git a/adapters/python/mini-swe-agent/README.md b/adapters/python/mini-swe-agent/README.md index 28ff55d01..467c93cce 100644 --- a/adapters/python/mini-swe-agent/README.md +++ b/adapters/python/mini-swe-agent/README.md @@ -23,7 +23,7 @@ environment: | `pip install nemo-fabric-adapters-mini-swe-agent` | No | Yes | No | No | The `harness` and `full` extras install the latest compatible mini-SWE-agent -2.x release. +2.x release and constrain LiteLLM below 1.106. LiteLLM 1.106.4 and 1.106.5 import MCP types unavailable in MCP 1.x during ordinary tool dispatch. The tested 1.105.6 dispatch works without an MCP runtime. This constraint does not enable MCP tools in the adapter. For provider `nvidia`, the adapter adds LiteLLM’s `nvidia_nim/` dispatch prefix while preserving the configured served model ID, including its vendor prefix. ## Configuration @@ -36,6 +36,32 @@ sampling fields through mini-SWE-agent's LiteLLM model arguments and rejects `harness.settings.timeout` to set the maximum duration of one command; the default is `30` seconds. +For a local or unpriced model, explicitly set +`models..settings={"cost_tracking": "disabled"}` and a positive +`runtime.max_turns`. The default is strict SDK pricing and its monetary budget; +ambient `MSWEA_COST_TRACKING` does not override it. Disabled tracking removes +the SDK monetary budget, so use it only when you accept execution without a +monetary cap. Turn and invocation time limits still apply. Cost remains +unavailable in Fabric output and Relay telemetry; the SDK's fallback zero is +not reported as measured cost. Provider token usage in Relay is preserved. + +For example, these fields explicitly opt into an unpriced model: + +```json +{ + "models": { + "default": { + "provider": "nvidia", + "model": "nvidia/nemotron-3-ultra-550b-a55b", + "base_url": "http://localhost:8000/v1", + "settings": {"cost_tracking": "disabled"} + } + }, + "runtime": {"max_turns": 50, "timeout_seconds": 1800} +} +``` + + Set `models..api_key_env` to the environment variable containing the model-provider credential. This adapter does not support MCP, skills, tool policy, or native OpenAI streaming. diff --git a/adapters/python/mini-swe-agent/mini-swe-agent.fabric-adapter.json b/adapters/python/mini-swe-agent/mini-swe-agent.fabric-adapter.json index 70a6e0929..70c720960 100644 --- a/adapters/python/mini-swe-agent/mini-swe-agent.fabric-adapter.json +++ b/adapters/python/mini-swe-agent/mini-swe-agent.fabric-adapter.json @@ -18,7 +18,14 @@ "base_url": {"type": "string", "minLength": 1}, "settings": { "type": "object", - "properties": {}, + "properties": { + "cost_tracking": { + "type": "string", + "enum": ["default", "disabled"], + "default": "default", + "description": "Use strict SDK pricing, or explicitly disable monetary budgeting for unpriced models. Disabled requires a positive runtime.max_turns; cost remains unavailable." + } + }, "additionalProperties": false } }, diff --git a/adapters/python/mini-swe-agent/pyproject.toml b/adapters/python/mini-swe-agent/pyproject.toml index 4ff30aa94..c7ad6d596 100644 --- a/adapters/python/mini-swe-agent/pyproject.toml +++ b/adapters/python/mini-swe-agent/pyproject.toml @@ -33,6 +33,8 @@ dependencies = [ harness = [ "fsspec>=2026.6.0", "mini-swe-agent>=2.0,<3", + # LiteLLM 1.106 tool dispatch imports MCP types unavailable in MCP 1.x. + "litellm<1.106", ] relay = [ "nemo-relay>=0.11,<0.12", @@ -40,6 +42,8 @@ relay = [ full = [ "fsspec>=2026.6.0", "mini-swe-agent>=2.0,<3", + # LiteLLM 1.106 tool dispatch imports MCP types unavailable in MCP 1.x. + "litellm<1.106", "nemo-relay>=0.11,<0.12", ] diff --git a/adapters/python/mini-swe-agent/src/nemo_fabric_adapters/mini_swe_agent/adapter.py b/adapters/python/mini-swe-agent/src/nemo_fabric_adapters/mini_swe_agent/adapter.py index 15df4f5c4..8339a8a2f 100644 --- a/adapters/python/mini-swe-agent/src/nemo_fabric_adapters/mini_swe_agent/adapter.py +++ b/adapters/python/mini-swe-agent/src/nemo_fabric_adapters/mini_swe_agent/adapter.py @@ -46,6 +46,7 @@ def __init__(self) -> None: self._relay_plugin = self._relay_scope = self._relay_scope_type = None self._relay_plugin_config: dict[str, Any] | None = None self._telemetry_quarantine: str | None = None + self._cost_tracking = "default" async def start(self, payload: dict[str, Any]) -> None: config: contract.AgentConfig = payload["config"] @@ -89,6 +90,27 @@ async def start(self, payload: dict[str, Any]) -> None: agent_type = RelayRetainingDefaultAgent model = _selected_model(config) + if model.settings.keys() - {"cost_tracking"}: + raise lifecycle.LifecycleError( + "mini_swe_agent_unsupported_model_settings", + "mini-SWE-agent supports only the cost_tracking model setting", + ) + self._cost_tracking = model.settings.get("cost_tracking", "default") + if self._cost_tracking not in ("default", "disabled"): + raise lifecycle.LifecycleError( + "mini_swe_agent_invalid_cost_tracking", + "models settings.cost_tracking must be 'default' or 'disabled'", + ) + if self._cost_tracking == "disabled" and ( + config.runtime is None + or config.runtime.max_turns is None + or config.runtime.max_turns <= 0 + ): + raise lifecycle.LifecycleError( + "mini_swe_agent_unbounded_unpriced_model", + "Disabling cost tracking requires a positive runtime.max_turns; " + "mini-SWE-agent cannot enforce a monetary budget without pricing", + ) model_kwargs: dict[str, Any] = {} if model.api_key_env and (api_key := os.environ.get(model.api_key_env)): model_kwargs["api_key"] = api_key @@ -100,17 +122,27 @@ async def start(self, payload: dict[str, Any]) -> None: model_kwargs["top_p"] = model.top_p if model.max_tokens is not None: model_kwargs["max_tokens"] = model.max_tokens + model_name = model.model + if model.provider == "nvidia" and not model_name.startswith("nvidia_nim/"): + # LiteLLM's dispatch prefix is separate from the NIM served model ID. + model_name = f"nvidia_nim/{model_name}" + elif "/" not in model_name: + model_name = f"{model.provider}/{model_name}" self._model = LitellmModel( - model_name=( - model.model if "/" in model.model else f"{model.provider}/{model.model}" - ), + model_name=model_name, model_kwargs=model_kwargs, + # Pin the default instead of inheriting MSWEA_COST_TRACKING. + cost_tracking=( + "ignore_errors" if self._cost_tracking == "disabled" else "default" + ), ) self._environment = LocalEnvironment( **(config.harness.settings if config.harness else {}) ) self._system_instruction = instruction.content if instruction else "" agent_kwargs: dict[str, Any] = {} + if self._cost_tracking == "disabled": + agent_kwargs["cost_limit"] = 0 if self._relay_enabled: agent_kwargs["relay_model_name"] = self._model.config.model_name self._agent = agent_type( @@ -141,6 +173,8 @@ async def invoke( "output": result.get("submission", ""), "usage": {"api_calls": self._agent.n_calls}, } + if self._cost_tracking == "disabled": + output["usage"].update(cost_tracking="disabled", cost_status="unavailable") if self._relay_enabled: output["telemetry"] = { "enabled": True, diff --git a/adapters/python/mini-swe-agent/src/nemo_fabric_adapters/mini_swe_agent/agents.py b/adapters/python/mini-swe-agent/src/nemo_fabric_adapters/mini_swe_agent/agents.py index 19e55749b..402040f12 100644 --- a/adapters/python/mini-swe-agent/src/nemo_fabric_adapters/mini_swe_agent/agents.py +++ b/adapters/python/mini-swe-agent/src/nemo_fabric_adapters/mini_swe_agent/agents.py @@ -187,7 +187,9 @@ def query(self) -> dict[str, Any]: metadata: dict[str, Any] = {"status": "succeeded"} try: message = super().query() - response = _relay_response(message) + response = _relay_response( + message, include_cost=self.model.config.cost_tracking != "ignore_errors" + ) return message except Exception as error: response = _relay_exception_response(error) @@ -266,7 +268,7 @@ def _end_tool_call( self._record_relay_error(error) -def _relay_response(message: dict[str, Any]) -> Any: +def _relay_response(message: dict[str, Any], *, include_cost: bool = True) -> Any: extra = message.get("extra", {}) response = extra.get("response") if not isinstance(response, dict): @@ -274,7 +276,7 @@ def _relay_response(message: dict[str, Any]) -> Any: response = deepcopy(response) cost = extra.get("cost") - if isinstance(cost, (int, float)) and not isinstance(cost, bool): + if include_cost and isinstance(cost, (int, float)) and not isinstance(cost, bool): usage = response.setdefault("usage", {}) if isinstance(usage, dict): usage.setdefault("cost", float(cost)) diff --git a/adapters/python/mini-swe-agent/uv.lock b/adapters/python/mini-swe-agent/uv.lock index c450e6e88..c2d53f80d 100644 --- a/adapters/python/mini-swe-agent/uv.lock +++ b/adapters/python/mini-swe-agent/uv.lock @@ -1211,11 +1211,13 @@ dependencies = [ [package.optional-dependencies] full = [ { name = "fsspec" }, + { name = "litellm" }, { name = "mini-swe-agent" }, { name = "nemo-relay" }, ] harness = [ { name = "fsspec" }, + { name = "litellm" }, { name = "mini-swe-agent" }, ] relay = [ @@ -1226,6 +1228,8 @@ relay = [ requires-dist = [ { name = "fsspec", marker = "extra == 'full'", specifier = ">=2026.6.0" }, { name = "fsspec", marker = "extra == 'harness'", specifier = ">=2026.6.0" }, + { name = "litellm", marker = "extra == 'full'", specifier = "<1.106" }, + { name = "litellm", marker = "extra == 'harness'", specifier = "<1.106" }, { name = "mini-swe-agent", marker = "extra == 'full'", specifier = ">=2.0,<3" }, { name = "mini-swe-agent", marker = "extra == 'harness'", specifier = ">=2.0,<3" }, { name = "nemo-fabric-adapter-contract", editable = "../../../adapter-contract/python" }, diff --git a/adapters/python/nooa/README.md b/adapters/python/nooa/README.md index 793fbe53a..5eee7418f 100644 --- a/adapters/python/nooa/README.md +++ b/adapters/python/nooa/README.md @@ -17,7 +17,9 @@ The shared `nvidia.fabric.nooa` adapter owns the common NOOA queue dispatcher. It also maps normalized skill paths and whole MCP servers into compatible registered targets. The dedicated `nvidia.fabric.nooa.bench-agent` adapter maps the benchmark-native -`BenchAgent` task contract directly into a NeMo Fabric invocation. +`BenchAgent` task contract directly into a NeMo Fabric invocation. Its +[observed token usage](docs/bench-agent.md#observed-token-usage) preserves the +SDK's inclusive input convention and documents missing-request limitations. ## Install @@ -28,6 +30,8 @@ tested NOOA packages together: pip install "nemo-fabric[nooa]" ``` +The `harness` and `full` extras constrain LiteLLM below 1.106 for NOOA and standalone BenchAgent runs. LiteLLM 1.106.4 and 1.106.5 import MCP types unavailable in MCP 1.x during ordinary tool dispatch. The tested 1.105.6 dispatch works without an MCP runtime. + For a split adapter environment, install `nemo-fabric-adapters-nooa[harness]`. Use the adapter package without an extra when the environment already manages compatible NOOA packages. The installed diff --git a/adapters/python/nooa/docs/bench-agent.md b/adapters/python/nooa/docs/bench-agent.md index f727f9f63..987688fe8 100644 --- a/adapters/python/nooa/docs/bench-agent.md +++ b/adapters/python/nooa/docs/bench-agent.md @@ -32,6 +32,21 @@ that `_run_evaluation()` replaces. During runtime shutdown, it closes the active shell and model. Native errors are not copied into the public result because they can contain model, tool, or environment details. +## Observed Token Usage + +The adapter resets NOOA's task accumulator for each invocation. Positive input +and output totals from the native metrics bridge are preserved on successful +and failed results. NOOA forwards LiteLLM's inclusive `prompt_tokens` count, so +input totals already include cached tokens. The bridge does not retain cache +breakdowns or provider cost; those fields remain unknown. + +NOOA 0.0.10 initializes absent usage to zero. Zero dimensions remain unknown +rather than being presented as measured zero. These are SDK-observed totals, +not an exhaustive billing audit: missing per-request usage and unsuccessful +retry attempts can be omitted upstream, and the public aggregate does not +identify those omissions. The adapter does not reconstruct missing requests or +add estimated cache, reasoning, or cost components. + ## Run with Harbor Harbor selects the adapter through `FabricAgent` with diff --git a/adapters/python/nooa/pyproject.toml b/adapters/python/nooa/pyproject.toml index 59e168c44..40bf4b61f 100644 --- a/adapters/python/nooa/pyproject.toml +++ b/adapters/python/nooa/pyproject.toml @@ -37,6 +37,8 @@ harness = [ "nooa>=0.0.10,<0.0.11", "nooa-cli>=0.0.10,<0.0.11", "nooa-bench>=0.0.10,<0.0.11", + # LiteLLM 1.106 tool dispatch imports MCP types unavailable in MCP 1.x. + "litellm<1.106", ] relay = [ "nemo-relay>=0.11,<0.12", @@ -46,6 +48,8 @@ full = [ "nooa>=0.0.10,<0.0.11", "nooa-cli>=0.0.10,<0.0.11", "nooa-bench>=0.0.10,<0.0.11", + # LiteLLM 1.106 tool dispatch imports MCP types unavailable in MCP 1.x. + "litellm<1.106", "nemo-relay>=0.11,<0.12", ] diff --git a/adapters/python/nooa/src/nemo_fabric_adapters/nooa/bench_adapter.py b/adapters/python/nooa/src/nemo_fabric_adapters/nooa/bench_adapter.py index ff103932b..02e3a4a83 100644 --- a/adapters/python/nooa/src/nemo_fabric_adapters/nooa/bench_adapter.py +++ b/adapters/python/nooa/src/nemo_fabric_adapters/nooa/bench_adapter.py @@ -113,7 +113,9 @@ async def _close_shell(agent: Any) -> None: await _await_if_needed(close()) -def _failure_output(code: str, message: str) -> AgentRunResult: +def _failure_output( + code: str, message: str, usage: AgentUsage | None = None +) -> AgentRunResult: return AgentRunResult( status=AgentRunStatus.FAILED, output={ @@ -124,6 +126,7 @@ def _failure_output(code: str, message: str) -> AgentRunResult: "completed": False, }, error=AgentRunError(code=code, message=message), + usage=usage, ) @@ -141,10 +144,23 @@ def _usage(tokens: Any) -> AgentUsage | None: return None if input_tokens < 0 or output_tokens < 0: return None + # NOOA initializes the accumulator to zero and substitutes zero for missing + # provider dimensions. Only positive observations distinguish reported usage. + observed_input = input_tokens if input_tokens > 0 else None + observed_output = output_tokens if output_tokens > 0 else None + if observed_input is None and observed_output is None: + return None return AgentUsage( - input_tokens=input_tokens, - output_tokens=output_tokens, - total_tokens=input_tokens + output_tokens, + input_tokens=observed_input, + output_tokens=observed_output, + # unifiedllm passes LiteLLM prompt_tokens through unchanged: its inclusive + # prompt count already contains cache hits. Do not add them again. + input_tokens_include_cache=True if observed_input is not None else None, + total_tokens=( + observed_input + observed_output + if observed_input is not None and observed_output is not None + else None + ), ) @@ -304,6 +320,7 @@ async def invoke_agent() -> AgentRunResult: } if self._instruction is not None: task_input["instructions"] = self._instruction + read_usage: Callable[[], AgentUsage | None] | None = None try: from nooa.runtime.token_usage import get_task_tokens from nooa.runtime.token_usage import start_task_tokens @@ -312,14 +329,16 @@ async def invoke_agent() -> AgentRunResult: # previous one first so persistent Fabric runtimes do not leak it. await _close_shell(agent) start_task_tokens() + read_usage = lambda: _usage(get_task_tokens()) native = await agent._run_evaluation(task_input) - usage = _usage(get_task_tokens()) + usage = read_usage() if not isinstance(native, dict): raise ValueError("BenchAgent result must be an object") if native.get("success") is not True: return _failure_output( "nooa_bench_task_failed", "OO Agents BenchAgent reported task failure; inspect adapter stderr for details", + usage=usage, ) return _success_output(native, usage) except asyncio.CancelledError: @@ -332,6 +351,7 @@ async def invoke_agent() -> AgentRunResult: return _failure_output( "nooa_bench_invoke_failed", "OO Agents BenchAgent invocation failed; inspect adapter stderr for details", + usage=read_usage() if read_usage is not None else None, ) try: diff --git a/adapters/python/nooa/uv.lock b/adapters/python/nooa/uv.lock index c8e0416e5..7e19e2417 100644 --- a/adapters/python/nooa/uv.lock +++ b/adapters/python/nooa/uv.lock @@ -684,6 +684,7 @@ dependencies = [ [package.optional-dependencies] full = [ { name = "fsspec" }, + { name = "litellm" }, { name = "nemo-relay" }, { name = "nooa" }, { name = "nooa-bench" }, @@ -691,6 +692,7 @@ full = [ ] harness = [ { name = "fsspec" }, + { name = "litellm" }, { name = "nooa" }, { name = "nooa-bench" }, { name = "nooa-cli" }, @@ -703,6 +705,8 @@ relay = [ requires-dist = [ { name = "fsspec", marker = "extra == 'full'", specifier = ">=2026.6.0" }, { name = "fsspec", marker = "extra == 'harness'", specifier = ">=2026.6.0" }, + { name = "litellm", marker = "extra == 'full'", specifier = "<1.106" }, + { name = "litellm", marker = "extra == 'harness'", specifier = "<1.106" }, { name = "nemo-fabric-adapter-contract", editable = "../../../adapter-contract/python" }, { name = "nemo-fabric-adapters-common", editable = "../common" }, { name = "nemo-relay", marker = "extra == 'full'", specifier = ">=0.11,<0.12" }, diff --git a/adapters/python/openclaw/README.md b/adapters/python/openclaw/README.md index 962b029c2..87e44db1e 100644 --- a/adapters/python/openclaw/README.md +++ b/adapters/python/openclaw/README.md @@ -66,6 +66,24 @@ adapter generates an OpenClaw custom provider using the `openai-completions` API adapter. Set `api_key_env` when that provider requires an API key. +If that endpoint supports `stream_options.include_usage`, set +`models..settings.supports_usage_in_streaming` to `true`. This maps to +the model's native `compat.supportsUsageInStreaming` setting; it requests +usage from the model endpoint, separately from the Gateway's trailing usage +chunk. Set it to `false` for endpoints that reject streaming usage, or omit +it to retain OpenClaw's endpoint capability detection. This setting requires +`base_url` and is unavailable when attaching to a deployment-owned service. + +OpenClaw 2026.9.4's Chat Completions Gateway can return aggregate usage or +fall back to the last model call, and synthesizes zero counters when usage +is missing. The response does not identify which source it used. The adapter +therefore leaves standard complete usage counters unknown and preserves the +reported counters under +`usage.extensions["nvidia.fabric.openclaw.usage_evidence.v1"].observed`, with +`scope_unverified` in `gaps`. Missing usage adds `missing_usage`; all-zero +observations add `zero_unverified`. These observations do not establish +complete accounting for retries, compaction, child agents, or background work. + ### Harness Settings Only OpenClaw's default `openclaw` agent runtime is supported. diff --git a/adapters/python/openclaw/openclaw.fabric-adapter.json b/adapters/python/openclaw/openclaw.fabric-adapter.json index 44609cfd8..862c01b2d 100644 --- a/adapters/python/openclaw/openclaw.fabric-adapter.json +++ b/adapters/python/openclaw/openclaw.fabric-adapter.json @@ -103,13 +103,48 @@ "max_tokens": {"type": "integer", "minimum": 1}, "settings": { "type": "object", - "properties": {}, + "properties": { + "supports_usage_in_streaming": { + "type": "boolean", + "description": "Whether the configured custom OpenAI-compatible endpoint supports stream_options.include_usage. Requires base_url; omitted values retain OpenClaw capability detection." + } + }, "additionalProperties": false } }, "required": ["provider", "model"], "additionalProperties": false }, + "extension_schemas": { + "usage": { + "type": "object", + "properties": { + "nvidia.fabric.openclaw.usage_evidence.v1": { + "type": "object", + "properties": { + "observed": { + "type": "object", + "properties": { + "input_tokens": {"type": "integer", "minimum": 0}, + "output_tokens": {"type": "integer", "minimum": 0}, + "total_tokens": {"type": "integer", "minimum": 0} + }, + "additionalProperties": false + }, + "gaps": { + "type": "array", + "items": {"enum": ["scope_unverified", "missing_usage", "zero_unverified"]}, + "uniqueItems": true + } + }, + "required": ["observed", "gaps"], + "additionalProperties": false + } + }, + "required": ["nvidia.fabric.openclaw.usage_evidence.v1"], + "additionalProperties": false + } + }, "requirements": {}, "config": { "accepts": [ diff --git a/adapters/python/openclaw/src/nemo_fabric_adapters/openclaw/adapter.py b/adapters/python/openclaw/src/nemo_fabric_adapters/openclaw/adapter.py index c05537196..b643d97c0 100644 --- a/adapters/python/openclaw/src/nemo_fabric_adapters/openclaw/adapter.py +++ b/adapters/python/openclaw/src/nemo_fabric_adapters/openclaw/adapter.py @@ -49,6 +49,7 @@ OPENCLAW_CHAT_MODEL = "openclaw/default" OPENCLAW_SERVICE_TYPE = "openclaw_gateway" OPENCLAW_ADAPTER_ID = "nvidia.fabric.openclaw" +OPENCLAW_USAGE_EVIDENCE = "nvidia.fabric.openclaw.usage_evidence.v1" SUPPORTED_OPENCLAW_VERSIONS = frozenset({"2026.9.4"}) MAX_PORT_ATTEMPTS = 20 MAX_DERIVED_PORT_OFFSET = 110 @@ -104,6 +105,8 @@ def _validate_attach_config(config: contract.AgentConfig) -> None: unsupported.append("models.base_url") if model.api_key_env is not None: unsupported.append("models.api_key_env") + if "supports_usage_in_streaming" in model.settings: + unsupported.append("models.settings.supports_usage_in_streaming") if config.tools is not None: unsupported.append("tools") if config.mcp is not None and config.mcp.servers: @@ -461,6 +464,15 @@ def _openclaw_config( ) -> dict[str, Any]: model = _selected_model(config) model_id = _model_id(model) + streaming_usage = model.settings.get("supports_usage_in_streaming") + if "supports_usage_in_streaming" in model.settings and ( + type(streaming_usage) is not bool or model.base_url is None + ): + raise lifecycle.LifecycleError( + "openclaw_invalid_configuration", + "OpenClaw supports_usage_in_streaming must be a boolean and requires models.base_url", + metadata={"field": "models.settings.supports_usage_in_streaming"}, + ) model_ref = f"{model.provider}/{model_id}" params = { key: value @@ -533,6 +545,10 @@ def _openclaw_config( ) if model.max_tokens is not None: provider["models"][0]["maxTokens"] = model.max_tokens + if streaming_usage is not None: + provider["models"][0]["compat"] = { + "supportsUsageInStreaming": streaming_usage + } if model.api_key_env is not None: provider["apiKey"] = { "source": "env", @@ -565,8 +581,8 @@ async def _invoke_gateway( max_tokens: int | None, user: str, agent_id: str = "default", -) -> tuple[str, contract.AgentUsage | None]: - return await openai_chat.invoke( +) -> tuple[str, contract.AgentUsage]: + text, usage = await openai_chat.invoke( client, url, model=f"openclaw/{agent_id}", @@ -576,6 +592,36 @@ async def _invoke_gateway( max_tokens=max_tokens, user=user, ) + return text, _gateway_usage_evidence(usage) + + +def _gateway_usage_evidence( + usage: contract.AgentUsage | None, +) -> contract.AgentUsage: + # OpenClaw 2026.9.4's public endpoint falls back from aggregate usage to + # the last model call, and synthesizes zero counters when both are missing. + # The response does not identify which source it used. + observed = ( + { + key: value + for key, value in { + "input_tokens": usage.input_tokens, + "output_tokens": usage.output_tokens, + "total_tokens": usage.total_tokens, + }.items() + if value is not None + } + if usage is not None + else {} + ) + gaps = ["scope_unverified"] + if not observed: + gaps.append("missing_usage") + elif not any(observed.values()): + gaps.append("zero_unverified") + return contract.AgentUsage( + extensions={OPENCLAW_USAGE_EVIDENCE: {"observed": observed, "gaps": gaps}} + ) async def _command_output( diff --git a/adapters/python/openhands/README.md b/adapters/python/openhands/README.md index 65e362270..28c513907 100644 --- a/adapters/python/openhands/README.md +++ b/adapters/python/openhands/README.md @@ -79,6 +79,11 @@ directory. Each skill path must be a directory containing `SKILL.md`. The adapter disables ambient OpenHands user, public, project, and memory loading; only explicitly configured NeMo Fabric skills are loaded. +For `provider="nvidia"`, the adapter adds LiteLLM's `nvidia_nim/` dispatch +prefix while preserving the served model ID, including its `nvidia/` +namespace. This also supports self-hosted NIM endpoints through `base_url`. +An existing `nvidia_nim/` prefix is retained once. + The adapter configures the OpenHands SDK standard summarizing condenser, `default_condenser`, with the agent model. When the history sent to the model exceeds 80 events, or exceeds the model input token limit when OpenHands knows @@ -130,6 +135,16 @@ closes the conversation and removes the adapter-owned temporary profile state. Create a new runtime to change the model, workspace, tools, MCP servers, or skills. +Usage requires invocation-local SDK token records. If a request fails before +the SDK records usage, or a response contains no usage, the adapter leaves +token counters unknown instead of reporting the SDK's zero-initialized +accumulator. Recorded tokens remain available after a later invocation failure. +Cost is unknown unless the SDK records a cost for every returned response; +an unmapped model's default accumulated cost does not establish a free request. +Reported input tokens include cached input, following LiteLLM's normalized +`prompt_tokens` convention. The adapter sets `input_tokens_include_cache` to +`true` and does not add the cache count again or infer a missing cache breakdown. + ## Runtime Health The adapter advertises `live` and `active` checks qualified against OpenHands SDK 1.50.0. `live` reads the owned `LocalConversation` native closure flags. `active` also requires the native lazy-initialization flag to confirm that the conversation's request interface has initialized. Before the first invocation initializes the conversation, `active` returns `unknown/capabilities_unverified`. Checks do not initialize the conversation. diff --git a/adapters/python/openhands/src/nemo_fabric_adapters/openhands/adapter.py b/adapters/python/openhands/src/nemo_fabric_adapters/openhands/adapter.py index 429347582..aa61a36e7 100644 --- a/adapters/python/openhands/src/nemo_fabric_adapters/openhands/adapter.py +++ b/adapters/python/openhands/src/nemo_fabric_adapters/openhands/adapter.py @@ -109,9 +109,12 @@ def _selected_model(config: contract.AgentConfig) -> contract.AgentModelConfig: def _llm(api: OpenHandsApi, model: contract.AgentModelConfig) -> Any: - wire_model = ( - model.model if "/" in model.model else f"{model.provider}/{model.model}" - ) + wire_model = model.model + if model.provider == "nvidia" and not wire_model.startswith("nvidia_nim/"): + # LiteLLM's dispatch prefix is separate from the NIM served model ID. + wire_model = f"nvidia_nim/{wire_model}" + elif "/" not in wire_model: + wire_model = f"{model.provider}/{wire_model}" kwargs: dict[str, Any] = {"model": wire_model, "usage_id": "agent"} if model.api_key_env is not None: api_key = os.environ.get(model.api_key_env) @@ -316,15 +319,24 @@ def _mcp_servers(api: OpenHandsApi, config: contract.AgentConfig) -> dict[str, A def _usage(api_llm: Any, baseline: Any) -> contract.AgentUsage | None: metrics = api_llm.metrics.diff(baseline) tokens = metrics.accumulated_token_usage - if tokens is None: + # Metrics initializes an all-zero accumulator even before the first response. + # Only per-call records establish that the SDK observed token usage. + if tokens is None or not metrics.token_usages: return None input_tokens = int(tokens.prompt_tokens) output_tokens = int(tokens.completion_tokens) return contract.AgentUsage( input_tokens=input_tokens, + # LiteLLM normalizes prompt_tokens to include cache reads and writes; + # OpenHands copies that count unchanged into its per-call metrics. + input_tokens_include_cache=True, output_tokens=output_tokens, total_tokens=input_tokens + output_tokens, - cost_usd=float(metrics.accumulated_cost), + # The SDK omits cost records when pricing fails. Its zero-initialized + # accumulated cost alone cannot distinguish missing pricing from free. + cost_usd=float(metrics.accumulated_cost) + if metrics.costs and len(metrics.costs) == len(metrics.response_latencies) + else None, ) diff --git a/adapters/typescript/kilo/README.md b/adapters/typescript/kilo/README.md index 3d80d3c5c..8ad145039 100644 --- a/adapters/typescript/kilo/README.md +++ b/adapters/typescript/kilo/README.md @@ -113,6 +113,8 @@ The Kilo Code harness is not embedded in the adapter process. `start` launches creates one session. `stop` deletes the session, terminates that server, and removes its temporary profile. +Kilo Code's `background_process` tool stops session-scoped processes when the session ends or the Kilo Code server exits. For a service that must survive runtime shutdown, set `persistent: true` on that tool call. This matters in Harbor, where verification runs after the runtime stops. Persistent processes require separate cleanup; Harbor removes them when it destroys the task container. + ```mermaid flowchart TB Fabric["NeMo Fabric runtime"] diff --git a/adapters/typescript/package-lock.json b/adapters/typescript/package-lock.json index 176385210..2b2d0fb54 100644 --- a/adapters/typescript/package-lock.json +++ b/adapters/typescript/package-lock.json @@ -8867,6 +8867,7 @@ }, "devDependencies": { "@earendil-works/pi-ai": "1.0.3", + "@earendil-works/pi-codemode": "1.0.3", "@earendil-works/pi-coding-agent": "1.0.3", "@earendil-works/pi-mcp": "1.0.3", "@types/node": "24.12.4", @@ -8876,14 +8877,18 @@ "node": ">=22.19.0" }, "peerDependencies": { - "@earendil-works/pi-ai": "^1.0.0", - "@earendil-works/pi-coding-agent": "^1.0.0", - "@earendil-works/pi-mcp": "^1.0.0" + "@earendil-works/pi-ai": "1.0.3", + "@earendil-works/pi-codemode": "1.0.3", + "@earendil-works/pi-coding-agent": "1.0.3", + "@earendil-works/pi-mcp": "1.0.3" }, "peerDependenciesMeta": { "@earendil-works/pi-ai": { "optional": true }, + "@earendil-works/pi-codemode": { + "optional": true + }, "@earendil-works/pi-coding-agent": { "optional": true }, diff --git a/adapters/typescript/pi/README.md b/adapters/typescript/pi/README.md index eb1f99f7a..06fab9b38 100644 --- a/adapters/typescript/pi/README.md +++ b/adapters/typescript/pi/README.md @@ -46,6 +46,23 @@ configured extensions are trusted code. The adapter raises the compaction reserve toward the selected model's maximum output while retaining at least half of the context window for input. +## Invocation Usage + +Each invocation reads only new entries from Pi's raw session ledger, including +retained failed attempts, compaction and branch summaries, nested tool usage, +and auxiliary usage records. Automatic cache warming is disabled so background +refresh calls cannot escape the invocation boundary. Session history and previous invocations are not +counted again. Input totals include cache reads and cache writes; output totals +already include reasoning tokens. Available breakdowns are returned through +`nemo_fabric.detailed_token_usage.v1`. + +Missing or inconsistent usage leaves the complete counters unknown. Pi 1.0.3 +initializes unreported provider usage to zero, so an all-zero record also remains +unknown. A failed or retried summary can omit attempt usage from the ledger; +complete counters remain unknown in that case. Nonzero records can retain real +zero input, output, or cache dimensions. Configured-rate SDK cost estimates are +not published as provider-reported `cost_usd`. + ## Install the Adapter Pi 1.0 requires Node.js 22.19.0 or newer. @@ -53,17 +70,21 @@ Pi 1.0 requires Node.js 22.19.0 or newer. ### Install for Consumers Install the adapter in the project that owns the NeMo Fabric configuration, -then install the compatible Pi SDK harness version selected by that project: +then install the supported Pi SDK package set: ```bash npm install nemo-fabric-adapters-pi -npm install @earendil-works/pi-ai@^1.0.0 @earendil-works/pi-coding-agent@^1.0.0 @earendil-works/pi-mcp@^1.0.0 +npm install @earendil-works/pi-ai@1.0.3 @earendil-works/pi-coding-agent@1.0.3 @earendil-works/pi-mcp@1.0.3 @earendil-works/pi-codemode@1.0.3 ``` The adapter declares the Pi packages as optional peers. Installing the adapter alone does not install a harness, and starting it without compatible Pi packages returns `pi_harness_unavailable`. +The supported SDK packages are pinned together at 1.0.3. The coding-agent +package otherwise permits newer `pi-codemode` versions with incompatible exports; +installing all four packages keeps fresh installs on the tested combination. + ### Install for Source Development For focused Pi development in the NeMo Fabric source tree, install the Pi diff --git a/adapters/typescript/pi/package.json b/adapters/typescript/pi/package.json index 0700e7698..ae81abe61 100644 --- a/adapters/typescript/pi/package.json +++ b/adapters/typescript/pi/package.json @@ -47,9 +47,10 @@ "nemo-fabric-adapters-common": "0.6.0" }, "peerDependencies": { - "@earendil-works/pi-ai": "^1.0.0", - "@earendil-works/pi-coding-agent": "^1.0.0", - "@earendil-works/pi-mcp": "^1.0.0" + "@earendil-works/pi-ai": "1.0.3", + "@earendil-works/pi-coding-agent": "1.0.3", + "@earendil-works/pi-mcp": "1.0.3", + "@earendil-works/pi-codemode": "1.0.3" }, "peerDependenciesMeta": { "@earendil-works/pi-ai": { @@ -60,6 +61,9 @@ }, "@earendil-works/pi-mcp": { "optional": true + }, + "@earendil-works/pi-codemode": { + "optional": true } }, "devDependencies": { @@ -67,6 +71,7 @@ "@earendil-works/pi-coding-agent": "1.0.3", "@earendil-works/pi-mcp": "1.0.3", "@types/node": "24.12.4", - "typescript": "5.9.3" + "typescript": "5.9.3", + "@earendil-works/pi-codemode": "1.0.3" } } diff --git a/adapters/typescript/pi/pi.fabric-adapter.json b/adapters/typescript/pi/pi.fabric-adapter.json index 3ecab5b48..80bb66e36 100644 --- a/adapters/typescript/pi/pi.fabric-adapter.json +++ b/adapters/typescript/pi/pi.fabric-adapter.json @@ -113,6 +113,48 @@ }, "additionalProperties": false }, + "usage": { + "type": "object", + "properties": { + "nemo_fabric.detailed_token_usage.v1": { + "type": "object", + "properties": { + "input_tokens": { + "type": "integer", + "minimum": 0 + }, + "output_tokens": { + "type": "integer", + "minimum": 0 + }, + "cached_input_tokens": { + "type": "integer", + "minimum": 0 + }, + "cache_write_input_tokens": { + "type": "integer", + "minimum": 0 + }, + "reasoning_tokens": { + "type": "integer", + "minimum": 0 + }, + "total_tokens": { + "type": "integer", + "minimum": 0 + }, + "input_tokens_include_cache": { + "type": "boolean" + }, + "output_tokens_include_reasoning": { + "type": "boolean" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, "run_result": { "type": "object", "properties": { diff --git a/adapters/typescript/pi/src/pi-sdk.ts b/adapters/typescript/pi/src/pi-sdk.ts index d7974f907..c02423c9a 100644 --- a/adapters/typescript/pi/src/pi-sdk.ts +++ b/adapters/typescript/pi/src/pi-sdk.ts @@ -28,6 +28,7 @@ import { type PiRelayControllerFactory, type PiRelayRuntime, } from "./relay.js"; +import { invocationUsage } from "./usage.js"; import type { PiPromptOutcome, PiSessionFactory, PiSessionHandle } from "./runtime.js"; // ProviderConfigInput isn't re-exported from pi's public entrypoint; derive it from @@ -106,7 +107,7 @@ type PiToolFactory = (context: PiToolFactoryContext) => ToolDefinition | Promise const PI_BUILTIN_TOOL_NAMES = new Set(["read", "bash", "edit", "write", "grep", "find", "ls"]); const TOOL_MODULE_EXTENSIONS = new Set([".js", ".mjs", ".cjs", ".ts", ".mts", ".cts"]); const PI_HARNESS_INSTALL_COMMAND = - "npm install @earendil-works/pi-ai@^1.0.0 @earendil-works/pi-coding-agent@^1.0.0 @earendil-works/pi-mcp@^1.0.0"; + "npm install @earendil-works/pi-ai@1.0.3 @earendil-works/pi-coding-agent@1.0.3 @earendil-works/pi-mcp@1.0.3 @earendil-works/pi-codemode@1.0.3"; const MCP_STARTUP_TIMEOUT_MS = 10_000; interface PiSdkModules { @@ -942,12 +943,19 @@ class PiSdkSessionHandle implements PiSessionHandle { } async prompt(text: string): Promise { + const before = this.session.sessionManager.getEntries().map((entry) => entry.id); + let accountingComplete = true; let accepted = false; let turnStarted = false; let finalAssistant: | { role: "assistant"; content: unknown; stopReason: string; errorMessage?: string } | undefined; const unsubscribe = this.session.subscribe((event) => { + if (event.type === "summarization_retry_scheduled" || + (event.type === "compaction_end" && (event.aborted || event.errorMessage !== undefined))) { + // Failed summary attempts do not survive in Pi's usage ledger. + accountingComplete = false; + } if (event.type === "turn_start") { turnStarted = true; } @@ -967,6 +975,7 @@ class PiSdkSessionHandle implements PiSessionHandle { unsubscribe(); } return { + usage: accountingComplete ? invocationUsage(before, this.session.sessionManager.getEntries()) : undefined, accepted, turnStarted, turnCount: this.cumulativeTurnCount, @@ -1060,7 +1069,9 @@ export class PiSdkSessionFactory implements PiSessionFactory { throw new LifecycleError("pi_credential_missing", `Credential environment variable ${apiKeyEnv} is not set`); } - const settings = pi.SettingsManager.inMemory({}, { projectTrusted: false }); + // Cache warming can finish outside the prompt and omit aborted refresh usage. + // Keep all adapter-owned model work inside the invocation ledger boundary. + const settings = pi.SettingsManager.inMemory({ cacheWarming: "off" }, { projectTrusted: false }); const extensionPaths = await resolveExtensionPaths(workspace, harnessSettings(input.config).extensions); const skillPaths = await resolveSkillPaths(input.baseDir, input.config.skills?.paths ?? []); const customTools = await resolveCustomTools(workspace, input.config.tools?.definitions ?? {}); diff --git a/adapters/typescript/pi/src/runtime.ts b/adapters/typescript/pi/src/runtime.ts index cd67f7446..e16a3665d 100644 --- a/adapters/typescript/pi/src/runtime.ts +++ b/adapters/typescript/pi/src/runtime.ts @@ -5,7 +5,7 @@ // to a PiSessionHandle and normalized results while keeping SDK construction // behind a factory for focused lifecycle testing. -import type { AdapterCheckResult, CheckLevel, AgentRunRequest, AgentRunResult, JsonObject, RuntimeContext } from "nemo-fabric-adapter-contract"; +import type { AdapterCheckResult, CheckLevel, AgentRunRequest, AgentRunResult, AgentUsage, JsonObject, RuntimeContext } from "nemo-fabric-adapter-contract"; import { LifecycleError, unobservedHealth, type AdapterRuntime, type AdapterStartInput } from "nemo-fabric-adapters-common"; import { collectRelayArtifacts, type RelayArtifact } from "./relay-config.js"; @@ -14,6 +14,7 @@ import type { PiRelayRuntime } from "./relay.js"; export type PiStopReason = "stop" | "length" | "toolUse" | "error" | "aborted" | string; export interface PiPromptOutcome { + usage?: AgentUsage; accepted: boolean; turnStarted: boolean; turnCount: number; @@ -118,6 +119,10 @@ export class PiAdapterRuntime implements AdapterRuntime { const relay = this.session.relay; const outcome = await this.session.prompt(request.input); + const annotate = (result: AgentRunResult) => withRelayOutput( + { ...result, ...(outcome.usage === undefined ? {} : { usage: outcome.usage }) }, + relay, outcome.turnStarted, outcome.turnCount, + ); if ( !Number.isInteger(outcome.turnCount) || outcome.turnCount < this.turnCount || @@ -127,18 +132,15 @@ export class PiAdapterRuntime implements AdapterRuntime { } this.turnCount = outcome.turnCount; if (!outcome.accepted) { - return withRelayOutput( + return annotate( failed("pi_prompt_rejected", "Pi rejected the prompt before starting an agent run"), - relay, - outcome.turnStarted, - outcome.turnCount, ); } if (outcome.shutdownRequested || outcome.stopReason === "aborted") { if (outcome.shutdownRequested) { this.unusable = true; } - return withRelayOutput( + return annotate( { status: "cancelled", output: null, @@ -150,32 +152,20 @@ export class PiAdapterRuntime implements AdapterRuntime { retryable: false, }, }, - relay, - outcome.turnStarted, - outcome.turnCount, ); } if (outcome.stopReason === "error") { - return withRelayOutput( + return annotate( failed("pi_model_error", outcome.errorMessage || "The Pi model invocation failed"), - relay, - outcome.turnStarted, - outcome.turnCount, ); } if (outcome.text === undefined || outcome.text.length === 0) { - return withRelayOutput( + return annotate( failed("pi_no_assistant_response", "Pi completed without a final assistant text response"), - relay, - outcome.turnStarted, - outcome.turnCount, ); } - return withRelayOutput( + return annotate( { status: "succeeded", output: { response: outcome.text } }, - relay, - outcome.turnStarted, - outcome.turnCount, ); } diff --git a/adapters/typescript/pi/src/usage.ts b/adapters/typescript/pi/src/usage.ts new file mode 100644 index 000000000..8c42976b3 --- /dev/null +++ b/adapters/typescript/pi/src/usage.ts @@ -0,0 +1,106 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import type { AgentSession } from "@earendil-works/pi-coding-agent"; +import type { AgentUsage } from "nemo-fabric-adapter-contract"; + +type Entries = ReturnType; + +// Pi 1.0.3 keeps retries and compacted/other-branch messages in its raw ledger. +// Its projected conversation and final assistant message omit billable calls. +export function invocationUsage( + before: readonly string[], + entries: Entries, +): AgentUsage | undefined { + if ( + entries.length < before.length || + before.some((id, index) => entries[index]?.id !== id) + ) + return undefined; + let input = 0; + let output = 0; + let cacheRead = 0; + let cacheWrite = 0; + let reasoning = 0; + let hasReasoning = true; + let count = 0; + for (const entry of entries.slice(before.length)) { + let usage; + if ( + entry.type === "usage" || + entry.type === "compaction" || + entry.type === "branch_summary" + ) { + usage = entry.usage; + if (usage === undefined) return undefined; + } else if (entry.type === "message" && entry.message.role === "assistant") { + usage = entry.message.usage; + } else if ( + entry.type === "message" && + entry.message.role === "toolResult" + ) { + // Ordinary tools need no tokens. Pi attaches nested model usage when present. + usage = entry.message.usage; + if (usage === undefined) continue; + } else { + continue; + } + if (usage === undefined) return undefined; + const counters = [ + usage.input, + usage.output, + usage.cacheRead, + usage.cacheWrite, + usage.totalTokens, + ]; + if (counters.some((value) => !Number.isSafeInteger(value) || value < 0)) + return undefined; + const total = + usage.input + usage.output + usage.cacheRead + usage.cacheWrite; + // Pi initializes missing provider usage to all zeros. It cannot distinguish + // a genuine all-zero response from an unreported call; neither proves totals. + if (total === 0 || total !== usage.totalTokens) return undefined; + if ( + usage.reasoning !== undefined && + (!Number.isSafeInteger(usage.reasoning) || + usage.reasoning < 0 || + usage.reasoning > usage.output) + ) + return undefined; + input += usage.input; + output += usage.output; + cacheRead += usage.cacheRead; + cacheWrite += usage.cacheWrite; + if (usage.reasoning === undefined) hasReasoning = false; + else reasoning += usage.reasoning; + count += 1; + } + if (count === 0) return undefined; + // Pi's input excludes reads AND writes. Its output already includes reasoning. + input += cacheRead + cacheWrite; + const total = input + output; + if ( + ![input, output, cacheRead, cacheWrite, total].every(Number.isSafeInteger) + ) + return undefined; + return { + input_tokens: input, + output_tokens: output, + cached_input_tokens: cacheRead, + input_tokens_include_cache: true, + total_tokens: total, + // SDK cost is computed using configured/catalog rates, not provider billing. + extensions: { + "nemo_fabric.detailed_token_usage.v1": { + input_tokens: input, + output_tokens: output, + cached_input_tokens: cacheRead, + cache_write_input_tokens: cacheWrite, + input_tokens_include_cache: true, + output_tokens_include_reasoning: true, + ...(hasReasoning ? { reasoning_tokens: reasoning } : {}), + total_tokens: total, + }, + }, + }; +} diff --git a/adapters/typescript/pi/test/usage-sdk.test.mjs b/adapters/typescript/pi/test/usage-sdk.test.mjs new file mode 100644 index 000000000..e504bf473 --- /dev/null +++ b/adapters/typescript/pi/test/usage-sdk.test.mjs @@ -0,0 +1,176 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import assert from "node:assert/strict"; +import { mkdtemp, realpath, rm } from "node:fs/promises"; +import { createServer } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; + +import { PiSdkSessionFactory } from "../dist/pi-sdk.js"; +import { invocationUsage } from "../dist/usage.js"; + +const [major, minor] = process.versions.node.split(".").map(Number); +const skip = + major > 22 || (major === 22 && minor >= 19) + ? false + : "Pi requires Node >=22.19"; + +test( + "real SDK raw ledger retains other branches and compaction usage", + { skip }, + async () => { + const { SessionManager } = await import("@earendil-works/pi-coding-agent"); + const manager = SessionManager.inMemory("/tmp"); + const usage = { + input: 10, + output: 3, + cacheRead: 2, + cacheWrite: 1, + totalTokens: 16, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }; + const message = { + role: "assistant", + content: [{ type: "text", text: "answer" }], + api: "openai-completions", + provider: "test", + model: "test", + stopReason: "stop", + timestamp: Date.now(), + usage, + }; + const old = manager.appendMessage(message); + const before = manager.getEntries().map((entry) => entry.id); + manager.appendMessage({ ...message, stopReason: "error" }); + manager.branch(old); + const current = manager.appendMessage(message); + manager.appendCompaction("summary", current, 50, undefined, false, usage); + assert.ok(manager.getEntries().length > manager.getBranch().length); + const result = invocationUsage(before, manager.getEntries()); + assert.equal(result.input_tokens, 39); + assert.equal(result.output_tokens, 9); + assert.equal(result.total_tokens, 48); + }, +); + +test( + "real SDK transports inclusive cache/reasoning usage without leaking previous invocations", + { skip }, + async () => { + const workspace = await realpath( + await mkdtemp(join(tmpdir(), "fabric-pi-usage-")), + ); + const requests = []; + const server = createServer(async (request, response) => { + const chunks = []; + for await (const chunk of request) chunks.push(chunk); + requests.push(JSON.parse(Buffer.concat(chunks).toString("utf8"))); + const common = { + id: `chatcmpl-${requests.length}`, + object: "chat.completion.chunk", + created: 0, + model: "openai/gpt-oss-20b", + }; + const frame = (value) => + `data: ${JSON.stringify({ ...common, ...value })}\n\n`; + response.writeHead(200, { "content-type": "text/event-stream" }); + response.end( + frame({ + choices: [ + { + index: 0, + delta: { + role: "assistant", + content: `answer-${requests.length}`, + }, + finish_reason: "stop", + }, + ], + }) + + (requests.length === 3 + ? "" + : frame({ + choices: [], + usage: { + prompt_tokens: requests.length * 100, + completion_tokens: 20, + total_tokens: requests.length * 100 + 20, + prompt_tokens_details: { + cached_tokens: 12, + cache_write_tokens: 8, + }, + completion_tokens_details: { reasoning_tokens: 7 }, + }, + })) + + "data: [DONE]\n\n", + ); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + let session; + try { + session = await new PiSdkSessionFactory().create({ + agentName: "usage-test", + baseDir: workspace, + config: { + models: { + default: { + provider: "nvidia", + model: "openai/gpt-oss-20b", + base_url: `http://127.0.0.1:${server.address().port}/v1`, + api_key_env: "TEST_API_KEY", + }, + }, + tools: { enabled: [] }, + }, + runtimeContext: { + artifacts: {}, + environment: { + control_location: "external_control", + environment_id: "usage-test", + ownership: "caller_owned", + provider: "local", + workspace, + env: { TEST_API_KEY: "unused" }, + }, + invocation_id: "start", + request_id: "start", + runtime_id: "usage", + }, + }); + // Observe the actual SDK settings, not just the adapter constructor arguments. + assert.equal( + session.session.settingsManager.getCacheWarmingMode(), + "off", + ); + const first = await session.prompt("Remember the nonce alpha."); + assert.equal(first.usage.input_tokens, 100); + assert.equal(first.usage.output_tokens, 20); + assert.equal(first.usage.total_tokens, 120); + assert.equal(first.usage.cost_usd, undefined); + assert.equal( + first.usage.extensions["nemo_fabric.detailed_token_usage.v1"] + .cache_write_input_tokens, + 8, + ); + assert.equal( + first.usage.extensions["nemo_fabric.detailed_token_usage.v1"] + .reasoning_tokens, + 7, + ); + const second = await session.prompt("Continue from the previous answer."); + assert.equal(second.usage.input_tokens, 200); + assert.equal(second.usage.output_tokens, 20); + assert.match(JSON.stringify(requests[1]), /answer-1/); + const missing = await session.prompt("Provider omits usage this time."); + assert.equal(missing.usage, undefined); + assert.equal(missing.text, "answer-3"); + } finally { + await session?.stop(); + server.closeAllConnections(); + await new Promise((resolve) => server.close(resolve)); + await rm(workspace, { recursive: true, force: true }); + } + }, +); diff --git a/adapters/typescript/pi/test/usage.test.mjs b/adapters/typescript/pi/test/usage.test.mjs new file mode 100644 index 000000000..085a6ad41 --- /dev/null +++ b/adapters/typescript/pi/test/usage.test.mjs @@ -0,0 +1,130 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import assert from "node:assert/strict"; +import test from "node:test"; + +import { invocationUsage } from "../dist/usage.js"; +import { PiAdapterRuntime } from "../dist/runtime.js"; + +const usage = (input = 10, output = 3, cacheRead = 2, cacheWrite = 1) => ({ + input, + output, + cacheRead, + cacheWrite, + totalTokens: input + output + cacheRead + cacheWrite, + reasoning: 1, + cost: { total: 123 }, +}); +const message = (id, value = usage()) => ({ + id, + type: "message", + message: { role: "assistant", usage: value }, +}); + +test("counts the invocation's raw ledger including retries, tools, summaries and all branches", () => { + const entries = [ + message("old"), + message("failed-attempt"), + message("retry"), + { + id: "tool", + type: "message", + message: { role: "toolResult", usage: usage() }, + }, + { id: "ordinary-tool", type: "message", message: { role: "toolResult" } }, + { id: "summary", type: "compaction", usage: usage() }, + { id: "branch", type: "branch_summary", usage: usage() }, + { id: "auxiliary", type: "usage", usage: usage() }, + ]; + const result = invocationUsage(["old"], entries); + assert.equal(result.input_tokens, 78); + assert.equal(result.output_tokens, 18); + assert.equal(result.total_tokens, 96); + assert.equal(result.cached_input_tokens, 12); + assert.equal(result.input_tokens_include_cache, true); + assert.equal(result.cost_usd, undefined); + assert.deepEqual(result.extensions["nemo_fabric.detailed_token_usage.v1"], { + input_tokens: 78, + output_tokens: 18, + total_tokens: 96, + cached_input_tokens: 12, + cache_write_input_tokens: 6, + input_tokens_include_cache: true, + output_tokens_include_reasoning: true, + reasoning_tokens: 6, + }); + // The next prompt cannot reuse any preceding work. + assert.equal( + invocationUsage( + entries.map((entry) => entry.id), + [...entries, message("next")], + ).input_tokens, + 13, + ); +}); + +test("keeps incomplete, default-zero, reset, invalid and unsafe usage unknown", () => { + const invalid = [ + undefined, + usage(0, 0, 0, 0), + { ...usage(), input: undefined }, + { ...usage(), totalTokens: 5 }, + { ...usage(), output: -1 }, + { ...usage(), reasoning: 4 }, + { ...usage(), input: Number.MAX_SAFE_INTEGER + 1 }, + ]; + for (const value of invalid) { + const entry = { + id: "missing", + type: "message", + message: { role: "assistant", usage: value }, + }; + assert.equal(invocationUsage([], [message("observed"), entry]), undefined); + } + assert.equal( + invocationUsage([], [{ id: "c", type: "compaction" }]), + undefined, + ); + assert.equal(invocationUsage(["old"], [message("reset")]), undefined); + assert.equal(invocationUsage(["old"], []), undefined); + assert.equal(invocationUsage([], []), undefined); +}); + +test("preserves real zero dimensions and does not invent unavailable reasoning", () => { + const value = { ...usage(10, 0, 0, 0), reasoning: undefined }; + const result = invocationUsage([], [message("zero-output", value)]); + assert.equal(result.output_tokens, 0); + assert.equal(result.cached_input_tokens, 0); + assert.equal( + result.extensions["nemo_fabric.detailed_token_usage.v1"].reasoning_tokens, + undefined, + ); +}); + +test("retains observed invocation usage on failed and cancelled results", async () => { + for (const stopReason of ["error", "aborted", "stop"]) { + const observed = invocationUsage([], [message("billed")]); + const runtime = new PiAdapterRuntime({ + async create() { + return { + async prompt() { + return { + accepted: true, + turnStarted: true, + turnCount: 1, + stopReason, + usage: observed, + }; + }, + async stop() {}, + }; + }, + }); + await runtime.start({}); + const result = await runtime.invoke({ input: "test" }, {}); + assert.notEqual(result.status, "succeeded"); + assert.deepEqual(result.usage, observed); + await runtime.stop(); + } +}); diff --git a/adapters/typescript/qwen/README.md b/adapters/typescript/qwen/README.md index 178cb2a05..cd471ef51 100644 --- a/adapters/typescript/qwen/README.md +++ b/adapters/typescript/qwen/README.md @@ -98,11 +98,29 @@ adapter returns the per-invocation difference. Create a new runtime to change the selected model, workspace, system instructions, skills, MCP servers, tool policy, or permission mode. +Input totals include cached tokens, and output totals include reasoning tokens +under the adapter's fixed OpenAI authentication path. Cache counters are +reported separately as a breakdown and are not added again. Missing or invalid +counters, cumulative-counter resets, and all-zero usage observations remain +unknown rather than being converted to zero. A valid snapshot establishes a +baseline for later invocation deltas. + +These are the pinned SDK's provider-reported observations, not independently +verified billing totals. The SDK normalizes missing provider usage to zero and +can omit usage from failed requests. An all-zero observation cannot distinguish +missing data from real zero consumption, and partial omissions inside a +nonzero summary cannot be detected through the public result. + The adapter gives the child process isolated temporary Qwen home and runtime directories. It disables usage statistics, telemetry, extensions, ambient skills, and ambient MCP servers. Explicit NeMo Fabric `environment.env` values and the selected model credential are forwarded to the child process. +The adapter disables Qwen's automatic CLI relaunch, which sizes the replacement +process's heap from host RAM. The SDK-owned child retains Node.js's default +memory sizing for its container and any explicit `NODE_OPTIONS` heap setting +forwarded through `environment.env`. + In SDK mode, Qwen Code's `default` permission mode denies tool calls that need interactive approval. Use `auto` for Qwen's classifier-mediated policy. Use `yolo` only inside an appropriate task sandbox because it bypasses edit diff --git a/adapters/typescript/qwen/src/qwen-sdk.ts b/adapters/typescript/qwen/src/qwen-sdk.ts index f00c88593..d794e4f58 100644 --- a/adapters/typescript/qwen/src/qwen-sdk.ts +++ b/adapters/typescript/qwen/src/qwen-sdk.ts @@ -19,6 +19,16 @@ import type { QwenSession, QwenSessionFactory, QwenTurn } from "./runtime.js"; const STREAM_CLOSE_TIMEOUT_MS = 10; const MCP_FAILURE_MARKER = "Warning: MCP server(s) failed to start:"; +type UsageSnapshot = { input: number | null; output: number | null; total: number | null; cached: number | null }; + +function tokenCount(value: unknown): number | null { + return typeof value === "number" && Number.isSafeInteger(value) && value >= 0 ? value : null; +} + +function tokenDelta(current: number | null, previous: number | null): number | null { + return current !== null && previous !== null && current >= previous ? current - previous : null; +} + class QwenDiagnostics { private markerPrefix = ""; private mcpUnavailable = false; @@ -90,7 +100,7 @@ class PromptQueue implements AsyncIterable { export class QwenSdkSession implements QwenSession { private closed = false; private healthObservation?: Promise; - private previousUsage = { input: 0, output: 0, total: 0 }; + private previousUsage: UsageSnapshot = { input: 0, output: 0, total: 0, cached: 0 }; constructor( private readonly query: Query, @@ -102,17 +112,39 @@ export class QwenSdkSession implements QwenSession { private turnUsage(result: SDKResultMessage): AgentUsage { // SDK result usage is cumulative for the live query. Fabric usage belongs // to one invocation, so report the delta since the previous result. - const current = { - input: result.usage.input_tokens, - output: result.usage.output_tokens, - total: result.usage.total_tokens ?? result.usage.input_tokens + result.usage.output_tokens, + const current: UsageSnapshot = { + input: tokenCount(result.usage?.input_tokens), + output: tokenCount(result.usage?.output_tokens), + total: tokenCount(result.usage?.total_tokens), + cached: tokenCount(result.usage?.cache_read_input_tokens), }; const previous = this.previousUsage; + // The CLI synthesizes zero metrics when the provider omits usage. Its + // public result cannot distinguish that case from real zero consumption. + if ((current.input ?? 0) === 0 && (current.output ?? 0) === 0 && (current.total ?? 0) === 0) { + current.input = current.output = current.total = current.cached = null; + } this.previousUsage = current; + const reset = (["input", "output", "total"] as const).some(key => { + const now = current[key]; + const before = previous[key]; + return now !== null && before !== null && now < before; + }); + const input = reset ? null : tokenDelta(current.input, previous.input); + const output = reset ? null : tokenDelta(current.output, previous.output); + const total = reset || input === null || output === null ? null : tokenDelta(current.total, previous.total); + // Missing usage after an earlier turn also leaves cumulative metrics + // unchanged. Do not misreport that invocation as free model work. + const unchanged = input === 0 && output === 0 && (total ?? 0) === 0; return { - input_tokens: Math.max(0, current.input - previous.input), - output_tokens: Math.max(0, current.output - previous.output), - total_tokens: Math.max(0, current.total - previous.total), + input_tokens: unchanged ? null : input, + output_tokens: unchanged ? null : output, + total_tokens: unchanged ? null : total, + cached_input_tokens: reset || unchanged ? null : tokenDelta(current.cached, previous.cached), + // This adapter fixes authType to OpenAI. The pinned CLI's OpenAI + // converters retain prompt_tokens inclusive of cache and completion_tokens + // inclusive of reasoning; adding either breakdown would count it twice. + input_tokens_include_cache: true, }; } @@ -264,6 +296,9 @@ export class QwenSdkSessionFactory implements QwenSessionFactory { QWEN_CODE_SYSTEM_SETTINGS_PATH: systemSettings, QWEN_USAGE_STATISTICS_ENABLED: "0", QWEN_TELEMETRY_ENABLED: "0", + // The CLI otherwise relaunches with a heap sized from host RAM, + // overriding Node's container-aware limit and caller NODE_OPTIONS. + QWEN_CODE_NO_RELAUNCH: "true", }, mcpServers, // Only explicitly normalized servers may load; this excludes ambient diff --git a/adapters/typescript/qwen/test/fixtures/record-startup.cjs b/adapters/typescript/qwen/test/fixtures/record-startup.cjs new file mode 100644 index 000000000..ed7f490a1 --- /dev/null +++ b/adapters/typescript/qwen/test/fixtures/record-startup.cjs @@ -0,0 +1,10 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +const { appendFileSync } = require("node:fs"); +const { getHeapStatistics } = require("node:v8"); + +appendFileSync(process.env.FABRIC_QWEN_STARTUP_LOG, `${JSON.stringify({ + pid: process.pid, + heap_size_limit: getHeapStatistics().heap_size_limit, +})}\n`); diff --git a/adapters/typescript/qwen/test/sdk.test.mjs b/adapters/typescript/qwen/test/sdk.test.mjs index c2042cfb0..1ec3ab004 100644 --- a/adapters/typescript/qwen/test/sdk.test.mjs +++ b/adapters/typescript/qwen/test/sdk.test.mjs @@ -2,7 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 import assert from "node:assert/strict"; -import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; +import { execFileSync } from "node:child_process"; +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; import { createServer } from "node:http"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -10,11 +11,14 @@ import test from "node:test"; import { fileURLToPath } from "node:url"; import { QwenAdapterRuntime } from "../dist/runtime.js"; -import { QwenSdkSessionFactory } from "../dist/qwen-sdk.js"; +import { QwenSdkSession, QwenSdkSessionFactory } from "../dist/qwen-sdk.js"; const mcpServerPath = fileURLToPath(new URL("./fixtures/mcp-server.mjs", import.meta.url)); test("bundled Qwen CLI keeps context across two SDK prompts", { timeout: 45_000 }, async () => { + const workspace = await mkdtemp(join(tmpdir(), "fabric-qwen-memory-")); + const startupLog = join(workspace, "startup.jsonl"); + const startupProbe = fileURLToPath(new URL("./fixtures/record-startup.cjs", import.meta.url)); const requests = []; const server = createServer(async (request, response) => { const chunks = []; @@ -25,7 +29,7 @@ test("bundled Qwen CLI keeps context across two SDK prompts", { timeout: 45_000 response.writeHead(200, { "content-type": "text/event-stream" }); const common = { id: `chatcmpl-${requests.length}`, object: "chat.completion.chunk", created: 0, model: "fabric-test-model" }; response.write(`data: ${JSON.stringify({ ...common, choices: [{ index: 0, delta: { role: "assistant", content: isSecondTurn ? "turn-2" : "turn-1" }, finish_reason: null }] })}\n\n`); - response.write(`data: ${JSON.stringify({ ...common, choices: [{ index: 0, delta: {}, finish_reason: "stop" }], usage: { prompt_tokens: 5, completion_tokens: 2, total_tokens: 7 } })}\n\n`); + response.write(`data: ${JSON.stringify({ ...common, choices: [{ index: 0, delta: {}, finish_reason: "stop" }], usage: { prompt_tokens: 5, completion_tokens: 2, total_tokens: 7, prompt_tokens_details: { cached_tokens: 3 }, completion_tokens_details: { reasoning_tokens: 1 } } })}\n\n`); response.end("data: [DONE]\n\n"); }); await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); @@ -33,29 +37,44 @@ test("bundled Qwen CLI keeps context across two SDK prompts", { timeout: 45_000 try { const port = server.address().port; session = await new QwenSdkSessionFactory().create({ - agentName: "qwen-test", baseDir: process.cwd(), + agentName: "qwen-test", baseDir: workspace, config: { models: { default: { provider: "openai", model: "fabric-test-model", api_key_env: "FABRIC_QWEN_TEST_KEY", base_url: `http://127.0.0.1:${port}/v1` } }, instructions: { system: { content: "Fabric Qwen instruction probe", mode: "replace" } }, }, runtimeContext: { - artifacts: {}, environment: { control_location: "external_control", env: { FABRIC_QWEN_TEST_KEY: "local-test-key" }, environment_id: "env-1", ownership: "caller_owned", provider: "local", workspace: process.cwd() }, + artifacts: {}, environment: { control_location: "external_control", env: { + FABRIC_QWEN_TEST_KEY: "local-test-key", + FABRIC_QWEN_STARTUP_LOG: startupLog, + NODE_OPTIONS: `--max-old-space-size=512 --require=${JSON.stringify(startupProbe)}`, + QWEN_CODE_NO_RELAUNCH: "", + }, environment_id: "env-1", ownership: "caller_owned", provider: "local", workspace }, invocation_id: "start", request_id: "start", runtime_id: "runtime-1", }, }); const first = await session.prompt("Remember token cobalt"); + const firstRequestCount = requests.length; const second = await session.prompt("Which token did I ask you to remember?"); assert.equal(first.text, "turn-1", JSON.stringify(first)); assert.equal(second.text, "turn-2", JSON.stringify(second)); - assert.deepEqual(first.usage, { input_tokens: 10, output_tokens: 4, total_tokens: 14 }); - assert.deepEqual(second.usage, { input_tokens: 10, output_tokens: 4, total_tokens: 14 }); + assert.deepEqual(first.usage, { input_tokens: 10, output_tokens: 4, total_tokens: 14, cached_input_tokens: 6, input_tokens_include_cache: true }); + assert.equal(first.usage.input_tokens, firstRequestCount * 5); + assert.deepEqual(second.usage, { input_tokens: 10, output_tokens: 4, total_tokens: 14, cached_input_tokens: 6, input_tokens_include_cache: true }); + assert.equal(second.usage.input_tokens, (requests.length - firstRequestCount) * 5); assert.ok(requests.length >= 2); assert.match(String(requests[0].messages[0].content), /Fabric Qwen instruction probe/); assert.match(JSON.stringify(requests.at(-1).messages), /Remember token cobalt/); + const startups = (await readFile(startupLog, "utf8")).trim().split("\n").map(JSON.parse); + assert.equal(startups.length, 1, "Qwen must not relaunch with a host-sized heap"); + const expectedHeapLimit = Number(execFileSync(process.execPath, [ + "--max-old-space-size=512", "-p", "require('node:v8').getHeapStatistics().heap_size_limit", + ], { env: { ...process.env, NODE_OPTIONS: "" }, encoding: "utf8" })); + assert.equal(startups[0].heap_size_limit, expectedHeapLimit, "Qwen must preserve the caller's Node heap limit"); } finally { await session?.close(); server.closeAllConnections(); await new Promise((resolve) => server.close(resolve)); + await rm(workspace, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); } }); @@ -287,3 +306,117 @@ test("native Qwen read-only control health passes before during and after a bloc } assert.equal((await runtime.check("live")).reason_code, "runtime_unavailable"); }); + +function usageSession(summaries) { + return new QwenSdkSession({ + isClosed: () => false, + getSessionId: () => "usage-test", + async next() { + return { done: false, value: { type: "result", subtype: "success", result: "ok", usage: summaries.shift() } }; + }, + }, { push() {} }, "/unused", { hasMcpFailure: () => false }); +} + +const unknownUsage = { + input_tokens: null, output_tokens: null, total_tokens: null, + cached_input_tokens: null, input_tokens_include_cache: true, +}; + +test("Qwen cumulative usage preserves cache deltas and real zero output", async () => { + const session = usageSession([ + { input_tokens: 10, output_tokens: 0, total_tokens: 10, cache_read_input_tokens: 3 }, + { input_tokens: 30, output_tokens: 4, total_tokens: 34, cache_read_input_tokens: 8 }, + ]); + assert.deepEqual((await session.prompt("one")).usage, { + input_tokens: 10, output_tokens: 0, total_tokens: 10, + cached_input_tokens: 3, input_tokens_include_cache: true, + }); + assert.deepEqual((await session.prompt("two")).usage, { + input_tokens: 20, output_tokens: 4, total_tokens: 24, + cached_input_tokens: 5, input_tokens_include_cache: true, + }); +}); + +test("Qwen reset snapshots remain unknown and establish a fresh baseline", async () => { + const session = usageSession([ + { input_tokens: 100, output_tokens: 20, total_tokens: 120, cache_read_input_tokens: 10 }, + { input_tokens: 10, output_tokens: 2, total_tokens: 12, cache_read_input_tokens: 1 }, + { input_tokens: 30, output_tokens: 5, total_tokens: 35, cache_read_input_tokens: 4 }, + ]); + await session.prompt("one"); + assert.deepEqual((await session.prompt("two")).usage, unknownUsage); + assert.deepEqual((await session.prompt("three")).usage, { + input_tokens: 20, output_tokens: 3, total_tokens: 23, + cached_input_tokens: 3, input_tokens_include_cache: true, + }); +}); + +for (const missing of [undefined, { input_tokens: 0, output_tokens: 0 }, { input_tokens: NaN, output_tokens: -1, total_tokens: 1.5 }]) { + test(`Qwen missing or invalid snapshot stays unknown through baseline recovery: ${JSON.stringify(missing)}`, async () => { + const session = usageSession([ + missing, + { input_tokens: 20, output_tokens: 2, total_tokens: 22, cache_read_input_tokens: 3 }, + { input_tokens: 30, output_tokens: 5, total_tokens: 35, cache_read_input_tokens: 4 }, + ]); + assert.deepEqual((await session.prompt("one")).usage, unknownUsage); + assert.deepEqual((await session.prompt("two")).usage, unknownUsage); + assert.deepEqual((await session.prompt("three")).usage, { + input_tokens: 10, output_tokens: 3, total_tokens: 13, + cached_input_tokens: 1, input_tokens_include_cache: true, + }); + }); +} + +test("Qwen does not invent missing total or cache counters", async () => { + const session = usageSession([{ input_tokens: 10, output_tokens: 2 }]); + assert.deepEqual((await session.prompt("one")).usage, { + input_tokens: 10, output_tokens: 2, total_tokens: null, + cached_input_tokens: null, input_tokens_include_cache: true, + }); +}); + +test("bundled Qwen CLI missing provider usage remains unknown", { timeout: 45_000 }, async () => { + const workspace = await mkdtemp(join(tmpdir(), "fabric-qwen-unknown-usage-")); + const server = createServer(async (request, response) => { + for await (const _chunk of request) { /* Drain the request. */ } + response.writeHead(200, { "content-type": "text/event-stream" }); + const common = { id: "chatcmpl-missing-usage", object: "chat.completion.chunk", created: 0, model: "fabric-test-model" }; + response.write(`data: ${JSON.stringify({ ...common, choices: [{ index: 0, delta: { role: "assistant", content: "ready" }, finish_reason: null }] })}\n\n`); + response.write(`data: ${JSON.stringify({ ...common, choices: [{ index: 0, delta: {}, finish_reason: "stop" }] })}\n\n`); + response.end("data: [DONE]\n\n"); + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + let session; + try { + session = await new QwenSdkSessionFactory().create({ + agentName: "qwen-usage-test", baseDir: workspace, + config: { models: { default: { provider: "openai", model: "fabric-test-model", api_key_env: "FABRIC_QWEN_TEST_KEY", base_url: `http://127.0.0.1:${server.address().port}/v1` } } }, + runtimeContext: { + artifacts: {}, environment: { control_location: "external_control", env: { FABRIC_QWEN_TEST_KEY: "local-test-key" }, environment_id: "env-1", ownership: "caller_owned", provider: "local", workspace }, + invocation_id: "start", request_id: "start", runtime_id: "runtime-usage", + }, + }); + const turn = await session.prompt("Answer ready"); + assert.equal(turn.text, "ready"); + assert.deepEqual(turn.usage, unknownUsage); + } finally { + await session?.close(); + server.closeAllConnections(); + await new Promise(resolve => server.close(resolve)); + await rm(workspace, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } +}); + +test("Qwen unchanged cumulative metrics do not invent a zero-cost invocation", async () => { + const session = usageSession([ + { input_tokens: 10, output_tokens: 2, total_tokens: 12, cache_read_input_tokens: 3 }, + { input_tokens: 10, output_tokens: 2, total_tokens: 12, cache_read_input_tokens: 3 }, + { input_tokens: 30, output_tokens: 5, total_tokens: 35, cache_read_input_tokens: 4 }, + ]); + await session.prompt("one"); + assert.deepEqual((await session.prompt("two")).usage, unknownUsage); + assert.deepEqual((await session.prompt("three")).usage, { + input_tokens: 20, output_tokens: 3, total_tokens: 23, + cached_input_tokens: 1, input_tokens_include_cache: true, + }); +}); diff --git a/adapters/typescript/scripts/check-install.mjs b/adapters/typescript/scripts/check-install.mjs index 182f68d65..97eb48f75 100644 --- a/adapters/typescript/scripts/check-install.mjs +++ b/adapters/typescript/scripts/check-install.mjs @@ -424,7 +424,7 @@ try { throw new Error("Installed Pi descriptor does not reference its packaged CLI"); } - for (const name of ["@earendil-works/pi-ai", "@earendil-works/pi-coding-agent", "@earendil-works/pi-mcp"]) { + for (const name of ["@earendil-works/pi-ai", "@earendil-works/pi-coding-agent", "@earendil-works/pi-mcp", "@earendil-works/pi-codemode"]) { if (await pathExists(join(consumerRoot, "node_modules", name, "package.json"))) { throw new Error(`Adapter-only install unexpectedly included ${name}`); } @@ -449,10 +449,10 @@ try { "--no-audit", "--no-fund", "--package-lock=false", - "@earendil-works/pi-ai@1.2.0", - "@earendil-works/pi-coding-agent@1.2.0", - "@earendil-works/pi-mcp@1.2.0", - "@earendil-works/pi-codemode@1.2.0", + "@earendil-works/pi-ai@1.0.3", + "@earendil-works/pi-coding-agent@1.0.3", + "@earendil-works/pi-mcp@1.0.3", + "@earendil-works/pi-codemode@1.0.3", ], consumerRoot, ); diff --git a/adapters/typescript/scripts/check-package.mjs b/adapters/typescript/scripts/check-package.mjs index 94a344d9e..42e8bf43c 100644 --- a/adapters/typescript/scripts/check-package.mjs +++ b/adapters/typescript/scripts/check-package.mjs @@ -70,6 +70,8 @@ const expectedByPackage = { "dist/node-version.js", "dist/pi-sdk.d.ts", "dist/pi-sdk.js", + "dist/usage.d.ts", + "dist/usage.js", "dist/relay-config.d.ts", "dist/relay-config.js", "dist/relay-gateway.d.ts", @@ -187,11 +189,11 @@ if (manifest.name === "nemo-fabric-adapters-pi") { if (descriptor.runner?.command !== "node" || descriptor.runner?.script !== "dist/cli.js") { throw new Error("The Pi descriptor runner must resolve inside the npm package"); } - for (const name of ["@earendil-works/pi-ai", "@earendil-works/pi-coding-agent", "@earendil-works/pi-mcp"]) { + for (const name of ["@earendil-works/pi-ai", "@earendil-works/pi-coding-agent", "@earendil-works/pi-mcp", "@earendil-works/pi-codemode"]) { if (manifest.dependencies?.[name] !== undefined) { throw new Error(`The Pi harness package ${name} must not be a production dependency`); } - if (manifest.peerDependencies?.[name] !== "^1.0.0") { + if (manifest.peerDependencies?.[name] !== "1.0.3") { throw new Error(`The Pi harness package ${name} must declare the supported peer range`); } if (manifest.peerDependenciesMeta?.[name]?.optional !== true) { diff --git a/adapters/typescript/strands/README.md b/adapters/typescript/strands/README.md index 71ec4f6a3..0a66cd06d 100644 --- a/adapters/typescript/strands/README.md +++ b/adapters/typescript/strands/README.md @@ -122,7 +122,7 @@ Each runtime owns one child process, an isolated temporary CLI home, and one liv Warm invocations retain conversation history without saving it to the workspace. Deadline expiry attempts ACP cancellation, invalidates the session, and terminates its process group. If cleanup fails after an invocation error, the runtime retains the session for a later `stop()` retry while rejecting further invocations and replacement sessions. Stop is idempotent. The adapter supports text input and returns `{ "response": "..." }`, with ACP session ID and stop reason in result metadata. Non-success stop reasons become failed or cancelled results. -Strands CLI usage is invocation-local and is not differenced across turns. Inclusive input, output, total, and cached-read counters are normalized when valid. Cache writes are retained in `nemo_fabric.detailed_token_usage.v1`. Missing or malformed usage remains unknown without discarding a completed response. Strands CLI 0.2.1 reports zero cache counters even when the provider omitted them. The adapter therefore omits zero cache-read and cache-write counts, including genuine zeros, preserves positive counts, and retains inclusive input/output totals. This workaround is specific to Strands; it does not change other adapters’ handling of reported zeros. Cost is not estimated. Public cancellation, native OpenAI streaming, live updates, services, Relay telemetry, normalized runtime limits, and normalized skills are not advertised. +Strands CLI usage is invocation-local and is not differenced across turns. Inclusive input, output, total, and cached-read counters are normalized when valid. Output totals include reasoning tokens; the adapter records this inclusion in `nemo_fabric.detailed_token_usage.v1` without inventing a separate reasoning count. Cache writes are retained in `nemo_fabric.detailed_token_usage.v1`. Missing or malformed usage remains unknown without discarding a completed response. Strands CLI 0.2.1 reports zero cache counters even when the provider omitted them. The adapter therefore omits zero cache-read and cache-write counts, including genuine zeros, preserves positive counts, and retains inclusive input/output totals. This workaround is specific to Strands; it does not change other adapters’ handling of reported zeros. Cost is not estimated. Public cancellation, native OpenAI streaming, live updates, services, Relay telemetry, normalized runtime limits, and normalized skills are not advertised. ## Validate diff --git a/adapters/typescript/strands/src/strands-acp.ts b/adapters/typescript/strands/src/strands-acp.ts index 12e0a949d..7a490ce93 100644 --- a/adapters/typescript/strands/src/strands-acp.ts +++ b/adapters/typescript/strands/src/strands-acp.ts @@ -175,6 +175,8 @@ export function normalizeUsage( "nemo_fabric.detailed_token_usage.v1": { input_tokens: input, output_tokens: output, + // CLI 0.2.1 forwards SDK output usage without subtracting reasoning. + output_tokens_include_reasoning: true, total_tokens: total, input_tokens_include_cache: true, ...((cache ?? 0) > 0 ? { cached_input_tokens: cache } : {}), diff --git a/adapters/typescript/strands/test/strands-acp.test.mjs b/adapters/typescript/strands/test/strands-acp.test.mjs index bfad458dc..dbdc1f051 100644 --- a/adapters/typescript/strands/test/strands-acp.test.mjs +++ b/adapters/typescript/strands/test/strands-acp.test.mjs @@ -190,6 +190,8 @@ test("omits ambiguous zero cache counts while preserving positive counts and tok assert.equal(usage.output_tokens, 5); assert.equal(usage.total_tokens, 15); assert.equal(usage.input_tokens_include_cache, true); + assert.equal(detail.output_tokens_include_reasoning, true); + assert.equal(Object.hasOwn(detail, "reasoning_tokens"), false); assert.equal(Object.hasOwn(usage, "cached_input_tokens"), read > 0); assert.equal(Object.hasOwn(detail, "cached_input_tokens"), read > 0); assert.equal(Object.hasOwn(detail, "cache_write_input_tokens"), write > 0); diff --git a/docs/integrations/consumer/harbor.mdx b/docs/integrations/consumer/harbor.mdx index 68371e3e4..ed93a227f 100644 --- a/docs/integrations/consumer/harbor.mdx +++ b/docs/integrations/consumer/harbor.mdx @@ -20,6 +20,21 @@ workflows: [Run NeMo Fabric agents with Harbor](https://github.com/NVIDIA/NeMo-Fabric/tree/main/examples/harbor) +## Model Provider + +Use `fabric_model_provider=openai` for an OpenAI-compatible endpoint whose served model ID starts with another vendor prefix. For example, combine it with `model_name="nvidia/nemotron-3-ultra-550b-a55b"` and `fabric_model_base_url` to select the OpenAI protocol while preserving the exact served model ID. Without an override, the provider is inferred from the model prefix as before. The override requires a model and remains subject to the selected adapter’s model schema. + +## Model Settings + +Pass adapter-declared model settings with `fabric_model_settings`, for example +`--ak 'fabric_model_settings={"cost_tracking":"disabled"}'` for an explicitly +unpriced Mini-SWE model. Pair that policy with a positive `fabric_max_turns`. +The bridge copies these settings into `models.default.settings`; the selected +adapter's model schema and runtime validation still apply. This option requires +a model. Keep credentials in managed environment variables, outside model +settings. This option applies to the SDK bridge; native Harbor owns a separate +implementation. + ## Resource Stats For an example of collecting resource stats inside a task container, follow the [SWE-bench walkthrough](../../../examples/harbor/swebench/README.md#run-deep-agents-with-resource-stats). The [Resource Usage and Stats](../../sdk/system-stats.mdx) guide explains the configuration model, collection scope, and metric interpretation. diff --git a/docs/integrations/harness/deepagents.mdx b/docs/integrations/harness/deepagents.mdx index 6692fc57e..c95d94e54 100644 --- a/docs/integrations/harness/deepagents.mdx +++ b/docs/integrations/harness/deepagents.mdx @@ -192,12 +192,26 @@ provides only its fallback path. Configure the required variables explicitly and use absolute executable paths when needed. The `execute` tool remains subject to `tools.enabled` and `tools.blocked`, like all other Deep Agents built-ins. -File tools and shell commands use different absolute-path namespaces when -`virtual_mode=True`. For file tools, `/report.txt` resolves to -`/report.txt`; in `execute`, the same path refers to `/report.txt` on -the adapter host. Shell commands start in `environment.workspace`, so use -workspace-relative paths, such as `cat report.txt`, for files surfaced by file -tools. Do not pass a file tool's virtual absolute path directly to `execute`. +On POSIX, with `local_shell`, file tools accept absolute paths inside +`environment.workspace` as aliases for workspace files. For example, with +workspace `/app`, both file-tool paths `/app/report.txt` and `/report.txt` refer +to `/app/report.txt` on disk. Shell commands use host paths and start in the +workspace, so `cat report.txt` and `cat /app/report.txt` read that file. + +On POSIX, listings and searches return canonical physical paths inside the +workspace. These paths identify the same file in subsequent reads and edits, +even when a nested directory repeats the workspace prefix. For example, `/app/app/report.txt` +remains distinct from `/app/report.txt`; the relative file-tool path +`app/report.txt` also identifies the nested file. + +On Windows, listings and searches retain workspace-rooted virtual paths, such +as `/nested/report.txt`. Use these paths with file tools; Deep Agents rejects +drive-prefixed paths such as `C:/workspace/nested/report.txt` before backend +resolution. Shell commands continue to use native host paths. + +File-tool traversal and symlink-escape checks remain enabled. Paths outside the +workspace retain the existing virtual-root interpretation; they do not grant +access to host files. `subagents` accepts declarative synchronous subagents and Agent Protocol asynchronous subagents. A declarative subagent requires `name`, `description`, diff --git a/docs/integrations/harness/kilo.mdx b/docs/integrations/harness/kilo.mdx index 4fb63213a..23fdc66bc 100644 --- a/docs/integrations/harness/kilo.mdx +++ b/docs/integrations/harness/kilo.mdx @@ -160,6 +160,8 @@ an available loopback port, connects through `@kilocode/sdk`, and creates one session. `stop` deletes the session, terminates that server, and removes its temporary profile. +Kilo Code's `background_process` tool stops session-scoped processes when the session ends or the Kilo Code server exits. For a service that must survive runtime shutdown, set `persistent: true` on that tool call. This matters in Harbor, where verification runs after the runtime stops. Persistent processes require separate cleanup; Harbor removes them when it destroys the task container. + ```mermaid flowchart TB Fabric["NeMo Fabric runtime"] diff --git a/docs/integrations/harness/mini-swe-agent.mdx b/docs/integrations/harness/mini-swe-agent.mdx index 4832eacf2..3033bfad4 100644 --- a/docs/integrations/harness/mini-swe-agent.mdx +++ b/docs/integrations/harness/mini-swe-agent.mdx @@ -61,6 +61,32 @@ model-provider credential. `harness.settings.timeout` to set the maximum duration of one command; the default is `30` seconds. +For a local or unpriced model, explicitly set +`models..settings={"cost_tracking": "disabled"}` and a positive +`runtime.max_turns`. The default is strict SDK pricing and its monetary budget; +ambient `MSWEA_COST_TRACKING` does not override it. Disabled tracking removes +the SDK monetary budget, so use it only when you accept execution without a +monetary cap. Turn and invocation time limits still apply. Cost remains +unavailable in Fabric output and Relay telemetry; the SDK's fallback zero is +not reported as measured cost. Provider token usage in Relay is preserved. + +For example, these fields explicitly opt into an unpriced model: + +```json +{ + "models": { + "default": { + "provider": "nvidia", + "model": "nvidia/nemotron-3-ultra-550b-a55b", + "base_url": "http://localhost:8000/v1", + "settings": {"cost_tracking": "disabled"} + } + }, + "runtime": {"max_turns": 50, "timeout_seconds": 1800} +} +``` + + This adapter does not support MCP, skills, tool policy, or native OpenAI streaming. diff --git a/docs/integrations/harness/openclaw.mdx b/docs/integrations/harness/openclaw.mdx index 51910792a..ec7ecd662 100644 --- a/docs/integrations/harness/openclaw.mdx +++ b/docs/integrations/harness/openclaw.mdx @@ -77,6 +77,21 @@ MCP configuration. Normalized MCP authentication is not supported. For a custom OpenAI-compatible endpoint, set the model's `base_url` and, when needed, `api_key_env`. +For an endpoint that supports `stream_options.include_usage`, set +`models..settings.supports_usage_in_streaming` to `true`. This maps to +OpenClaw's native `compat.supportsUsageInStreaming` and requires `base_url`. +Use `false` for an endpoint that rejects streaming usage, or omit the setting +to retain OpenClaw's endpoint capability detection. Attached services own +their model configuration and do not accept this setting. + +OpenClaw 2026.9.4's Gateway does not distinguish aggregate usage from its +last-call fallback, and converts missing usage to zero counters. NeMo Fabric +leaves standard complete counters unknown and preserves the Gateway's counts +in `usage.extensions["nvidia.fabric.openclaw.usage_evidence.v1"].observed`. +The evidence includes `scope_unverified` in `gaps`, plus `missing_usage` or +`zero_unverified` when applicable. Reported counts do not establish complete +coverage of retries, compaction, child agents, or background work. + For a NeMo Fabric-owned service, `harness.settings.channel_config` accepts OpenClaw-native `channels` and `bindings`. This supports multiple accounts for Telegram and other OpenClaw channels without a NeMo Fabric-specific channel schema. diff --git a/docs/integrations/harness/openhands.mdx b/docs/integrations/harness/openhands.mdx index d28e82709..7ebdb2e36 100644 --- a/docs/integrations/harness/openhands.mdx +++ b/docs/integrations/harness/openhands.mdx @@ -64,6 +64,11 @@ workspace. Supported tool names are `terminal` (alias `bash`) and `file_editor` (alias `edit`). OpenHands retains its internal finish and reasoning tools. +For `provider="nvidia"`, the adapter adds LiteLLM's `nvidia_nim/` dispatch +prefix while preserving the served model ID, including its `nvidia/` +namespace. This also supports self-hosted NIM endpoints through `base_url`. +An existing `nvidia_nim/` prefix is retained once. + Skill paths resolve from the configuration base directory and must contain a `SKILL.md` file. The adapter disables ambient OpenHands user, public, project, and memory loading, so only explicitly configured NeMo Fabric skills load. @@ -119,6 +124,15 @@ closes the conversation and removes the adapter-owned temporary profile state. Create a new runtime to change the model, workspace, tools, MCP servers, or skills. +The adapter requires invocation-local SDK token records before reporting usage. +Missing records leave counters unknown, including requests that fail before +the SDK observes usage. Recorded tokens remain available after a later failure. +Cost remains unknown unless the SDK records a cost for every returned response; +its zero-initialized accumulator does not establish a free request. +Input tokens follow LiteLLM's inclusive `prompt_tokens` convention. The adapter +sets `input_tokens_include_cache=true` without adding cache tokens again or +inventing a missing cache breakdown. + ## Run the Code Review Example Set `NVIDIA_API_KEY`. For a source checkout, also point NeMo Fabric at the Python diff --git a/docs/integrations/harness/pi.mdx b/docs/integrations/harness/pi.mdx index 7a7e93949..74ba46e46 100644 --- a/docs/integrations/harness/pi.mdx +++ b/docs/integrations/harness/pi.mdx @@ -8,6 +8,23 @@ SPDX-License-Identifier: Apache-2.0 */} Use the NVIDIA NeMo Fabric `nvidia.fabric.pi` adapter to run tasks with the Pi SDK in a persistent Node.js process. +## Invocation Usage + +Each invocation reads only new entries from Pi's raw session ledger, including +retained failed attempts, compaction and branch summaries, nested tool usage, +and auxiliary usage records. Automatic cache warming is disabled so background +refresh calls cannot escape the invocation boundary. Session history and previous invocations are not +counted again. Input totals include cache reads and cache writes; output totals +already include reasoning tokens. Available breakdowns are returned through +`nemo_fabric.detailed_token_usage.v1`. + +Missing or inconsistent usage leaves the complete counters unknown. Pi 1.0.3 +initializes unreported provider usage to zero, so an all-zero record also remains +unknown. A failed or retried summary can omit attempt usage from the ledger; +complete counters remain unknown in that case. Nonzero records can retain real +zero input, output, or cache dimensions. Configured-rate SDK cost estimates are +not published as provider-reported `cost_usd`. + ## Install the Adapter Pi requires Node.js 22.19.0 or newer. @@ -15,17 +32,21 @@ Pi requires Node.js 22.19.0 or newer. ### Install for Consumers Install the npm adapter in the project that owns the NeMo Fabric configuration, -then install the compatible Pi SDK harness version selected by that project: +then install the supported Pi SDK package set: ```bash npm install nemo-fabric-adapters-pi -npm install @earendil-works/pi-ai@^1.0.0 @earendil-works/pi-coding-agent@^1.0.0 @earendil-works/pi-mcp@^1.0.0 +npm install @earendil-works/pi-ai@1.0.3 @earendil-works/pi-coding-agent@1.0.3 @earendil-works/pi-mcp@1.0.3 @earendil-works/pi-codemode@1.0.3 ``` The Pi packages are optional peers: installing `nemo-fabric-adapters-pi` alone does not install the harness. Starting the adapter without compatible Pi packages returns `pi_harness_unavailable`. +The supported SDK packages are pinned together at 1.0.3. The coding-agent +package otherwise permits newer `pi-codemode` versions with incompatible exports; +installing all four packages keeps fresh installs on the tested combination. + ### Install for Source Development For focused Pi development in the NeMo Fabric source tree, install the Pi diff --git a/docs/integrations/harness/qwen.mdx b/docs/integrations/harness/qwen.mdx index 61425ce02..b8e6f55f9 100644 --- a/docs/integrations/harness/qwen.mdx +++ b/docs/integrations/harness/qwen.mdx @@ -134,6 +134,19 @@ adapter returns the per-invocation difference. Create a new runtime to change the selected model, workspace, system instructions, skills, MCP servers, tool policy, or permission mode. +Input totals include cached tokens, and output totals include reasoning tokens +under the adapter's fixed OpenAI authentication path. Cache counters are +reported separately as a breakdown and are not added again. Missing or invalid +counters, cumulative-counter resets, and all-zero usage observations remain +unknown rather than being converted to zero. A valid snapshot establishes a +baseline for later invocation deltas. + +These are the pinned SDK's provider-reported observations, not independently +verified billing totals. The SDK normalizes missing provider usage to zero and +can omit usage from failed requests. An all-zero observation cannot distinguish +missing data from real zero consumption, and partial omissions inside a +nonzero summary cannot be detected through the public result. + The adapter gives the child process isolated temporary Qwen home and runtime directories. It disables Qwen Code's optional product-usage analytics and telemetry. This does not affect model-provider metering or token usage reported @@ -141,6 +154,11 @@ by NeMo Fabric. The adapter also disables extensions, ambient skills, and ambient MCP servers. Explicit NeMo Fabric `environment.env` values and the selected model credential are forwarded to the child process. +The adapter disables Qwen's automatic CLI relaunch, which sizes the replacement +process's heap from host RAM. The SDK-owned child retains Node.js's default +memory sizing for its container and any explicit `NODE_OPTIONS` heap setting +forwarded through `environment.env`. + In SDK mode, set `harness.settings.permission_mode` to control Qwen Code's permission policy. The default value, `default`, denies tool calls that need interactive approval. Use `auto` for Qwen's classifier-mediated policy. Use diff --git a/examples/harbor/README.md b/examples/harbor/README.md index 4b0b989fa..178c71273 100644 --- a/examples/harbor/README.md +++ b/examples/harbor/README.md @@ -13,6 +13,8 @@ you want to exercise. ## Walkthroughs +For reproducible repository integration checks, use the [`qualify-harbor` maintainer skill](../../.agents/skills/qualify-harbor/SKILL.md). It runs one shared Terminal-Bench task across a catalog-derived harness matrix with isolated host/task installs and reports blocked and failed coverage alongside deterministic verifier results. The individual walkthroughs below do not establish an all-harness integration pass. + | Walkthrough | What it demonstrates | | --- | --- | | [Calculator walkthrough](calculator/README.md) | Validate the complete integration and Harbor reward with a deterministic, credential-free smoke test, then run the same task with OpenClaw, Claude, Pi, or Codex. | @@ -109,6 +111,7 @@ container boundary: | `--skill` | `skills.paths` | | `--mcp-config` | `mcp.servers` | | `--ak fabric_telemetry=relay` | `telemetry.providers.relay` and `relay.observability` | +| `--ak fabric_model_provider=` | `models.default.provider` (overrides model-prefix inference) | | `--ak fabric_model_base_url=` | `models.default.base_url` | | `--ak fabric_model_api_key_env=` | `models.default.api_key_env` | | `--ak fabric_discovery_paths='["/task/adapter.fabric-adapter.json"]'` | `discovery.local_paths` for task-local descriptors such as Pi | diff --git a/sdk/python/nemo-fabric-adapter-catalog/src/nemo_fabric_adapter_catalog/catalog.json b/sdk/python/nemo-fabric-adapter-catalog/src/nemo_fabric_adapter_catalog/catalog.json index 4063f4185..08444100c 100644 --- a/sdk/python/nemo-fabric-adapter-catalog/src/nemo_fabric_adapter_catalog/catalog.json +++ b/sdk/python/nemo-fabric-adapter-catalog/src/nemo_fabric_adapter_catalog/catalog.json @@ -1601,7 +1601,17 @@ }, "settings": { "additionalProperties": false, - "properties": {}, + "properties": { + "cost_tracking": { + "default": "default", + "description": "Use strict SDK pricing, or explicitly disable monetary budgeting for unpriced models. Disabled requires a positive runtime.max_turns; cost remains unavailable.", + "enum": [ + "default", + "disabled" + ], + "type": "string" + } + }, "type": "object" }, "temperature": { @@ -1653,7 +1663,7 @@ "name": "nemo-fabric-adapters-mini-swe-agent", "version": "0.6.0" }, - "sha256": "40ee9364117fd6d4d1fe4e9a822ecd7b9022a5f0f8053d8b550d065e3726df37", + "sha256": "2480f84f77fdbc681ddd83ebedf3b30cbb766273f7104ad54408e54307f121db", "source": "adapters/python/mini-swe-agent/mini-swe-agent.fabric-adapter.json" }, "nvidia.fabric.nooa": { @@ -1885,6 +1895,56 @@ ] }, "contract_version": "fabric.adapter/v1alpha2", + "extension_schemas": { + "usage": { + "additionalProperties": false, + "properties": { + "nvidia.fabric.openclaw.usage_evidence.v1": { + "additionalProperties": false, + "properties": { + "gaps": { + "items": { + "enum": [ + "scope_unverified", + "missing_usage", + "zero_unverified" + ] + }, + "type": "array", + "uniqueItems": true + }, + "observed": { + "additionalProperties": false, + "properties": { + "input_tokens": { + "minimum": 0, + "type": "integer" + }, + "output_tokens": { + "minimum": 0, + "type": "integer" + }, + "total_tokens": { + "minimum": 0, + "type": "integer" + } + }, + "type": "object" + } + }, + "required": [ + "observed", + "gaps" + ], + "type": "object" + } + }, + "required": [ + "nvidia.fabric.openclaw.usage_evidence.v1" + ], + "type": "object" + } + }, "model_schema": { "$schema": "https://json-schema.org/draft/2020-12/schema", "additionalProperties": false, @@ -1911,7 +1971,12 @@ }, "settings": { "additionalProperties": false, - "properties": {}, + "properties": { + "supports_usage_in_streaming": { + "description": "Whether the configured custom OpenAI-compatible endpoint supports stream_options.include_usage. Requires base_url; omitted values retain OpenClaw capability detection.", + "type": "boolean" + } + }, "type": "object" }, "temperature": { @@ -2040,7 +2105,7 @@ "name": "nemo-fabric-adapters-openclaw", "version": "0.6.0" }, - "sha256": "b2a9ef847226bc5858024776b50fb87c6d57cd4332accc6583aacb5d348a17f5", + "sha256": "bc95c4dfeee7638a490df68073543d8632b1eb7d9957c4874d72dd8522e99c56", "source": "adapters/python/openclaw/openclaw.fabric-adapter.json" }, "nvidia.fabric.opencode": { @@ -2519,6 +2584,48 @@ "pi_turn_started" ], "type": "object" + }, + "usage": { + "additionalProperties": false, + "properties": { + "nemo_fabric.detailed_token_usage.v1": { + "additionalProperties": false, + "properties": { + "cache_write_input_tokens": { + "minimum": 0, + "type": "integer" + }, + "cached_input_tokens": { + "minimum": 0, + "type": "integer" + }, + "input_tokens": { + "minimum": 0, + "type": "integer" + }, + "input_tokens_include_cache": { + "type": "boolean" + }, + "output_tokens": { + "minimum": 0, + "type": "integer" + }, + "output_tokens_include_reasoning": { + "type": "boolean" + }, + "reasoning_tokens": { + "minimum": 0, + "type": "integer" + }, + "total_tokens": { + "minimum": 0, + "type": "integer" + } + }, + "type": "object" + } + }, + "type": "object" } }, "model_schema": { @@ -2637,7 +2744,7 @@ "name": "nemo-fabric-adapters-pi", "version": "0.6.0" }, - "sha256": "cae10ac33988f37ef1fc81f26a3850701ef210a2701622feb9741fccf134872f", + "sha256": "2c171e2520f7add91df4542a278800df4734febb301a098571d317a3d9a24124", "source": "adapters/typescript/pi/pi.fabric-adapter.json" }, "nvidia.fabric.qwen": { diff --git a/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor/README.md b/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor/README.md index 17ad4c1fc..092922194 100644 --- a/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor/README.md +++ b/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor/README.md @@ -16,6 +16,21 @@ Refer to the [Harbor example](../../../../../../../examples/harbor/README.md) for runnable SWE-Bench commands, configuration variations, reward checks, and Relay artifacts. +## Model Provider + +Use `fabric_model_provider=openai` for an OpenAI-compatible endpoint whose served model ID starts with another vendor prefix. For example, combine it with `model_name="nvidia/nemotron-3-ultra-550b-a55b"` and `fabric_model_base_url` to select the OpenAI protocol while preserving the exact served model ID. Without an override, the provider is inferred from the model prefix as before. The override requires a model and remains subject to the selected adapter’s model schema. + +## Model Settings + +Pass adapter-declared model settings with `fabric_model_settings`, for example +`--ak 'fabric_model_settings={"cost_tracking":"disabled"}'` for an explicitly +unpriced Mini-SWE model. Pair that policy with a positive `fabric_max_turns`. +The bridge copies these settings into `models.default.settings`; the selected +adapter's model schema and runtime validation still apply. This option requires +a model. Keep credentials in managed environment variables, outside model +settings. This option applies to the SDK bridge; native Harbor owns a separate +implementation. + ## Credential Transport Use `fabric_model_api_key_env` to name the model credential variable. The name diff --git a/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor/fabric_agent.py b/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor/fabric_agent.py index c3da2bfb7..adcb5d21a 100644 --- a/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor/fabric_agent.py +++ b/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor/fabric_agent.py @@ -141,6 +141,14 @@ class FabricAgentOptions(AgentOptions): fabric_harness_settings: dict[str, Any] | None = Field( default=None, description="Selected adapter settings." ) + fabric_model_provider: str | None = Field( + default=None, + description="Fabric provider independent of the served model identifier.", + ) + fabric_model_settings: dict[str, Any] | None = Field( + default=None, + description="Adapter-declared settings for the default model.", + ) fabric_model_base_url: str | None = Field( default=None, description="Model provider base URL; requires a Harbor model.", @@ -207,6 +215,15 @@ def validate_adapter_id(cls, value: str) -> str: raise ValueError("fabric_adapter_id must not be empty") return value + @field_validator("fabric_model_provider") + @classmethod + def validate_model_provider(cls, value: str | None) -> str | None: + if value is not None and (not value or value != value.strip()): + raise ValueError( + "fabric_model_provider must be non-empty without surrounding whitespace" + ) + return value + @field_validator("fabric_model_api_key_env") @classmethod def validate_model_api_key_env(cls, value: str | None) -> str | None: @@ -323,6 +340,8 @@ def __init__( extra_env: dict[str, str] | None = None, *args: Any, fabric_model_api_key_env: str | None = None, + fabric_model_provider: str | None = None, + fabric_model_settings: dict[str, Any] | None = None, fabric_discovery_paths: list[str] | None = None, fabric_adapter_descriptor: Path | None = None, **kwargs: Any, @@ -339,6 +358,8 @@ def __init__( fabric_adapter_descriptor=fabric_adapter_descriptor, fabric_workspace=fabric_workspace, fabric_harness_settings=fabric_harness_settings, + fabric_model_provider=fabric_model_provider, + fabric_model_settings=fabric_model_settings, fabric_model_base_url=fabric_model_base_url, fabric_model_api_key_env=fabric_model_api_key_env, fabric_system_instruction=fabric_system_instruction, @@ -365,6 +386,8 @@ def __init__( self.fabric_adapter_descriptor = options.fabric_adapter_descriptor self.fabric_workspace = options.fabric_workspace self.fabric_harness_settings = dict(options.fabric_harness_settings or {}) + self.fabric_model_provider = options.fabric_model_provider + self.fabric_model_settings = options.fabric_model_settings self.fabric_model_base_url = options.fabric_model_base_url self.fabric_model_api_key_env = options.fabric_model_api_key_env self.fabric_system_instruction = options.fabric_system_instruction @@ -573,6 +596,8 @@ def _build_config(self) -> FabricConfig: adapter_id=self.fabric_adapter_id, workspace=self.fabric_workspace, harness_settings=self.fabric_harness_settings, + model_provider_name=self.fabric_model_provider, + model_settings=self.fabric_model_settings, model_base_url=self.fabric_model_base_url, model_api_key_env=self.fabric_model_api_key_env, system_instruction=self.fabric_system_instruction, @@ -671,6 +696,8 @@ def build_harbor_config( adapter_id: str, workspace: str, harness_settings: dict[str, Any] | None = None, + model_provider_name: str | None = None, + model_settings: dict[str, Any] | None = None, model_base_url: str | None = None, model_api_key_env: str | None = None, system_instruction: str | None = None, @@ -735,11 +762,18 @@ def build_harbor_config( ) if model_name: config.models["default"] = ModelConfig( - provider=model_provider(model_name), + provider=model_provider_name + if model_provider_name is not None + else model_provider(model_name), model=model_name, base_url=model_base_url, api_key_env=model_api_key_env, + settings=dict(model_settings or {}), ) + elif model_provider_name is not None: + raise ValueError("model_provider_name requires model_name") + elif model_settings is not None: + raise ValueError("model_settings requires model_name") elif model_base_url is not None: raise ValueError("model_base_url requires model_name") elif model_api_key_env is not None: diff --git a/sdk/python/nemo-fabric/uv.lock b/sdk/python/nemo-fabric/uv.lock index c82f36121..ca412986a 100644 --- a/sdk/python/nemo-fabric/uv.lock +++ b/sdk/python/nemo-fabric/uv.lock @@ -2138,6 +2138,8 @@ dependencies = [ [package.optional-dependencies] harness = [ { name = "fsspec" }, + { name = "litellm", version = "1.96.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12' or python_full_version >= '3.14'" }, + { name = "litellm", version = "1.100.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12' and python_full_version < '3.14'" }, { name = "mini-swe-agent" }, ] @@ -2145,6 +2147,8 @@ harness = [ requires-dist = [ { name = "fsspec", marker = "extra == 'full'", specifier = ">=2026.6.0" }, { name = "fsspec", marker = "extra == 'harness'", specifier = ">=2026.6.0" }, + { name = "litellm", marker = "extra == 'full'", specifier = "<1.106" }, + { name = "litellm", marker = "extra == 'harness'", specifier = "<1.106" }, { name = "mini-swe-agent", marker = "extra == 'full'", specifier = ">=2.0,<3" }, { name = "mini-swe-agent", marker = "extra == 'harness'", specifier = ">=2.0,<3" }, { name = "nemo-fabric-adapter-contract", editable = "../../../adapter-contract/python" }, @@ -2166,6 +2170,7 @@ dependencies = [ [package.optional-dependencies] harness = [ { name = "fsspec" }, + { name = "litellm", version = "1.100.1", source = { registry = "https://pypi.org/simple" } }, { name = "nooa" }, { name = "nooa-bench" }, { name = "nooa-cli" }, @@ -2175,6 +2180,8 @@ harness = [ requires-dist = [ { name = "fsspec", marker = "extra == 'full'", specifier = ">=2026.6.0" }, { name = "fsspec", marker = "extra == 'harness'", specifier = ">=2026.6.0" }, + { name = "litellm", marker = "extra == 'full'", specifier = "<1.106" }, + { name = "litellm", marker = "extra == 'harness'", specifier = "<1.106" }, { name = "nemo-fabric-adapter-contract", editable = "../../../adapter-contract/python" }, { name = "nemo-fabric-adapters-common", editable = "../../../adapters/python/common" }, { name = "nemo-relay", marker = "extra == 'full'", specifier = ">=0.11,<0.12" }, diff --git a/skills/nemo-fabric-integrate/SKILL.md b/skills/nemo-fabric-integrate/SKILL.md index 4fbb71928..24237fbe1 100644 --- a/skills/nemo-fabric-integrate/SKILL.md +++ b/skills/nemo-fabric-integrate/SKILL.md @@ -424,6 +424,12 @@ Link to these canonical sources instead of duplicating them: - Platform and evaluation-harness integration: [examples/harbor](https://github.com/NVIDIA/NeMo-Fabric/tree/main/examples/harbor) and [nemo_fabric.integrations.harbor](https://github.com/NVIDIA/NeMo-Fabric/tree/main/sdk/python/nemo-fabric-runtime/src/nemo_fabric/integrations/harbor). + For OpenAI-compatible endpoints with vendor-prefixed served model IDs, set + `fabric_model_provider=openai` and retain the exact Harbor model name. Omit + this option to keep provider inference from the model prefix. + Use `fabric_model_settings` for adapter-declared `models.default.settings`; + a model is required and its descriptor still validates those settings. Keep + model credentials in Harbor's managed environment variables. Harbor constructs a typed config from explicit agent inputs and transports it inside a private transient run specification at the task-process boundary. Follow the code-review example for consumer integration code; Harbor's diff --git a/tests/adapters/test_deepagents.py b/tests/adapters/test_deepagents.py index baa45868f..4e16fd8fd 100644 --- a/tests/adapters/test_deepagents.py +++ b/tests/adapters/test_deepagents.py @@ -173,6 +173,8 @@ def fake_sdks_fixture(monkeypatch): recorder["init_chat_model"] = mock_init_chat_model recorder["fs_backend"] = mock_fs_backend recorder["local_shell_backend"] = mock_local_shell_backend + recorder["local_shell_factory"] = adapter._local_shell_backend + monkeypatch.setattr(adapter, "_local_shell_backend", mock_local_shell_backend) def build_agent(**kwargs): recorder["create_kwargs"] = kwargs @@ -485,6 +487,9 @@ def use_real_langgraph_fixture(fake_sdks, monkeypatch): def use_real_deepagents_fixture(fake_sdks, monkeypatch): """Drop the fake Deep Agents stubs so the real backend package resolves.""" + monkeypatch.setattr( + adapter, "_local_shell_backend", fake_sdks["local_shell_factory"] + ) for name in ( "deepagents", "deepagents.backends", @@ -524,6 +529,22 @@ async def test_single_invocation_normalizes_response_usage_and_thread( assert "instructions" not in fake_sdks["create_kwargs"] +async def test_invocation_preserves_inclusive_input_usage( + tmp_path, make_payload, fake_sdks +): + payload = make_payload(tmp_path) + runtime = adapter.DeepAgentsRuntime() + await runtime.start(lifecycle_start_payload(payload)) + try: + result = await runtime.invoke(*lifecycle_invocation(payload)) + assert result.usage.input_tokens == 5 + assert result.usage.input_tokens_include_cache is True + assert result.usage.cached_input_tokens is None + assert result.usage.output_tokens == 7 + finally: + await runtime.stop() + + @pytest.mark.parametrize( ("request_input", "encoded"), [(0, "0"), (False, "false"), ({}, "{}"), ([], "[]")], @@ -1660,6 +1681,133 @@ def test_local_shell_backend_keeps_file_and_shell_path_namespaces_distinct( assert host_absolute.exit_code != 0 +@pytest.mark.usefixtures("use_real_deepagents") +@pytest.mark.parametrize("nested", [False, True]) +@pytest.mark.parametrize("filename", ["report.txt", "report ' $ `.txt"]) +def test_local_shell_file_tools_accept_physical_workspace_paths( + tmp_path, make_payload, nested, filename +): + workspace = tmp_path / "workspace" + workspace.mkdir() + target = workspace / "nested" / filename if nested else workspace / filename + payload = make_payload(tmp_path) + payload["runtime_context"]["environment"]["workspace"] = str(workspace) + payload["runtime_context"]["environment"]["env"] = { + "FABRIC_TEST_TARGET": str(target) + } + backend = adapter.resolve_backend( + RuntimeContext.from_mapping(payload["runtime_context"]), + str(tmp_path), + {"type": "local_shell"}, + ) + result = backend.write(str(target), "same workspace") + + assert result.error is None + assert target.read_text() == "same workspace" + shell = backend.execute( + f'"{sys.executable}" -c "import os; from pathlib import Path; ' + "assert Path(os.environ['FABRIC_TEST_TARGET']).read_text() == 'same workspace'\"" + ) + assert shell.exit_code == 0 + read = backend.read(str(target)) + assert read.error is None + assert "same workspace" in read.file_data["content"] + + +@pytest.mark.usefixtures("use_real_deepagents") +@pytest.mark.parametrize("operation", ["ls", "glob", "grep"]) +def test_local_shell_listed_paths_round_trip_with_workspace_prefix( + tmp_path, make_payload, operation +): + from deepagents.middleware.filesystem import FilesystemMiddleware + from langchain.tools import ToolRuntime + + workspace = tmp_path / "workspace" + workspace.mkdir() + payload = make_payload(tmp_path) + payload["runtime_context"]["environment"]["workspace"] = str(workspace) + backend = adapter.resolve_backend( + RuntimeContext.from_mapping(payload["runtime_context"]), + str(tmp_path), + {"type": "local_shell"}, + ) + outer = workspace / "report.txt" + outer.write_text("outer file") + # On POSIX, the native virtual path of this file equals the outer file's + # physical path. On Windows this checks that virtual output paths survive + # the tools' validation instead of exposing rejected drive-prefixed paths. + nested = workspace / Path(*backend.cwd.parts[1:]) / "report.txt" + nested.parent.mkdir(parents=True) + nested.write_text("nested file") + + if operation == "ls": + result = backend.ls(str(nested.parent)) + entries = result.entries + elif operation == "glob": + result = backend.glob("*.txt", path=str(nested.parent)) + entries = result.matches + else: + result = backend.grep("nested file", path=str(nested.parent)) + entries = result.matches + + assert result.error is None + assert len(entries) == 1 + listed_path = entries[0]["path"] + tools = { + tool.name: tool + for tool in FilesystemMiddleware( + backend=backend, tools=["read_file", "edit_file"] + ).tools + } + runtime = ToolRuntime( + state={}, + context=None, + config={}, + stream_writer=lambda _: None, + tool_call_id="path-round-trip", + store=None, + ) + read = tools["read_file"].func(file_path=listed_path, runtime=runtime) + assert read.status == "success", read.content + assert "nested file" in read.content + edit = tools["edit_file"].func( + file_path=listed_path, + old_string="nested file", + new_string="edited nested file", + runtime=runtime, + ) + assert edit.status == "success", edit.content + assert nested.read_text() == "edited nested file" + assert outer.read_text() == "outer file" + + +@pytest.mark.usefixtures("use_real_deepagents") +@pytest.mark.parametrize("escape", ["traversal", "symlink"]) +def test_local_shell_workspace_alias_preserves_file_confinement( + tmp_path, make_payload, escape +): + workspace = tmp_path / "workspace" + workspace.mkdir() + outside = tmp_path / "outside" + outside.mkdir() + payload = make_payload(tmp_path) + payload["runtime_context"]["environment"]["workspace"] = str(workspace) + backend = adapter.resolve_backend( + RuntimeContext.from_mapping(payload["runtime_context"]), + str(tmp_path), + {"type": "local_shell"}, + ) + if escape == "symlink": + (workspace / "link").symlink_to(outside, target_is_directory=True) + target = workspace / "link" / "escaped.txt" + else: + target = workspace / ".." / "outside" / "escaped.txt" + + with pytest.raises(ValueError, match="traversal|outside root"): + backend.write(str(target), "must not escape") + assert not (outside / "escaped.txt").exists() + + @pytest.mark.usefixtures("use_real_deepagents") def test_local_shell_backend_forwards_only_explicit_environment(tmp_path, make_payload): os.environ["AMBIENT_ONLY"] = "ambient" @@ -2110,6 +2258,31 @@ def test_aggregate_usage_discards_tokens_larger_than_uint64(): assert adapter._aggregate_usage([{"usage": {"input_tokens": 1 << 64}}]) is None +@pytest.mark.parametrize("missing", [None, {}, {"input_tokens": True}]) +def test_aggregate_usage_does_not_promote_partial_assistant_totals(missing): + messages = [ + { + "role": "assistant", + "usage": {"input_tokens": 5, "output_tokens": 7, "total_tokens": 12}, + }, + {"role": "assistant", "usage": missing}, + ] + assert adapter._aggregate_usage(messages) is None + + +def test_aggregate_usage_keeps_independently_complete_dimensions(): + assert adapter._aggregate_usage( + [ + { + "role": "assistant", + "usage": {"input_tokens": 5, "output_tokens": 7, "total_tokens": 12}, + }, + {"role": "assistant", "usage": {"input_tokens": 3}}, + {"role": "tool", "content": "no model usage"}, + ] + ) == {"prompt_tokens": 8} + + async def test_replayed_state_usage_counts_current_turn_only( tmp_path, make_payload, monkeypatch ): diff --git a/tests/adapters/test_mini_swe_agent.py b/tests/adapters/test_mini_swe_agent.py index 63399525f..c19e00ee4 100644 --- a/tests/adapters/test_mini_swe_agent.py +++ b/tests/adapters/test_mini_swe_agent.py @@ -308,7 +308,8 @@ async def test_mini_swe_agent_maps_config_and_returns_normalized_output( await runtime.start(start) mock_mini["model_factory"].assert_called_once_with( - model_name="nvidia/test-model", + cost_tracking="default", + model_name="nvidia_nim/nvidia/test-model", model_kwargs={ "api_key": "test-key", "api_base": "https://example.test/v1", @@ -630,3 +631,184 @@ def test_mini_swe_agent_module_entrypoint_exits_cleanly(): ) assert result.returncode == 0, result.stderr + + +@pytest.mark.parametrize( + "model_name", + [ + "nvidia/nemotron-3-ultra-550b-a55b", + "nvidia_nim/nvidia/nemotron-3-ultra-550b-a55b", + ], +) +async def test_nvidia_dispatch_preserves_the_served_model_id( + mock_mini, mini_payload, monkeypatch, model_name +): + from litellm import get_llm_provider + + monkeypatch.setenv("TEST_MINI_API_KEY", "test-key") + mini_payload["config"]["models"]["default"]["model"] = model_name + runtime = adapter.MiniSweAgentRuntime() + await runtime.start( + {**mini_payload, "config": AgentConfig.from_mapping(mini_payload["config"])} + ) + dispatched = mock_mini["model_factory"].call_args.kwargs["model_name"] + model, provider, _, _ = get_llm_provider(dispatched) + assert provider == "nvidia_nim" + assert model == "nvidia/nemotron-3-ultra-550b-a55b" + + +@pytest.mark.parametrize("policy", [None, "default", "disabled"]) +async def test_cost_policy_is_explicit_and_controls_native_budget( + mock_mini, mini_payload, policy +): + import os + + os.environ["MSWEA_COST_TRACKING"] = "ignore_errors" + if policy is not None: + mini_payload["config"]["models"]["default"]["settings"] = { + "cost_tracking": policy + } + runtime = adapter.MiniSweAgentRuntime() + await runtime.start( + {**mini_payload, "config": AgentConfig.from_mapping(mini_payload["config"])} + ) + + assert mock_mini["model_factory"].call_args.kwargs["cost_tracking"] == ( + "ignore_errors" if policy == "disabled" else "default" + ) + assert runtime._agent.config.cost_limit == (0 if policy == "disabled" else 3) + result = await runtime.invoke(*invocation(mini_payload)) + if policy == "disabled": + assert result.output["usage"] == { + "api_calls": 1, + "cost_tracking": "disabled", + "cost_status": "unavailable", + } + else: + assert result.output["usage"] == {"api_calls": 1} + + +@pytest.mark.parametrize("runtime_config", [None, {}]) +async def test_unpriced_model_requires_finite_turn_budget(mini_payload, runtime_config): + mini_payload["config"]["models"]["default"]["settings"] = { + "cost_tracking": "disabled" + } + mini_payload["config"]["runtime"] = runtime_config + with pytest.raises(adapter.lifecycle.LifecycleError) as caught: + await adapter.MiniSweAgentRuntime().start( + {**mini_payload, "config": AgentConfig.from_mapping(mini_payload["config"])} + ) + assert caught.value.code == "mini_swe_agent_unbounded_unpriced_model" + + +@pytest.mark.parametrize( + "settings", + [ + {"cost_tracking": "ignore_errors"}, + {"cost_tracking": "disabled", "cost_limit": 2}, + ], +) +async def test_cost_policy_rejects_unknown_settings_or_conflicting_cap( + mini_payload, settings +): + mini_payload["config"]["models"]["default"]["settings"] = settings + with pytest.raises(adapter.lifecycle.LifecycleError): + await adapter.MiniSweAgentRuntime().start( + {**mini_payload, "config": AgentConfig.from_mapping(mini_payload["config"])} + ) + + +@pytest.mark.parametrize("policy", ["default", "disabled"]) +async def test_real_sdk_unpriced_response_requires_opt_in( + mini_payload, monkeypatch, policy +): + from litellm import ModelResponse + from minisweagent.models.litellm_model import LitellmModel + from nemo_fabric_adapters.mini_swe_agent.agents import _relay_response + + mini_payload["config"]["models"]["default"]["settings"] = {"cost_tracking": policy} + response = ModelResponse( + model="nvidia_nim/nvidia/test-model", + choices=[ + { + "message": { + "role": "assistant", + "content": "Working.", + "tool_calls": [ + { + "id": "call-1", + "type": "function", + "function": { + "name": "bash", + "arguments": '{"command":"submit"}', + }, + } + ], + }, + "finish_reason": "tool_calls", + } + ], + usage={"prompt_tokens": 8, "completion_tokens": 4, "total_tokens": 12}, + ) + completion = MagicMock(return_value=response) + monkeypatch.setattr("litellm.completion", completion) + monkeypatch.setattr( + "litellm.cost_calculator.completion_cost", + MagicMock(side_effect=ValueError("model has no registered price")), + ) + environment = MagicMock() + environment.execute.side_effect = Submitted( + { + "role": "exit", + "content": "done", + "extra": {"exit_status": "Submitted", "submission": "done"}, + } + ) + monkeypatch.setattr( + "minisweagent.environments.local.LocalEnvironment", + MagicMock(return_value=environment), + ) + runtime = adapter.MiniSweAgentRuntime() + await runtime.start( + {**mini_payload, "config": AgentConfig.from_mapping(mini_payload["config"])} + ) + assert isinstance(runtime._model, LitellmModel) + if policy == "default": + with pytest.raises(RuntimeError, match="Error calculating cost"): + await runtime.invoke(*invocation(mini_payload)) + environment.execute.assert_not_called() + else: + result = await runtime.invoke(*invocation(mini_payload)) + assert result.status.value == "succeeded" + assert result.output["usage"]["cost_status"] == "unavailable" + message = next(m for m in runtime._agent.messages if m["role"] == "assistant") + assert message["extra"]["cost"] == 0 # SDK sentinel, not a measured cost. + relayed = _relay_response(message, include_cost=False) + assert relayed["usage"] == response.model_dump()["usage"] + assert "cost" not in relayed["usage"] + completion.assert_called_once() + + +async def test_unpriced_relay_preserves_tokens_without_synthetic_cost( + mock_mini, mini_payload, mock_relay +): + mini_payload["config"]["models"]["default"]["settings"] = { + "cost_tracking": "disabled" + } + mini_payload["runtime_context"]["telemetry"] = { + "relay_enabled": True, + "metadata": {"telemetry_providers": ["relay"]}, + } + mock_mini["model"].config.cost_tracking = "ignore_errors" + runtime = adapter.MiniSweAgentRuntime() + await runtime.start( + {**mini_payload, "config": AgentConfig.from_mapping(mini_payload["config"])} + ) + result = await runtime.invoke(*invocation(mini_payload)) + assert result.status == "succeeded" + assert result.output["usage"]["cost_status"] == "unavailable" + assert mock_relay["llm_ends"][0][1]["usage"] == { + "prompt_tokens": 8, + "completion_tokens": 4, + "total_tokens": 12, + } diff --git a/tests/adapters/test_nooa_bench_adapter.py b/tests/adapters/test_nooa_bench_adapter.py index 9bab0a9d3..687d80399 100644 --- a/tests/adapters/test_nooa_bench_adapter.py +++ b/tests/adapters/test_nooa_bench_adapter.py @@ -165,6 +165,7 @@ async def invoke_telemetry(*, call: Any, **_kwargs: Any): "agents": mock_agents, "bench_agent": mock_bench_agent, "get_llm_client": mock_get_llm_client, + "get_task_tokens": mock_get_task_tokens, "initial_shell": mock_initial_shells[0], "initial_shells": mock_initial_shells, "model": mock_model, @@ -215,6 +216,78 @@ def test_bench_descriptor_is_a_closed_harness_adapter(tmp_path: Path): assert plan["adapter"]["adapter_kind"] == "python" +@pytest.mark.parametrize( + "tokens", + [ + None, + {}, + {"n_input_tokens": 0, "n_output_tokens": 0}, + {"n_input_tokens": True, "n_output_tokens": 2}, + {"n_input_tokens": -1, "n_output_tokens": 2}, + {"n_input_tokens": 4, "n_output_tokens": None}, + ], +) +def test_bench_usage_does_not_promote_missing_or_default_zero(tokens: Any): + assert bench_adapter._usage(tokens) is None + + +@pytest.mark.parametrize( + ("tokens", "expected"), + [ + ( + {"n_input_tokens": 8, "n_output_tokens": 0}, + {"input_tokens": 8, "input_tokens_include_cache": True}, + ), + ({"n_input_tokens": 0, "n_output_tokens": 3}, {"output_tokens": 3}), + ], +) +def test_bench_usage_keeps_only_positive_observed_dimensions( + tokens: Any, expected: Any +): + assert bench_adapter._usage(tokens).to_mapping() == expected + + +@pytest.mark.skipif( + sys.platform == "win32", + reason="Pinned NOOA SDK imports POSIX-only fcntl during initialization", +) +def test_pinned_nooa_metrics_bridge_preserves_inclusive_prompt_tokens_and_resets_tasks(): + os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True" + from litellm.types.utils import ModelResponse + from nooa.runtime.actor import _make_llm_metrics_bridge + from nooa.runtime.token_usage import get_task_tokens + from nooa.runtime.token_usage import start_task_tokens + from nooa.unifiedllm.unifiedllm import _extract_reasoning_and_usage + + bridge = _make_llm_metrics_bridge(MagicMock()) + start_task_tokens() + assert bench_adapter._usage(get_task_tokens()) is None + response = ModelResponse( + usage={ + "prompt_tokens": 100, + "completion_tokens": 20, + "total_tokens": 120, + "prompt_tokens_details": {"cached_tokens": 30}, + "completion_tokens_details": {"reasoning_tokens": 5}, + } + ) + _, usage = _extract_reasoning_and_usage(response) + bridge("token_usage", usage) + result = bench_adapter._usage(get_task_tokens()) + assert result.to_mapping() == { + "input_tokens": 100, + "output_tokens": 20, + "total_tokens": 120, + "input_tokens_include_cache": True, + } + assert result.cached_input_tokens is None + assert result.cost_usd is None + start_task_tokens() + assert bench_adapter._usage(get_task_tokens()) is None + bridge("token_usage", {"completion_tokens": 5}) + assert bench_adapter._usage(get_task_tokens()).to_mapping() == {"output_tokens": 5} + + @pytest.mark.skipif( sys.platform == "win32", reason="NOOA BenchAgent storage requires the Unix-only fcntl module", @@ -369,6 +442,7 @@ async def test_bench_runtime_maps_task_result_usage_and_cleanup( assert result.usage is not None assert result.usage.to_mapping() == { "input_tokens": 12, + "input_tokens_include_cache": True, "output_tokens": 4, "total_tokens": 16, } @@ -425,6 +499,7 @@ async def test_bench_runtime_isolates_agents_between_invocations( ): shell.close.assert_awaited_once_with() assert first.status is second.status is AgentRunStatus.SUCCEEDED + assert bench_dependencies["start_task_tokens"].call_count == 2 bench_dependencies["model"].aclose.assert_awaited_once_with() @@ -451,9 +526,30 @@ async def test_bench_runtime_returns_safe_failure( assert result.status is AgentRunStatus.FAILED assert result.error is not None assert result.error.code == "nooa_bench_task_failed" + assert result.usage.input_tokens == 12 + assert result.usage.input_tokens_include_cache is True assert "secret-token-value" not in json.dumps(result.to_mapping()) +async def test_bench_runtime_exception_keeps_observed_usage( + tmp_path: Path, + bench_dependencies: dict[str, MagicMock], +): + os.environ["OPENAI_API_KEY"] = "test-key" + bench_dependencies["agent"]._run_evaluation.side_effect = RuntimeError("private") + runtime = bench_adapter.BenchRuntime() + await runtime.start(_start_payload(tmp_path)) + result = await runtime.invoke( + AgentRunRequest.from_mapping({"input": "task"}), _runtime_context() + ) + await runtime.stop() + assert result.status is AgentRunStatus.FAILED + assert result.usage.input_tokens == 12 + assert result.usage.output_tokens == 4 + assert result.usage.cost_usd is None + assert "private" not in json.dumps(result.to_mapping()) + + def test_bench_persistent_host_runs_start_invoke_stop(tmp_path: Path): workspace = tmp_path / "workspace" workspace.mkdir() @@ -505,7 +601,9 @@ def test_bench_persistent_host_runs_start_invoke_stop(tmp_path: Path): ] environment = os.environ.copy() environment["OPENAI_API_KEY"] = "fixture-key" - environment["PYTHONPATH"] = str(fixture_source) + environment["PYTHONPATH"] = os.pathsep.join( + (str(fixture_source), str(NOOA_ROOT / "src")) + ) completed = subprocess.run( [sys.executable, "-m", "nemo_fabric_adapters.nooa.bench_adapter"], @@ -528,6 +626,7 @@ def test_bench_persistent_host_runs_start_invoke_stop(tmp_path: Path): assert invocation["output"]["response"] == "test -f bench-agent-result.txt" assert invocation["usage"] == { "input_tokens": 12, + "input_tokens_include_cache": True, "output_tokens": 4, "total_tokens": 16, } diff --git a/tests/adapters/test_openclaw.py b/tests/adapters/test_openclaw.py index b1db58c31..0a71993f8 100644 --- a/tests/adapters/test_openclaw.py +++ b/tests/adapters/test_openclaw.py @@ -402,7 +402,11 @@ async def test_openclaw_runtime_generates_config_invokes_and_cleans_up( request = json.loads(request_capture.read_text(encoding="utf-8")) assert result.status == "succeeded" assert result.output == {"response": "OpenClaw response"} - assert result.usage.total_tokens == 5 + assert result.usage.total_tokens is None + assert result.usage.extensions[adapter.OPENCLAW_USAGE_EVIDENCE] == { + "observed": {"input_tokens": 3, "output_tokens": 2, "total_tokens": 5}, + "gaps": ["scope_unverified"], + } assert request == { "model": "openclaw/default", "messages": [ @@ -1820,3 +1824,113 @@ def test_openclaw_descriptor_and_module_entrypoint(repo_root: Path): assert descriptor["capabilities"]["streaming"] is False assert "telemetry" not in descriptor assert result.returncode == 0, result.stderr + + +@pytest.mark.parametrize("supports_usage", [True, False, None]) +def test_openclaw_config_maps_streaming_usage_capability( + tmp_path: Path, supports_usage: bool | None +): + config = _config(Path("openclaw")) + if supports_usage is not None: + config.models["default"].settings["supports_usage_in_streaming"] = ( + supports_usage + ) + generated = adapter._openclaw_config( + config, _context(tmp_path), base_dir=tmp_path, port=20000, token_env="TOKEN" + ) + model = generated["models"]["providers"]["test"]["models"][0] + if supports_usage is None: + assert "compat" not in model + else: + assert model["compat"] == {"supportsUsageInStreaming": supports_usage} + + +@pytest.mark.parametrize("value", [None, "true", 1]) +def test_openclaw_config_rejects_invalid_streaming_usage_capability( + tmp_path: Path, value +): + config = _config(Path("openclaw")) + config.models["default"].settings["supports_usage_in_streaming"] = value + with pytest.raises(adapter.lifecycle.LifecycleError) as caught: + adapter._openclaw_config( + config, _context(tmp_path), base_dir=tmp_path, port=20000, token_env="TOKEN" + ) + assert caught.value.code == "openclaw_invalid_configuration" + + +def test_openclaw_streaming_usage_capability_requires_custom_endpoint(tmp_path: Path): + config = _config(Path("openclaw")) + config.models["default"].base_url = None + config.models["default"].settings["supports_usage_in_streaming"] = True + with pytest.raises(adapter.lifecycle.LifecycleError): + adapter._openclaw_config( + config, _context(tmp_path), base_dir=tmp_path, port=20000, token_env="TOKEN" + ) + + +@pytest.mark.parametrize( + ("counters", "observed", "gaps"), + [ + (None, {}, ["scope_unverified", "missing_usage"]), + ({}, {}, ["scope_unverified", "missing_usage"]), + ( + {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}, + {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}, + ["scope_unverified", "zero_unverified"], + ), + ( + {"input_tokens": 42, "output_tokens": 8, "total_tokens": 50}, + {"input_tokens": 42, "output_tokens": 8, "total_tokens": 50}, + ["scope_unverified"], + ), + ], +) +def test_openclaw_gateway_usage_remains_observational(counters, observed, gaps): + from nemo_fabric_adapter_contract.models import AgentUsage + + usage = adapter._gateway_usage_evidence( + AgentUsage(**counters) if counters is not None else None + ) + assert usage.input_tokens is None + assert usage.output_tokens is None + assert usage.total_tokens is None + assert usage.cost_usd is None + assert usage.extensions == { + adapter.OPENCLAW_USAGE_EVIDENCE: {"observed": observed, "gaps": gaps} + } + + +async def test_openclaw_gateway_stream_keeps_usage_observational(): + import httpx + + chunks = [ + {"choices": [{"delta": {"content": "done"}}]}, + { + "choices": [], + "usage": {"prompt_tokens": 42, "completion_tokens": 8, "total_tokens": 50}, + }, + ] + body = "".join(f"data: {json.dumps(chunk)}\n\n" for chunk in chunks) + body += "data: [DONE]\n\n" + response = httpx.Response( + 200, text=body, headers={"content-type": "text/event-stream"} + ) + async with httpx.AsyncClient( + transport=httpx.MockTransport(lambda request: response) + ) as client: + text, usage = await adapter._invoke_gateway( + client, + "http://gateway.example.test/v1/chat/completions", + messages=[{"role": "user", "content": "finish"}], + temperature=None, + top_p=None, + max_tokens=None, + user="runtime-test", + ) + assert text == "done" + assert usage.input_tokens is None + assert usage.output_tokens is None + assert usage.extensions[adapter.OPENCLAW_USAGE_EVIDENCE] == { + "observed": {"input_tokens": 42, "output_tokens": 8, "total_tokens": 50}, + "gaps": ["scope_unverified"], + } diff --git a/tests/adapters/test_openhands.py b/tests/adapters/test_openhands.py index 63de825ac..879c58f9a 100644 --- a/tests/adapters/test_openhands.py +++ b/tests/adapters/test_openhands.py @@ -100,6 +100,9 @@ def mock_openhands_fixture(monkeypatch: pytest.MonkeyPatch) -> dict: metrics_delta = SimpleNamespace( accumulated_token_usage=usage, accumulated_cost=0.25, + token_usages=[usage], + costs=[SimpleNamespace(cost=0.25)], + response_latencies=[SimpleNamespace(response_id="response-test")], ) metrics = MagicMock() metrics.deep_copy.return_value = object() @@ -248,7 +251,7 @@ async def test_start_maps_normalized_configuration( await runtime.start(_start_payload(openhands_payload)) mock_openhands["llm_factory"].assert_called_once_with( - model="nvidia/test-model", + model="nvidia_nim/nvidia/test-model", usage_id="agent", api_key="secret:test-key", base_url="https://example.test/v1", @@ -482,10 +485,99 @@ async def test_start_prefixes_an_unqualified_model_with_its_provider( await adapter.OpenHandsRuntime().start(_start_payload(openhands_payload)) assert mock_openhands["llm_factory"].call_args.kwargs["model"] == ( - "nvidia/test-model" + "nvidia_nim/test-model" ) +@pytest.mark.parametrize( + "model_name", + ["nvidia/test-model", "nvidia_nim/nvidia/test-model"], +) +@pytest.mark.parametrize("with_tools", [False, True]) +def test_nvidia_custom_endpoint_preserves_served_model_id( + model_name: str, + with_tools: bool, + monkeypatch: pytest.MonkeyPatch, +): + pytest.importorskip("openhands.sdk") + pytest.importorskip("openhands.tools") + import httpx + import litellm + from litellm.caching.caching import InMemoryCache + from openhands.sdk.llm import Message + from openhands.sdk.llm import TextContent + from openhands.sdk.tool import FinishTool + + requests = [] + + def respond(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response( + 200, + json={ + "id": "chatcmpl-nim", + "object": "chat.completion", + "created": 0, + "model": "nvidia/test-model", + "choices": [ + { + "index": 0, + "finish_reason": "stop", + "message": {"role": "assistant", "content": "done"}, + } + ], + "usage": { + "prompt_tokens": 11, + "prompt_tokens_details": {"cached_tokens": 3}, + "completion_tokens": 5, + "total_tokens": 16, + }, + }, + ) + + model = AgentConfig.from_mapping( + { + "models": { + "default": { + "provider": "nvidia", + "model": model_name, + "base_url": "http://nim.example.test/v1", + "api_key_env": "TEST_OPENHANDS_API_KEY", + } + } + } + ).models["default"] + monkeypatch.setenv("TEST_OPENHANDS_API_KEY", "unused") + # LiteLLM caches OpenAI clients across calls, including their HTTP transport. + monkeypatch.setattr(litellm, "in_memory_llm_clients_cache", InMemoryCache()) + with httpx.Client(transport=httpx.MockTransport(respond)) as client: + monkeypatch.setattr(litellm, "client_session", client) + llm = adapter._llm(adapter._load_openhands_api(), model) + llm.num_retries = 0 + baseline = llm.metrics.deep_copy() + llm.completion( + [Message(role="user", content=[TextContent(text="finish")])], + tools=FinishTool.create() if with_tools else None, + ) + + assert len(requests) == 1 + assert str(requests[0].url) == "http://nim.example.test/v1/chat/completions" + payload = json.loads(requests[0].content) + assert payload["model"] == "nvidia/test-model" + if with_tools: + assert payload["tools"][0]["function"]["name"] == "finish" + assert llm.metrics.accumulated_token_usage.prompt_tokens == 11 + assert llm.metrics.accumulated_token_usage.completion_tokens == 5 + assert llm.metrics.accumulated_token_usage.cache_read_tokens == 3 + usage = adapter._usage(llm, baseline) + assert usage.input_tokens == 11 + assert usage.input_tokens_include_cache is True + assert usage.output_tokens == 5 + assert usage.total_tokens == 16 + assert usage.cached_input_tokens is None + assert usage.cost_usd is None + + async def test_repeated_invocation_returns_only_current_turn_and_usage( openhands_payload: dict, mock_openhands: dict, @@ -504,6 +596,7 @@ async def test_repeated_invocation_returns_only_current_turn_and_usage( assert second.usage is not None assert second.usage.to_mapping() == { "input_tokens": 11, + "input_tokens_include_cache": True, "output_tokens": 5, "total_tokens": 16, "cost_usd": 0.25, @@ -589,6 +682,9 @@ async def test_invocation_failure_reports_the_underlying_sdk_error( assert result.error.message == "OpenHands could not complete the invocation." assert result.error.retryable is False assert result.error.extensions == {"sdk_error": sdk_error} + assert result.usage.input_tokens == 11 + assert result.usage.output_tokens == 5 + assert result.usage.cost_usd == 0.25 finally: await runtime.stop() @@ -704,9 +800,7 @@ async def test_start_rejects_unapplied_mcp_extensions( monkeypatch: pytest.MonkeyPatch, ): monkeypatch.setenv("TEST_OPENHANDS_API_KEY", "test-key") - openhands_payload["config"]["mcp"]["extensions"] = { - "example.test/option": True - } + openhands_payload["config"]["mcp"]["extensions"] = {"example.test/option": True} with pytest.raises(adapter.lifecycle.LifecycleError) as caught: await adapter.OpenHandsRuntime().start(_start_payload(openhands_payload)) @@ -894,3 +988,66 @@ def changed(): None, ) assert "private" not in json.dumps(result) + + +@pytest.mark.parametrize("response_without_usage", [False, True]) +def test_native_metrics_defaults_do_not_establish_zero_usage(response_without_usage): + pytest.importorskip("openhands.sdk") + from openhands.sdk.llm.utils.metrics import Metrics + + metrics = Metrics(model_name="test-model") + baseline = metrics.deep_copy() + if response_without_usage: + metrics.add_response_latency(0.1, "response-test") + assert adapter._usage(SimpleNamespace(metrics=metrics), baseline) is None + + +@pytest.mark.parametrize("tokens", [(11, 5), (0, 0)]) +@pytest.mark.parametrize("cost", [None, 0.0, 0.25]) +def test_native_metrics_preserve_recorded_usage_and_only_recorded_cost(tokens, cost): + pytest.importorskip("openhands.sdk") + from openhands.sdk.llm.utils.metrics import Metrics + + metrics = Metrics(model_name="test-model") + baseline = metrics.deep_copy() + metrics.add_response_latency(0.1, "response-test") + metrics.add_token_usage( + prompt_tokens=tokens[0], + completion_tokens=tokens[1], + cache_read_tokens=0, + cache_write_tokens=0, + context_window=0, + response_id="response-test", + ) + if cost is not None: + metrics.add_cost(cost) + usage = adapter._usage(SimpleNamespace(metrics=metrics), baseline) + assert usage.input_tokens == tokens[0] + assert usage.output_tokens == tokens[1] + assert usage.total_tokens == sum(tokens) + assert usage.cost_usd == cost + # An unchanged warm conversation supplies no new per-call evidence. + assert adapter._usage(SimpleNamespace(metrics=metrics), metrics.deep_copy()) is None + + +def test_native_metrics_do_not_report_partially_priced_cost(): + pytest.importorskip("openhands.sdk") + from openhands.sdk.llm.utils.metrics import Metrics + + metrics = Metrics(model_name="test-model") + baseline = metrics.deep_copy() + for response_id in ["first", "second"]: + metrics.add_response_latency(0.1, response_id) + metrics.add_token_usage( + prompt_tokens=11, + completion_tokens=5, + cache_read_tokens=0, + cache_write_tokens=0, + context_window=0, + response_id=response_id, + ) + metrics.add_cost(0.25) + usage = adapter._usage(SimpleNamespace(metrics=metrics), baseline) + assert usage.input_tokens == 22 + assert usage.output_tokens == 10 + assert usage.cost_usd is None diff --git a/tests/e2e/test_mini_swe_agent.py b/tests/e2e/test_mini_swe_agent.py index 8e03b7ccf..c5ab1e013 100644 --- a/tests/e2e/test_mini_swe_agent.py +++ b/tests/e2e/test_mini_swe_agent.py @@ -35,6 +35,7 @@ def mini_config(api_server: str, workspace: Path) -> FabricConfig: api_key_env="FABRIC_TEST_API_KEY", base_url=f"{api_server}/v1", temperature=0.0, + settings={"cost_tracking": "disabled"}, ) }, instructions=InstructionsConfig( @@ -75,7 +76,6 @@ async def test_mini_swe_agent_plans_diagnoses_and_runs( ) response.raise_for_status() monkeypatch.setenv("MSWEA_SILENT_STARTUP", "1") - monkeypatch.setenv("MSWEA_COST_TRACKING", "ignore_errors") monkeypatch.setenv("MSWEA_GLOBAL_CONFIG_DIR", str(tmp_path / "mini-config")) monkeypatch.setenv("ADAPTER_PYTHON", sys.executable) config = mini_config(api_server, tmp_path) @@ -116,7 +116,6 @@ async def test_mini_swe_agent_streams_relay_telemetry( ) response.raise_for_status() monkeypatch.setenv("MSWEA_SILENT_STARTUP", "1") - monkeypatch.setenv("MSWEA_COST_TRACKING", "ignore_errors") monkeypatch.setenv("MSWEA_GLOBAL_CONFIG_DIR", str(tmp_path / "mini-config")) monkeypatch.setenv("ADAPTER_PYTHON", sys.executable) config = mini_config(api_server, tmp_path) diff --git a/tests/e2e/test_nooa_bench_harbor.py b/tests/e2e/test_nooa_bench_harbor.py index 1a5d54254..122811971 100644 --- a/tests/e2e/test_nooa_bench_harbor.py +++ b/tests/e2e/test_nooa_bench_harbor.py @@ -151,6 +151,7 @@ async def execute(command: str, **kwargs: Any) -> SimpleNamespace: } assert result["usage"] == { "input_tokens": 12, + "input_tokens_include_cache": True, "output_tokens": 4, "total_tokens": 16, "metadata": {}, @@ -162,3 +163,5 @@ async def execute(command: str, **kwargs: Any) -> SimpleNamespace: assert context.metadata["fabric"]["adapter_id"] == ( "nvidia.fabric.nooa.bench-agent" ) + assert context.n_input_tokens == 12 + assert context.n_output_tokens == 4 diff --git a/tests/e2e/test_qwen.py b/tests/e2e/test_qwen.py index 0785f1542..4d3397852 100644 --- a/tests/e2e/test_qwen.py +++ b/tests/e2e/test_qwen.py @@ -106,7 +106,11 @@ async def test_qwen_doctor_and_run_against_local_provider( assert "user_count=1" in first["output"]["response"] assert "second" in second["output"]["response"] assert "user_count=2" in second["output"]["response"] - assert second["usage"]["input_tokens"] >= 0 + # This provider fixture emits all-zero usage, which the pinned SDK also + # synthesizes for missing usage. Neither proves a zero-token invocation. + for result in (single, first, second): + assert "input_tokens" not in result["usage"] + assert "output_tokens" not in result["usage"] captured = requests.get(f"{api_server}/_requests", timeout=5).json() evidence = [ diff --git a/tests/examples/test_nooa_bench_harbor_example.py b/tests/examples/test_nooa_bench_harbor_example.py index 2f78f8059..501bc238f 100644 --- a/tests/examples/test_nooa_bench_harbor_example.py +++ b/tests/examples/test_nooa_bench_harbor_example.py @@ -168,6 +168,7 @@ def test_harbor_example_uses_published_nooa_packages(): "nooa>=0.0.10,<0.0.11", "nooa-cli>=0.0.10,<0.0.11", "nooa-bench>=0.0.10,<0.0.11", + "litellm<1.106", } assert set(adapter_project["optional-dependencies"]["full"]) == { *adapter_project["optional-dependencies"]["harness"], @@ -210,6 +211,6 @@ def test_harbor_example_uses_published_nooa_packages(): calculator_dockerfile ) assert ( - '&& uv pip install --python /opt/nemo-fabric-venv/bin/python \\\n' + "&& uv pip install --python /opt/nemo-fabric-venv/bin/python \\\n" ' "nemo-fabric-adapters-nooa[full]"' ) in swebench_dockerfile diff --git a/tests/integrations/test_harbor_runner.py b/tests/integrations/test_harbor_runner.py index c5d057076..ef7e0a9ac 100644 --- a/tests/integrations/test_harbor_runner.py +++ b/tests/integrations/test_harbor_runner.py @@ -745,6 +745,8 @@ def test_harbor_023_options_schema_and_preflight(): "fabric_max_turns", "fabric_model_api_key_env", "fabric_model_base_url", + "fabric_model_provider", + "fabric_model_settings", "fabric_package", "fabric_python", "fabric_runtime_timeout_seconds", diff --git a/tests/python/test_harbor_integration.py b/tests/python/test_harbor_integration.py index 663006f70..fc6d2617c 100644 --- a/tests/python/test_harbor_integration.py +++ b/tests/python/test_harbor_integration.py @@ -328,6 +328,64 @@ def test_harbor_generated_config_names_the_model_credential(tmp_path: Path): assert model.base_url == "https://integrate.api.nvidia.com/v1" +def test_harbor_model_provider_preserves_the_served_identifier(tmp_path): + agent = FabricAgent( + logs_dir=tmp_path, + fabric_adapter_id="nvidia.fabric.qwen", + fabric_model_provider="openai", + fabric_model_base_url="https://nim.example.test/v1", + fabric_model_api_key_env="MODEL_API_KEY", + model_name="nvidia/nemotron-3-ultra-550b-a55b", + ) + model = agent._build_spec("fix it").config.models["default"] + assert model.provider == "openai" + assert model.model == "nvidia/nemotron-3-ultra-550b-a55b" + assert model.base_url == "https://nim.example.test/v1" + assert model.api_key_env == "MODEL_API_KEY" + + +def test_harbor_transports_adapter_model_settings(tmp_path): + agent = FabricAgent( + logs_dir=tmp_path, + fabric_adapter_id="nvidia.fabric.mini-swe-agent", + fabric_model_settings={"cost_tracking": "disabled"}, + fabric_max_turns=20, + model_name="nvidia/nemotron-3-ultra-550b-a55b", + ) + config = agent._build_spec("fix it").config + assert config.models["default"].settings == {"cost_tracking": "disabled"} + assert config.runtime.max_turns == 20 + assert config.harness.settings == {} + + +def test_harbor_model_settings_require_a_model(tmp_path): + with pytest.raises(ValueError, match="model_settings requires model_name"): + FabricAgent( + logs_dir=tmp_path, + fabric_adapter_id="nvidia.fabric.mini-swe-agent", + fabric_model_settings={"cost_tracking": "disabled"}, + )._build_spec("fix it") + + +@pytest.mark.parametrize("provider", ["", " ", " openai", "openai "]) +def test_harbor_rejects_invalid_model_provider(tmp_path, provider): + with pytest.raises(ValueError, match="fabric_model_provider must be"): + FabricAgent( + logs_dir=tmp_path, + fabric_adapter_id="nvidia.fabric.qwen", + fabric_model_provider=provider, + ) + + +def test_harbor_model_provider_requires_a_model(tmp_path): + with pytest.raises(ValueError, match="model_provider_name requires model_name"): + FabricAgent( + logs_dir=tmp_path, + fabric_adapter_id="nvidia.fabric.qwen", + fabric_model_provider="openai", + ) + + def test_harbor_model_credential_without_a_model_is_rejected(tmp_path: Path): with pytest.raises(ValueError, match="model_api_key_env requires model_name"): FabricAgent( diff --git a/tests/scripts/test_harbor_qualification.py b/tests/scripts/test_harbor_qualification.py new file mode 100644 index 000000000..2d78d3df1 --- /dev/null +++ b/tests/scripts/test_harbor_qualification.py @@ -0,0 +1,1183 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import argparse +import copy +import json +import os +import subprocess +import sys +from pathlib import Path +from unittest.mock import patch + +import pytest + +sys.path.insert( + 0, + str(Path(__file__).resolve().parents[2] / ".agents/skills/qualify-harbor/scripts"), +) +import qualify # noqa: E402 +from prepare import adapter_dockerfile # noqa: E402 + +pytestmark = pytest.mark.skipif( + os.name != "posix", + reason="Qualification uses POSIX process groups and Linux task images", +) + + +@pytest.fixture(name="qualification_plan") +def qualification_plan_fixture() -> dict: + return qualify.make_plan( + argparse.Namespace( + task=qualify.ROOT / "examples/harbor/calculator/task", + mode="source", + version=qualify.load(qualify.ROOT / qualify.CATALOG)["catalog_version"], + harbor_version="0.24.0", + bridge="native", + model="nvidia/local-model", + base_url="http://127.0.0.1:8000/v1", + api_key_env="MODEL_API_KEY", + endpoint_kind="openai-and-anthropic", + trusted_http_host=[], + estimated_cost_usd=0, + cost_basis="local", + timeout=60, + ) + ) + + +@pytest.fixture(name="clean_qualification_checkout") +def clean_qualification_checkout_fixture(): + real_run = subprocess.run + + def run_with_clean_checkout(command, *args, **kwargs): + if command[:2] == ["git", "diff"] or command[1:] == [ + str(qualify.ROOT / "scripts/ci/generate_adapter_catalog.py"), + "--check", + ]: + return subprocess.CompletedProcess(command, 0) + return real_run(command, *args, **kwargs) + + with patch.object(qualify.subprocess, "run", side_effect=run_with_clean_checkout): + yield + + +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_plan_uses_complete_catalog_and_distinguishes_prerequisites(qualification_plan): + qualify.validate_plan(qualification_plan) + catalog = qualify.load(qualify.ROOT / qualify.CATALOG) + rows = {row["adapter_id"]: row for row in qualification_plan["adapters"]} + assert set(rows) == set(catalog["adapters"]) + assert rows["nvidia.fabric.claude"]["action"] == "run" + assert rows["nvidia.fabric.kiro"]["action"] == "blocked" + assert rows["nvidia.fabric.kiro"]["api_key_env"] == "KIRO_API_KEY" + assert "BYOM is unsupported" in rows["nvidia.fabric.kiro"]["reason"] + assert rows["nvidia.fabric.remote-agent"]["reason"] + assert rows["nvidia.fabric.pi"]["capabilities"]["mcp"] == "untested" + + +@pytest.fixture(name="workflow_row") +def workflow_row_fixture(qualification_plan): + qualification_plan["bridge"] = "sdk" + row = next( + r + for r in qualification_plan["adapters"] + if r["adapter_id"] == "nvidia.fabric.nooa" + ) + row.update( + workflow_target_id="nvidia.nooa.coding-agent", + workflow_settings={}, + action="run", + reason="", + ) + return row + + +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_workflow_selection_keeps_the_full_catalog(qualification_plan, workflow_row): + qualify.validate_plan(qualification_plan) + assert len(qualification_plan["adapters"]) == 18 + assert workflow_row["action"] == "run" + assert "nvidia.nooa.coding-agent" in workflow_row["registered_targets"] + + +@pytest.mark.usefixtures("clean_qualification_checkout") +@pytest.mark.parametrize( + "mutation", + [ + "native", + "empty", + "spaces", + "type", + "missing", + "wrong_adapter", + "settings_without_target", + "settings_type", + "settings_secret", + "nonfinite", + "target_drift", + ], +) +def test_workflow_selection_rejects_unattributable_routes( + qualification_plan, workflow_row, mutation +): + if mutation == "native": + qualification_plan["bridge"] = "native" + elif mutation == "empty": + workflow_row["workflow_target_id"] = "" + elif mutation == "spaces": + workflow_row["workflow_target_id"] = " nvidia.nooa.coding-agent" + elif mutation == "type": + workflow_row["workflow_target_id"] = 1 + elif mutation == "missing": + workflow_row["workflow_target_id"] = "not.registered" + elif mutation == "wrong_adapter": + other = next( + r + for r in qualification_plan["adapters"] + if r["adapter_id"] == "nvidia.fabric.nooa.bench-agent" + ) + other.update( + workflow_target_id=workflow_row.pop("workflow_target_id"), + workflow_settings=workflow_row.pop("workflow_settings"), + ) + workflow_row.update(action="unsupported", reason="Not selected") + elif mutation == "settings_without_target": + workflow_row.pop("workflow_target_id") + elif mutation == "settings_type": + workflow_row["workflow_settings"] = [] + elif mutation == "settings_secret": + workflow_row["workflow_settings"] = {"nested": {"api_key": "secret"}} + elif mutation == "nonfinite": + workflow_row["workflow_settings"] = {"value": float("nan")} + else: + qualification_plan["mode"] = "registry" + qualification_plan["catalog_snapshot"]["targets"][ + workflow_row["workflow_target_id"] + ]["descriptor"]["spec"]["entrypoint"]["ref"] = "changed:factory" + with pytest.raises(ValueError): + qualify.validate_plan(qualification_plan) + + +@pytest.mark.usefixtures("requires_harbor", "clean_qualification_checkout") +def test_workflow_command_payload_and_report_agree( + qualification_plan, workflow_row, tmp_path +): + command = qualify.harbor_command( + qualification_plan, workflow_row, "python", tmp_path, tmp_path, "trial" + ) + assert "fabric_workflow_target_id=nvidia.nooa.coding-agent" in command + assert "fabric_workflow_settings={}" in command + payload = qualify.selection_payload(qualification_plan, workflow_row) + assert payload["config"]["workflow"] == { + "target_id": workflow_row["workflow_target_id"] + } + target = qualification_plan["catalog_snapshot"]["targets"][ + workflow_row["workflow_target_id"] + ] + combined = qualify.admission_fingerprint( + payload, workflow_row["catalog"]["descriptor"], target["descriptor"] + ) + assert isinstance(combined, str) and len(combined) == 64 + changed = copy.deepcopy(target["descriptor"]) + changed["spec"]["entrypoint"]["ref"] = "different:factory" + assert ( + qualify.admission_fingerprint( + payload, workflow_row["catalog"]["descriptor"], changed + ) + != combined + ) + assert ( + qualify.run( + qualification_plan, + {"plan": qualification_plan}, + tmp_path / "report", + [workflow_row["adapter_id"]], + ) + == 1 + ) + report = qualify.load(tmp_path / "report/report.json") + selected = next( + r["selected"] + for r in report["adapters"] + if r["adapter_id"] == workflow_row["adapter_id"] + ) + assert selected["workflow_target_id"] == workflow_row["workflow_target_id"] + assert selected["workflow_settings"] == {} + assert len(report["adapters"]) == 18 + + +@pytest.mark.parametrize( + "mutation", [None, "missing", "factory", "source", "target_id"] +) +def test_workflow_provenance_requires_installed_matching_target( + qualification_plan, workflow_row, mutation +): + target = qualification_plan["catalog_snapshot"]["targets"][ + workflow_row["workflow_target_id"] + ] + selected = { + "adapter_id": workflow_row["adapter_id"], + "python_prefix": "/usr/local", + "provenance": [ + { + "source": "installed_package", + "path": "/usr/local/share/nemo-fabric/adapters/nooa/nooa.fabric-adapter.json", + } + ], + "workflow": { + "target_id": workflow_row["workflow_target_id"], + "entrypoint": copy.deepcopy(target["descriptor"]["spec"]["entrypoint"]), + "provenance": [ + { + "source": "installed_package", + "path": "/usr/local/share/nemo-fabric/adapters/nooa/targets/coding-agent.fabric-target.json", + } + ], + }, + } + if mutation == "missing": + selected.pop("workflow") + elif mutation == "factory": + selected["workflow"]["entrypoint"]["ref"] = "other:factory" + elif mutation == "source": + selected["workflow"]["provenance"][0]["source"] = "bundled" + elif mutation == "target_id": + selected["workflow"]["target_id"] = "other.target" + assert qualify.valid_selection(selected, workflow_row, target) is (mutation is None) + + +@pytest.mark.usefixtures("requires_harbor") +def test_workflow_settings_are_transported_and_admitted_by_target_schema( + qualification_plan, workflow_row, tmp_path +): + from nemo_fabric import FabricConfigError + + workflow_row.update( + workflow_target_id="nvidia.nooa.arc-solver", + workflow_settings={"reflect_every": 3}, + ) + command = qualify.harbor_command( + qualification_plan, workflow_row, "python", tmp_path, tmp_path, "trial" + ) + assert 'fabric_workflow_settings={"reflect_every": 3}' in command + payload = qualify.selection_payload(qualification_plan, workflow_row) + assert payload["config"]["workflow"]["settings"] == {"reflect_every": 3} + target = qualification_plan["catalog_snapshot"]["targets"][ + workflow_row["workflow_target_id"] + ] + assert qualify.admission_fingerprint( + payload, workflow_row["catalog"]["descriptor"], target["descriptor"] + ) + payload["config"]["workflow"]["settings"] = {"reflect_every": 0} + with pytest.raises(FabricConfigError): + qualify.admission_fingerprint( + payload, workflow_row["catalog"]["descriptor"], target["descriptor"] + ) + + +@pytest.mark.usefixtures("requires_harbor") +def test_workflow_identity_retains_target_without_importing_factory( + qualification_plan, workflow_row, monkeypatch +): + from identity import selection + + target = qualification_plan["catalog_snapshot"]["targets"][ + workflow_row["workflow_target_id"] + ] + payload = qualify.selection_payload(qualification_plan, workflow_row) + payload["config"]["discovery"] = { + "local_paths": [str(qualify.ROOT / "adapters/python/nooa")] + } + monkeypatch.setitem( + sys.modules, "nemo_fabric_adapters.nooa.targets.coding_agent", None + ) + selected = selection(payload) + assert selected["workflow"]["target_id"] == workflow_row["workflow_target_id"] + assert ( + selected["workflow"]["entrypoint"] == target["descriptor"]["spec"]["entrypoint"] + ) + assert selected["workflow"]["provenance"] + assert selected["admission_sha256"] == qualify.admission_fingerprint( + payload, workflow_row["catalog"]["descriptor"], target["descriptor"] + ) + + +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_registry_prerelease_keeps_python_and_npm_versions_distinct(qualification_plan): + plan = copy.deepcopy(qualification_plan) + base = plan["version"] + plan["mode"] = "registry" + plan["version"] = base + "rc1" + plan["catalog_snapshot"]["catalog_version"] = plan["version"] + for row in plan["adapters"]: + row["catalog"]["package"]["version"] = base + ( + "rc1" if row["catalog"]["ecosystem"] == "python" else "-rc.1" + ) + qualify.validate_plan(plan) + row = next( + row for row in plan["adapters"] if row["adapter_id"] == "nvidia.fabric.pi" + ) + assert f"nemo-fabric-adapters-pi@{base}-rc.1" in adapter_dockerfile( + plan, row, "base:exact" + ) + plan["mode"] = "source" + with pytest.raises(ValueError, match="unmodified catalog"): + qualify.validate_plan(plan) + + +@pytest.mark.parametrize( + "mutation", + [ + "missing", + "duplicate", + "descriptor", + "source", + "version", + "credential_url", + "untrusted_http", + "inline_secret", + "checkpoint", + "catalog_extra", + ], +) +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_plan_rejects_incomplete_or_unattributable_coverage( + qualification_plan, mutation +): + plan = copy.deepcopy(qualification_plan) + row = plan["adapters"][0] + if mutation == "missing": + plan["adapters"].pop() + elif mutation == "duplicate": + plan["adapters"].append(row) + elif mutation == "descriptor": + row["catalog"]["sha256"] = "0" * 64 + elif mutation == "source": + plan["source_sha"] = "0" * 40 + elif mutation == "version": + plan["version"] = "999.0.0" + elif mutation == "credential_url": + row["base_url"] = "https://user:secret@example.com/v1" + elif mutation == "untrusted_http": + row["base_url"] = "http://10.0.0.1:8000/v1" + elif mutation == "inline_secret": + row["harness_settings"] = {"config_overrides": {"api_key": "literal-secret"}} + elif mutation == "checkpoint": + row["model"] = "/mnt/checkpoint" + else: + plan["mode"] = "registry" + plan["catalog_snapshot"]["adapters"]["extra.adapter"] = copy.deepcopy( + row["catalog"] + ) + with pytest.raises(ValueError): + qualify.validate_plan(plan) + + +@pytest.mark.parametrize("outcome", ["blocked", "failed", "unsupported", "excluded"]) +def test_nonpasses_never_turn_full_catalog_green(tmp_path, qualification_plan, outcome): + report = { + "scope": "full_catalog", + "plan": qualification_plan, + "adapters": [ + {"adapter_id": "one", "outcome": "passed"}, + { + "adapter_id": "two", + "outcome": outcome, + "reason": "coverage gap", + }, + ], + } + qualify.write_report(tmp_path, report) + assert qualify.load(tmp_path / "report.json")["full_catalog_passed"] is False + assert "denominator: 2" in (tmp_path / "summary.md").read_text() + + +def test_subset_is_never_a_full_catalog_pass(tmp_path, qualification_plan): + report = { + "scope": "developer_subset", + "plan": qualification_plan, + "adapters": [{"adapter_id": "one", "outcome": "passed"}], + } + qualify.write_report(tmp_path, report) + assert report["all_harnesses_passed"] is True + assert report["full_catalog_passed"] is False + + +@pytest.mark.parametrize("state", ["completed", "running", "cancelled"]) +def test_full_catalog_pass_requires_completed_run(tmp_path, qualification_plan, state): + report = { + "scope": "full_catalog", + "state": state, + "plan": qualification_plan, + "adapters": [{"adapter_id": "one", "outcome": "passed"}], + } + qualify.write_report(tmp_path, report) + assert report["full_catalog_passed"] is (state == "completed") + assert "release_qualified" not in report + + +def test_credential_is_transported_as_reference_and_logs_are_redacted( + tmp_path, qualification_plan +): + os.environ["MODEL_API_KEY"] = "qualification-secret-value" + row = qualification_plan["adapters"][0] + command = qualify.harbor_command( + qualification_plan, row, "python", tmp_path, tmp_path, "trial" + ) + assert "MODEL_API_KEY=${MODEL_API_KEY}" in command + assert "qualification-secret-value" not in " ".join(command) + log = tmp_path / "run.log" + code = qualify.execute( + [sys.executable, "-c", "import os; print(os.environ['MODEL_API_KEY'])"], + log, + 10, + key_env="MODEL_API_KEY", + ) + assert code == 0 + assert "qualification-secret-value" not in log.read_text() + assert "[REDACTED]" in log.read_text() + assert "MODEL_API_KEY" not in qualify.process_env() + + +def test_timeout_terminates_process_and_retains_partial_log(tmp_path): + log = tmp_path / "timeout.log" + code = qualify.execute( + [sys.executable, "-u", "-c", "import time; print('started'); time.sleep(60)"], + log, + 1, + ) + assert code == 124 + assert "started" in log.read_text() + + +def test_interrupt_terminates_only_its_process_group(tmp_path): + with ( + patch("qualify.subprocess.Popen", autospec=True) as mock_popen, + patch("qualify.os.killpg") as kill, + ): + process = mock_popen.return_value + process.pid = 12345 + process.stdout = iter([]) + process.wait.side_effect = [KeyboardInterrupt, 0] + assert qualify.execute(["test-process"], tmp_path / "cancel.log", 10) == 130 + kill.assert_called_once_with(process.pid, qualify.signal.SIGINT) + + +def test_cleanup_does_not_remove_another_jobs_resources(tmp_path): + with ( + patch("qualify.subprocess.run") as run, + patch("qualify.subprocess.check_output") as output, + patch("qualify.execute") as execute, + ): + run.side_effect = [ + subprocess.CompletedProcess([], 0, stdout="probe"), + subprocess.CompletedProcess([], 0, stdout="owned unrelated"), + subprocess.CompletedProcess([], 0, stdout="owned unrelated"), + ] + output.side_effect = [ + json.dumps({"com.docker.compose.project": "fq-unique-0__trial"}), + json.dumps({"com.docker.compose.project": "another-user"}), + json.dumps({"com.docker.compose.project": "fq-unique-0__trial"}), + json.dumps({"com.docker.compose.project": "another-user"}), + ] + execute.return_value = 0 + assert qualify.cleanup("fq-unique", tmp_path / "cleanup.log") is True + assert [call.args[0] for call in execute.call_args_list] == [ + ["docker", "rm", "-f", "probe"], + ["docker", "rm", "-f", "owned"], + ["docker", "network", "rm", "owned"], + ] + + +@pytest.mark.parametrize( + "message", + [ + "Provider List: nvidia; ModuleNotFoundError: No module named 'mcp'", + "APIConnectionError: Nvidia_nimException - No module named 'mcp'", + ], +) +def test_missing_dependency_is_not_classified_as_a_provider_failure(message): + assert qualify.classify(message) == "install" + + +@pytest.fixture(name="qualification_job") +def qualification_job_fixture(tmp_path, qualification_plan): + from nemo_fabric.integrations.harbor.fabric_agent import build_harbor_config + + row = next( + row + for row in qualification_plan["adapters"] + if row["adapter_id"] == "nvidia.fabric.codex" + ) + trial = tmp_path / "trial" + (trial / "agent").mkdir(parents=True) + (trial / "verifier").mkdir() + qualify.save( + tmp_path / "result.json", + {"stats": {"n_completed_trials": 1, "n_errored_trials": 0}}, + ) + result = { + "status": "succeeded", + "adapter_id": row["adapter_id"], + "harness": "codex", + "agent_name": "qualification", + "adapter_kind": "python", + "runtime_id": "runtime", + "invocation_id": "invocation", + "request_id": "request", + "output": {"response": "fixed"}, + "events": [], + "metadata": {}, + "artifacts": {"artifacts": []}, + "telemetry": [], + "usage": { + "input_tokens": 12, + "input_tokens_include_cache": True, + "output_tokens": 3, + }, + "error": None, + } + qualify.save(trial / "agent/fabric-result-one.json", result) + spec = { + "config": build_harbor_config( + adapter_id=row["adapter_id"], + workspace="/app", + model_name=row["model"], + model_base_url=row["base_url"], + model_api_key_env=row["api_key_env"], + ).to_mapping() + } + spec["adapter_descriptor_sha256"] = qualify.admission_fingerprint( + spec, row["catalog"]["descriptor"] + ) + qualify.save(trial / "agent/fabric-run-one.json", spec) + (trial / "verifier/reward.txt").write_text("1\n") + identity = {"editable": [], "packages": {"nemo-fabric-runtime": "0.6.0"}} + qualify.save(trial / "verifier/qualification-identity.json", identity) + selection = { + "adapter_id": row["adapter_id"], + "python_prefix": "/usr/local", + "provenance": [ + { + "source": "installed_package", + "path": "/usr/local/share/nemo-fabric/adapters/codex/codex.fabric-adapter.json", + } + ], + } + qualify.save(trial / "verifier/qualification-selection.json", selection) + qualify.save( + trial / "result.json", + { + "agent_result": { + "n_input_tokens": 12, + "n_output_tokens": 3, + "metadata": { + "fabric": { + "adapter_id": row["adapter_id"], + "invocation_id": "invocation", + } + }, + } + }, + ) + return tmp_path, row, identity, result, selection + + +@pytest.mark.parametrize( + "mutation,category", + [ + ("none", None), + ("reward", None), + ("missing_reward", "harbor_verifier"), + ("invalid_reward", "harbor_verifier"), + ("nonfinite_reward", "harbor_verifier"), + ("identity", "adapter_admission"), + ("usage", None), + ("counter", "usage"), + ("source_origin", "adapter_admission"), + ("fingerprint", "adapter_admission"), + ("missing_result", "transport_runner"), + ("runner_error", "adapter_admission"), + ], +) +@pytest.mark.usefixtures("requires_harbor") +def test_result_checks_remain_independent(qualification_job, mutation, category): + job, row, identity, result, selection = qualification_job + if mutation == "reward": + (job / "trial/verifier/reward.txt").write_text("0") + elif mutation == "missing_reward": + (job / "trial/verifier/reward.txt").unlink() + elif mutation == "invalid_reward": + (job / "trial/verifier/reward.txt").write_text("invalid") + elif mutation == "nonfinite_reward": + (job / "trial/verifier/reward.txt").write_text("nan") + elif mutation == "identity": + identity = {"editable": ["runtime"]} + elif mutation == "usage": + result["usage"] = None + qualify.save(job / "trial/agent/fabric-result-one.json", result) + trial = qualify.load(job / "trial/result.json") + trial["agent_result"]["n_input_tokens"] = None + trial["agent_result"]["n_output_tokens"] = None + qualify.save(job / "trial/result.json", trial) + elif mutation == "counter": + trial = qualify.load(job / "trial/result.json") + trial["agent_result"]["n_input_tokens"] = 999 + qualify.save(job / "trial/result.json", trial) + elif mutation == "source_origin": + observed = copy.deepcopy(selection) + observed["provenance"][0] = { + "source": "bundled", + "path": "/src/adapters/python/codex/codex.fabric-adapter.json", + } + qualify.save(job / "trial/verifier/qualification-selection.json", observed) + elif mutation == "fingerprint": + spec = qualify.load(job / "trial/agent/fabric-run-one.json") + spec["adapter_descriptor_sha256"] = "wrong" + qualify.save(job / "trial/agent/fabric-run-one.json", spec) + elif mutation == "missing_result": + (job / "trial/agent/fabric-result-one.json").unlink() + elif mutation == "runner_error": + qualify.save( + job / "trial/agent/fabric-result-one.json", + {"runner_error": {"message": "descriptor fingerprint mismatch"}}, + ) + summary = qualify.inspect_results(job, row, identity, selection) + assert {failure["category"] for failure in summary["failures"]} == ( + {category} if category else set() + ) + + +@pytest.mark.parametrize("known_usage", [True, False]) +@pytest.mark.parametrize("transport_matches", [True, False]) +@pytest.mark.usefixtures("requires_harbor") +def test_task_failure_and_usage_limit_do_not_mask_transport_failure( + qualification_job, qualification_plan, known_usage, transport_matches +): + job, row, identity, result, selection = qualification_job + (job / "trial/verifier/reward.txt").write_text("0") + if not known_usage: + result["usage"] = None + qualify.save(job / "trial/agent/fabric-result-one.json", result) + trial = qualify.load(job / "trial/result.json") + trial["agent_result"]["n_input_tokens"] = 12 if known_usage else None + trial["agent_result"]["n_output_tokens"] = 3 if known_usage else None + if not transport_matches: + trial["agent_result"]["n_input_tokens"] = 999 + qualify.save(job / "trial/result.json", trial) + summary = qualify.inspect_results(job, row, identity, selection) + assert summary["checks"]["task_success"] is False + assert summary["checks"]["verifier_completed"] is True + assert summary["checks"]["usage_available"] is known_usage + assert bool(summary["limitations"]) is not known_usage + assert {f["category"] for f in summary["failures"]} == ( + set() if transport_matches else {"usage"} + ) + summary.update( + adapter_id=row["adapter_id"], + outcome="failed" if summary["failures"] else "passed", + ) + report = { + "scope": "developer_subset", + "plan": qualification_plan, + "adapters": [summary], + } + qualify.write_report(job, report) + assert report["all_harnesses_passed"] is transport_matches + assert report["full_catalog_passed"] is False + text = (job / "summary.md").read_text() + assert "Task reward" in text + assert "| 0.0 |" in text + + +@pytest.mark.usefixtures("requires_harbor") +def test_admission_compares_normalized_descriptor_not_raw_catalog_hash( + qualification_job, +): + job, row, identity, _, selection = qualification_job + row["catalog"]["sha256"] = "0" * 64 + summary = qualify.inspect_results(job, row, identity, selection) + assert summary["checks"]["descriptor_fingerprint"] is True + assert summary["failures"] == [] + + +def test_source_task_uses_packed_installs_and_exact_runtime(qualification_plan): + row = next( + row + for row in qualification_plan["adapters"] + if row["adapter_id"] == "nvidia.fabric.pi" + ) + dockerfile = adapter_dockerfile(qualification_plan, row, "qualification-base:exact") + assert "npm pack" in dockerfile + assert "npm install --prefix /opt/harness /packs/*.tgz" in dockerfile + assert "@earendil-works/pi-ai@1.0.3" in dockerfile + assert "-e /src" not in dockerfile + + +def test_droid_install_uses_pinned_native_cli_and_sdk(qualification_plan): + row = next( + r + for r in qualification_plan["adapters"] + if r["adapter_id"] == "nvidia.fabric.droid" + ) + dockerfile = adapter_dockerfile(qualification_plan, row, "qualification-base:exact") + assert "npm install --global droid@0.233.0" in dockerfile + assert "@factory/droid-sdk@0.9.1" in dockerfile + + +@pytest.mark.usefixtures("clean_qualification_checkout") +@pytest.mark.parametrize("mode", ["source", "registry"]) +def test_openhands_install_uses_the_documented_compatible_sdk_stack( + qualification_plan, mode +): + qualification_plan["mode"] = mode + row = next( + row + for row in qualification_plan["adapters"] + if row["adapter_id"] == "nvidia.fabric.openhands" + ) + dockerfile = adapter_dockerfile(qualification_plan, row, "qualification-base:exact") + assert ( + 'pip install "openhands-sdk==1.50.0" "openhands-tools==1.50.0" ' + '"litellm==1.100.1" "mcp==1.28.1" && pip check' + ) in dockerfile + assert '"grpcio==' not in dockerfile + assert '"protobuf==' not in dockerfile + + +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_failed_host_environment_preserves_complete_report( + tmp_path, qualification_plan +): + output = tmp_path / "report" + assert ( + qualify.run( + qualification_plan, {"plan": qualification_plan, "adapters": {}}, output, [] + ) + == 1 + ) + report = qualify.load(output / "report.json") + assert len(report["adapters"]) == len(qualification_plan["adapters"]) + for row, planned in zip(report["adapters"], qualification_plan["adapters"]): + if planned["action"] == "run": + assert row["outcome"] == "blocked" + assert row["reason"].startswith("Host installation unavailable") + else: + assert row["outcome"] == planned["action"] + assert row["reason"] == planned["reason"] + assert report["full_catalog_passed"] is False + + +@pytest.mark.usefixtures("clean_qualification_checkout") +@pytest.mark.parametrize("key_present", [False, True]) +def test_kiro_missing_native_key_skips_before_task_install_or_launch( + tmp_path, qualification_plan, monkeypatch, key_present +): + row = next( + r + for r in qualification_plan["adapters"] + if r["adapter_id"] == "nvidia.fabric.kiro" + ) + row.update(action="run", reason="", model="kiro/advertised-model", base_url=None) + monkeypatch.setenv("MODEL_API_KEY", "dummy-local-model-key") + if key_present: + monkeypatch.setenv("KIRO_API_KEY", "native-test-key") + else: + monkeypatch.delenv("KIRO_API_KEY", raising=False) + identity = { + "editable": [], + "packages": { + "nemo-fabric-runtime": qualification_plan["version"], + "harbor": qualification_plan["harbor_version"], + }, + "catalog_sha256": qualify.fingerprint(qualify.plan_catalog(qualification_plan)), + "wheel_sha256": {"nemo-fabric-runtime": "test-wheel-hash"}, + } + environment = { + "plan": qualification_plan, + "python": sys.executable, + "wheel_sha256": {"nemo_fabric_runtime-test.whl": "test-wheel-hash"}, + "adapters": {}, + } + monkeypatch.setattr(qualify, "execute", lambda *args, **kwargs: 0) + real_check_output = qualify.subprocess.check_output + + def check_output(command, *args, **kwargs): + if str(command[-1]).endswith("identity.py"): + return json.dumps(identity) + return real_check_output(command, *args, **kwargs) + + monkeypatch.setattr(qualify.subprocess, "check_output", check_output) + output = tmp_path / "report" + assert ( + qualify.run(qualification_plan, environment, output, [row["adapter_id"]]) == 1 + ) + report = qualify.load(output / "report.json") + kiro = next(r for r in report["adapters"] if r["adapter_id"] == row["adapter_id"]) + assert kiro["outcome"] == ("failed" if key_present else "blocked") + assert kiro["failures"][0]["category"] == ("install" if key_present else "auth") + if not key_present: + assert kiro["reason"] == "Missing credential environment variable KIRO_API_KEY" + assert len(report["adapters"]) == len(qualification_plan["adapters"]) + assert report["full_catalog_passed"] is False + assert not (output / str(qualification_plan["adapters"].index(row))).exists() + + +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_single_task_is_pinned_and_dataset_roots_are_rejected( + tmp_path, qualification_plan +): + with pytest.raises(ValueError, match="single Harbor task missing"): + qualify.task_identity(tmp_path) + task = tmp_path / "task" + qualify.stage_task(qualification_plan, task, "prepared:exact") + wrapper = (task / "tests/test.sh").read_text() + assert wrapper.index("qualification-identity.json") < wrapper.index( + "bash /tests/upstream-test.sh" + ) + assert not (task / "solution").exists() + assert "docker_image" not in (task / "task.toml").read_text() + assert (task / "environment/Dockerfile").read_text() == "FROM prepared:exact\n" + assert (task / "tests/upstream-test.sh").read_bytes() == ( + Path(qualification_plan["task"]["path"]) / "tests/test.sh" + ).read_bytes() + changed = copy.deepcopy(qualification_plan) + changed["task"]["sha256"] = "wrong" + with pytest.raises(ValueError, match="task changed"): + qualify.validate_plan(changed) + + +@pytest.mark.parametrize("changed", ["nemo-fabric-runtime", "openhands-sdk"]) +def test_task_dependencies_can_change_but_fabric_and_harness_identity_cannot(changed): + expected = { + "editable": [], + "packages": { + "nemo-fabric-runtime": "0.6.0", + "openhands-sdk": "1.50.0", + "protobuf": "6.33.0", + }, + } + observed = copy.deepcopy(expected) + observed["packages"]["grpcio-tools"] = "1.73.0" + assert qualify.unchanged_install(expected, observed) + observed["packages"]["protobuf"] = "5.0.0" + assert qualify.unchanged_install(expected, observed) + observed["packages"][changed] = "different" + assert not qualify.unchanged_install(expected, observed) + + +@pytest.mark.parametrize( + "mutation", [None, "fabric_editable", "dirty", "revision", "unrecorded"] +) +def test_pinned_hermes_source_install_is_allowed_without_editable_fabric(mutation): + revision = "a" * 40 + expected = { + "editable": ["hermes-agent"], + "editable_sources": {"hermes-agent": {"revision": revision, "dirty": False}}, + "harness_sources": {"hermes-agent": revision}, + "packages": {"nemo-fabric-runtime": "0.6.0", "hermes-agent": "0.0.0"}, + } + observed = copy.deepcopy(expected) + if mutation == "fabric_editable": + observed["editable"].append("nemo-fabric-runtime") + elif mutation == "dirty": + observed["editable_sources"]["hermes-agent"]["dirty"] = True + elif mutation == "revision": + observed["editable_sources"]["hermes-agent"]["revision"] = "b" * 40 + elif mutation == "unrecorded": + observed["harness_sources"] = {} + assert qualify.unchanged_install(expected, observed) is (mutation is None) + + +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_task_interpreter_reaches_runner_and_identity_without_rewriting_verifier( + qualification_plan, tmp_path +): + row = next( + r + for r in qualification_plan["adapters"] + if r["adapter_id"] == "nvidia.fabric.hermes" + ) + row["task_python"] = "/opt/hermes-runtime/bin/python" + qualify.validate_plan(qualification_plan) + command = qualify.harbor_command( + qualification_plan, row, "host-python", tmp_path, tmp_path, "trial" + ) + assert "fabric_python=/opt/hermes-runtime/bin/python" in command + task = tmp_path / "task" + qualify.stage_task(qualification_plan, task, "prepared:exact", row["task_python"]) + wrapper = (task / "tests/test.sh").read_text() + assert "/opt/hermes-runtime/bin/python /qualification/identity.py" in wrapper + assert "bash /tests/upstream-test.sh" in wrapper + assert (task / "tests/upstream-test.sh").read_bytes() == ( + Path(qualification_plan["task"]["path"]) / "tests/test.sh" + ).read_bytes() + + +def test_dirty_checkout_is_rejected(tmp_path, qualification_plan): + repository = tmp_path / "repo" + repository.mkdir() + subprocess.run(["git", "init", "-q", str(repository)], check=True) + (repository / "crates").mkdir() + probe = repository / "crates/probe" + probe.write_text("committed\n") + subprocess.run(["git", "add", "crates/probe"], cwd=repository, check=True) + subprocess.run( + [ + "git", + "-c", + "user.name=Qualification test", + "-c", + "user.email=test@example.invalid", + "-c", + "commit.gpgsign=false", + "-c", + "core.hooksPath=/dev/null", + "commit", + "-q", + "-m", + "fixture", + ], + cwd=repository, + check=True, + ) + plan = copy.deepcopy(qualification_plan) + plan["source_sha"] = subprocess.check_output( + ["git", "rev-parse", "HEAD"], cwd=repository, text=True + ).strip() + probe.write_text("dirty\n") + catalog = qualify.ROOT / qualify.CATALOG + with ( + patch.object(qualify, "ROOT", repository), + patch.object(qualify, "CATALOG", catalog), + ): + with pytest.raises(subprocess.CalledProcessError) as error: + qualify.validate_plan(plan) + assert error.value.cmd[:2] == ["git", "diff"] + + +def test_report_redaction_preserves_types_and_custom_credentials( + tmp_path, qualification_plan +): + os.environ["MODEL_API_KEY"] = "true" + os.environ["AWS_ACCESS_KEY_ID"] = "fake-access-key-id" + os.environ["AZURE_STORAGE_CONNECTION_STRING"] = "fake-connection-string" + os.environ["BROKER_CREDENTIAL_HANDLE"] = "opaque-custom-secret" + os.environ["DOCKER_AUTH_CONFIG"] = ( + '{"auths":{"registry":{"auth":"encoded-secret"}}}' + ) + report = { + "scope": "full_catalog", + "state": "completed", + "plan": qualification_plan, + "adapters": [ + { + "adapter_id": "one", + "outcome": "passed", + "selected": {"api_key_env": "BROKER_CREDENTIAL_HANDLE"}, + "reason": "opaque-custom-secret fake-access-key-id fake-connection-string " + + os.environ["DOCKER_AUTH_CONFIG"], + "nested": [True, False, None, 1, "true"], + } + ], + } + qualify.write_report(tmp_path, report) + text = (tmp_path / "report.json").read_text() + redacted = json.loads(text) + assert redacted["full_catalog_passed"] is True + assert redacted["adapters"][0]["nested"] == [True, False, None, 1, "[REDACTED]"] + assert "fake-access-key-id" not in text + assert "fake-connection-string" not in text + assert "opaque-custom-secret" not in text + assert os.environ["DOCKER_AUTH_CONFIG"] not in text + assert "opaque-custom-secret" not in (tmp_path / "summary.md").read_text() + + +def test_custom_credential_is_redacted_before_log_write(tmp_path): + os.environ["BROKER_CREDENTIAL_HANDLE"] = "opaque-custom-secret" + log = tmp_path / "process.log" + assert ( + qualify.execute( + [ + sys.executable, + "-c", + "import os; print(os.environ['BROKER_CREDENTIAL_HANDLE'])", + ], + log, + 10, + key_env="BROKER_CREDENTIAL_HANDLE", + ) + == 0 + ) + assert "opaque-custom-secret" not in log.read_text() + assert "[REDACTED]" in log.read_text() + + +@pytest.mark.parametrize( + "text,category", + [ + ("status=401", "auth"), + ("HTTP 403", "auth"), + ("HTTP 404", "provider_model"), + ("HTTP 429", "infrastructure"), + ("HTTP 502", "infrastructure"), + ("HTTP 503", "infrastructure"), + ("run-1401-1403-1404-1429-1502-1503", "transport_runner"), + ("id_401_403_404_429_502_503", "transport_runner"), + ], +) +def test_http_codes_do_not_match_inside_identifiers(text, category): + assert qualify.classify(text) == category + + +@pytest.mark.parametrize("suffix", [".json", ".jsonl"]) +def test_artifact_redaction_preserves_machine_readable_evidence(suffix): + os.environ["MODEL_API_KEY"] = "true" + original = {"flag": True, "value": None, "credential": "true"} + text = json.dumps(original) + "\n" + if suffix == ".jsonl": + text += json.dumps({"flag": False, "count": 1}) + "\n" + result = qualify.sanitize_artifact(text, suffix, ("MODEL_API_KEY",)) + objects = [json.loads(line) for line in result.splitlines()] + assert objects[0] == {"flag": True, "value": None, "credential": "[REDACTED]"} + if suffix == ".jsonl": + assert objects[1] == {"flag": False, "count": 1} + os.environ["MODEL_API_KEY"] = "different-secret" + assert qualify.sanitize_artifact(text, suffix, ("MODEL_API_KEY",)) == text + + +@pytest.mark.parametrize("suffix", [".jsonl", ".ndjson"]) +def test_jsonl_redaction_keeps_unicode_separators_inside_records(suffix): + secret = "prefix\u2028suffix" + os.environ["MODEL_API_KEY"] = secret + records = [{"credential": secret, "flag": True}, {"text": "other\u2028text"}] + original = "\n".join(json.dumps(r, ensure_ascii=False) for r in records) + "\n" + cleaned = qualify.sanitize_artifact(original, suffix, ("MODEL_API_KEY",)) + assert secret not in cleaned + assert cleaned.endswith("\n") + parsed = [json.loads(line) for line in cleaned.split("\n") if line] + assert parsed == [{"credential": "[REDACTED]", "flag": True}, records[1]] + + +@pytest.mark.usefixtures("clean_qualification_checkout") +@pytest.mark.parametrize("provider", ["", " ", " openai", "openai ", 1]) +def test_provider_override_rejects_invalid_values(qualification_plan, provider): + qualification_plan["adapters"][0]["model_provider"] = provider + with pytest.raises(ValueError, match="model_provider must be"): + qualify.validate_plan(qualification_plan) + + +@pytest.mark.usefixtures("clean_qualification_checkout") +@pytest.mark.parametrize("max_turns", [0, -1, True, 1.5, "10"]) +def test_rejects_invalid_turn_bound(qualification_plan, max_turns): + qualification_plan["adapters"][0]["max_turns"] = max_turns + with pytest.raises(ValueError, match="max_turns must be"): + qualify.validate_plan(qualification_plan) + + +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_model_settings_require_sdk_and_reject_credentials(qualification_plan): + row = qualification_plan["adapters"][0] + row["model_settings"] = {"cost_tracking": "disabled"} + with pytest.raises(ValueError, match="model_settings requires"): + qualify.validate_plan(qualification_plan) + qualification_plan["bridge"] = "sdk" + row["model_settings"] = {"api_key": "credential"} + with pytest.raises(ValueError, match="credentials belong"): + qualify.validate_plan(qualification_plan) + + +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_model_settings_and_bound_match_command_selection_and_report( + qualification_plan, tmp_path +): + qualification_plan["bridge"] = "sdk" + row = next( + r + for r in qualification_plan["adapters"] + if r["adapter_id"] == "nvidia.fabric.mini-swe-agent" + ) + row.update(model_settings={"cost_tracking": "disabled"}, max_turns=100) + qualify.validate_plan(qualification_plan) + command = qualify.harbor_command( + qualification_plan, row, "python", tmp_path, tmp_path, "trial" + ) + assert 'fabric_model_settings={"cost_tracking": "disabled"}' in command + assert "fabric_max_turns=100" in command + payload = qualify.selection_payload(qualification_plan, row) + assert payload["config"]["models"]["default"]["settings"] == row["model_settings"] + assert payload["config"]["runtime"]["max_turns"] == 100 + assert ( + qualify.run( + qualification_plan, + {"plan": qualification_plan}, + tmp_path / "out", + [row["adapter_id"]], + ) + == 1 + ) + report = qualify.load(tmp_path / "out/report.json") + selected = next( + r["selected"] + for r in report["adapters"] + if r["adapter_id"] == row["adapter_id"] + ) + assert selected["model_settings"] == row["model_settings"] + assert selected["max_turns"] == 100 + + +@pytest.mark.usefixtures("clean_qualification_checkout") +def test_provider_override_preserves_model_in_command_selection_and_report( + qualification_plan, tmp_path, monkeypatch +): + from nemo_fabric.integrations.harbor import fabric_agent + + row = next( + row + for row in qualification_plan["adapters"] + if row["adapter_id"] == "nvidia.fabric.qwen" + ) + row["model_provider"] = "openai" + model = row["model"] + command = qualify.harbor_command( + qualification_plan, row, "python", tmp_path, tmp_path, "trial" + ) + assert command[command.index("--model") + 1] == model + assert "fabric_model_provider=openai" in command + + original = fabric_agent.build_harbor_config + received = {} + + def compatible_bridge(**kwargs): + # Emulate the optional argument while keeping this gate usable with older SDKs. + received.update(kwargs) + provider = kwargs.pop("model_provider_name") + config = original(**kwargs) + config.models["default"].provider = provider + return config + + monkeypatch.setattr(fabric_agent, "build_harbor_config", compatible_bridge) + payload = qualify.selection_payload(qualification_plan, row) + assert received["model_provider_name"] == "openai" + assert payload["config"]["models"]["default"]["provider"] == "openai" + assert payload["config"]["models"]["default"]["model"] == model + + output = tmp_path / "qualification" + assert ( + qualify.run( + qualification_plan, + {"plan": qualification_plan}, + output, + [row["adapter_id"]], + ) + == 1 + ) + report = qualify.load(output / "report.json") + selected = next( + item for item in report["adapters"] if item["adapter_id"] == row["adapter_id"] + )["selected"] + assert selected["provider"] == "openai" + assert selected["model"] == model diff --git a/uv.lock b/uv.lock index a3bfd771d..6dd303aea 100644 --- a/uv.lock +++ b/uv.lock @@ -2440,6 +2440,8 @@ dependencies = [ [package.optional-dependencies] harness = [ { name = "fsspec" }, + { name = "litellm", version = "1.89.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "litellm", version = "1.100.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, { name = "mini-swe-agent" }, ] relay = [ @@ -2450,6 +2452,8 @@ relay = [ requires-dist = [ { name = "fsspec", marker = "extra == 'full'", specifier = ">=2026.6.0" }, { name = "fsspec", marker = "extra == 'harness'", specifier = ">=2026.6.0" }, + { name = "litellm", marker = "extra == 'full'", specifier = "<1.106" }, + { name = "litellm", marker = "extra == 'harness'", specifier = "<1.106" }, { name = "mini-swe-agent", marker = "extra == 'full'", specifier = ">=2.0,<3" }, { name = "mini-swe-agent", marker = "extra == 'harness'", specifier = ">=2.0,<3" }, { name = "nemo-fabric-adapter-contract", editable = "adapter-contract/python" }, @@ -2471,6 +2475,7 @@ dependencies = [ [package.optional-dependencies] full = [ { name = "fsspec" }, + { name = "litellm", version = "1.100.1", source = { registry = "https://pypi.org/simple" } }, { name = "nemo-relay" }, { name = "nooa" }, { name = "nooa-bench" }, @@ -2478,6 +2483,7 @@ full = [ ] harness = [ { name = "fsspec" }, + { name = "litellm", version = "1.100.1", source = { registry = "https://pypi.org/simple" } }, { name = "nooa" }, { name = "nooa-bench" }, { name = "nooa-cli" }, @@ -2487,6 +2493,8 @@ harness = [ requires-dist = [ { name = "fsspec", marker = "extra == 'full'", specifier = ">=2026.6.0" }, { name = "fsspec", marker = "extra == 'harness'", specifier = ">=2026.6.0" }, + { name = "litellm", marker = "extra == 'full'", specifier = "<1.106" }, + { name = "litellm", marker = "extra == 'harness'", specifier = "<1.106" }, { name = "nemo-fabric-adapter-contract", editable = "adapter-contract/python" }, { name = "nemo-fabric-adapters-common", editable = "adapters/python/common" }, { name = "nemo-relay", marker = "extra == 'full'", specifier = ">=0.11,<0.12" },