feat(tier3): support Claude Code on Agent Platform and add active preflight probe by kweinmeister · Pull Request #140 · NVIDIA/SkillEvaluator

kweinmeister · 2026-09-13T23:09:06Z

Summary

Adds support for routing Claude Code through Google Cloud Agent Platform (Vertex AI) endpoints, hardens Harbor GKE runtime execution boundaries, and introduces an active 1-token preflight probe for Agent Platform OpenAPI endpoints.

Why this change is needed:

When evaluating skills with Claude Code via Harbor, users need to route requests to Google Cloud Vertex AI Anthropic endpoints (CLAUDE_CODE_USE_VERTEX=1) and authenticate via Application Default Credentials (ADC).
Harbor GKE execution mode (--env-mode gke) requires enforcing security boundaries to reject cluster infrastructure settings in skill configs in favor of host CLI flags/environment variables, while ensuring credentials passed in --ek keys/values are redacted from process listings without dropping rate limits or token configs.
Endpoint validation previously lacked active connectivity checks for Vertex AI OpenAPI endpoints, leading to delayed failures during evaluation runs. The new active 1-token chat/completions probe verifies credentials, permissions, and model availability upfront and fails fast on 401/403/404.

Verification

I am familiar with the Contributing Guidelines
Added or updated focused tests
Updated documentation for user-visible changes
Ran make lint
Ran make test
Ran make build
Did not add credentials, private datasets, or proprietary benchmark content

Release Impact

No user-visible release note needed
Updated CHANGELOG.md

Add Vertex AI Anthropic endpoint routing and ADC authentication for Claude Code. Enforce security boundaries rejecting cluster infrastructure kwargs in skill configs. Harden sensitive credential redaction across CLI, environment, and process args. Support task.path resolution and stdio MCP env forwarding in local agents. Signed-off-by: Karl Weinmeister <kweinmeister@google.com>

Implement an active 1-token preflight probe against Vertex AI Agent Platform OpenAPI endpoints to verify credentials, permissions, and model availability with minimal latency. - Detect Vertex AI OpenAPI base URLs and extract project/location metadata. - Send a 1-token chat/completions probe using HTTPStatus enum values. - Fall back to Google Cloud ADC when an explicit API key is not supplied. - Classify successful probes as VERIFIED and fail fast as FATAL on 401/403/404. - Add unit tests in test_harbor_runtime_preflight covering 200, 401, 403, 404, 429, timeouts, and redirects. Signed-off-by: Karl Weinmeister <kweinmeister@google.com>

rng1995 · 2026-09-16T03:42:02Z

+    return ProviderConfig(
+        provider=provider,
+        model=model,
+        api_key=token,


[P1] Refresh ADC credentials in the process that performs grading. This creates a one-time access-token snapshot. Tier 3 later copies it into OPENAI_API_KEY and stages it into the Harbor verifier, whose standalone template has no ADC acquisition or refresh path. A multi-agent or multi-attempt run that outlives the token will begin failing grading after a successful preflight. Acquire and refresh ADC in the verifier/pod (or issue a fresh credential per bounded job) and add an expiry/retry integration test for the Harbor path.

Implemented. I added an in-process Google ADC refresh handler in templates/eval.py that intercepts 401 Unauthorized errors during grading and refreshes the token on the fly, with integration tests covering expiry and retry.

@kweinmeister This is still incomplete outside GKE, so I am reopening the thread. The verifier now tries ADC discovery after a 401, but the generated Docker runtime receives a token snapshot without an ADC credential source or gcloud configuration. Runtime plans reuse that snapshot across the evaluation matrix; the new test mocks token discovery and does not verify this handoff.

There is also a concrete overwrite before launch: with provider.api_key="expired-token" and host refresh returning "fresh-token", _resolve_agent_runtime_plan still puts "expired-token" in OPENAI_API_KEY for OpenCode/Codex. _agent_credentials reads the original provider snapshot, then subprocess_env.update(credentials) overwrites the fresh token.

Please preserve the refreshed credential consistently and provide a working refresh handoff for supported runtimes, or reissue credentials for bounded jobs. Add regressions through the real runtime plan and staged verifier, including token expiry.

Addressed in 9d1793c:

Token Overwrite: In _resolve_agent_runtime_plan, we now replace provider.api_key with the refreshed token (effective_provider = replace(provider, api_key=refreshed_key)). This ensures _agent_credentials(), _harbor_subprocess_environment(), and _agent_provider_config() all receive and preserve the refreshed token rather than reading the stale snapshot and overwriting it.

Job Lifecycle Outside GKE: In Docker environments without Workload Identity or container gcloud access (and where mounting host credentials into untrusted evaluation containers would violate isolation boundaries), _run_harbor() now reissues a fresh host-side ADC access token immediately prior to building the Harbor command and executing bounded jobs for Vertex endpoints.

Regression Tests: Added test_resolve_agent_runtime_plan_preserves_refreshed_adc_token (asserting OPENAI_API_KEY retains the fresh token), test_run_harbor_reissues_adc_token_for_bounded_jobs (asserting host reissuance before Harbor dispatch), and updated the staged verifier tests to exercise the actual GKE metadata server discovery and gcloud fallback paths without monkeypatching _get_vertex_access_token.

- Address PR review feedback for GKE execution and Vertex AI routing. - Decouple live agent Claude/Vertex provider routing from evaluator configs. - Support Google Application Default Credentials (ADC) for Vertex AI OpenAPI endpoints with automatic 401 token refresh in LLMClient. - Harden CLI environment kwargs parsing to fail fast on credentials and malformed input. - Expand GKE infrastructure kwargs blocklist in evals config and improve credential redaction precision. - Normalize multi-path KUBECONFIG resolution and ensure path resolution consistency. - Add packaging dependency guards and comprehensive unit and regression tests. Signed-off-by: Karl Weinmeister <kweinmeister@google.com>

# Conflicts: # CHANGELOG.md

… handling - Enforce safe allowlist for Harbor environment kwargs in skill evals config. - Sanitize runner constructor kwargs and expand sensitive key detection to prevent credential leaks in CLI argv. - Maintain fatal preflight checks for Vertex OpenAPI judge while gracefully handling Claude Vertex Workload Identity. - Implement in-process access token refresh for Google ADC on 401 Unauthorized responses. - Cache and validate multi-path KUBECONFIG files via KubeConfigMerger with fail-fast error handling. - Replace external kubectl process checks with direct Kubernetes API health check. - Inject OPENAI_API_KEY into SkillSpector child process environment when using Vertex OpenAPI with ADC. Signed-off-by: Karl Weinmeister <kweinmeister@google.com>

rng1995 · 2026-09-16T22:43:34Z

@kweinmeister - Please resolve and reply to all review comments and let us know when the PR is ready for re-review. Thanks!

kweinmeister · 2026-09-17T00:53:24Z

@kweinmeister - Please resolve and reply to all review comments and let us know when the PR is ready for re-review. Thanks!

@rng1995 All of the comments have been addressed and replied to. Thanks in advance for another review!

rng1995 · 2026-09-17T20:27:36Z

@chrisknvidia - Please review this PR.

rng1995

Re-reviewed the latest head against every inline finding. All 15 substantive review comments are addressed in the implementation, including the skill/config and argv allowlists, GKE kubeconfig merge, Vertex/ADC routing and refresh, fatal OpenAPI auth handling, and Tier 3 dependency packaging. The full GitHub check suite is green; an independent targeted run passed 594 tests and Ruff. The head also merges cleanly with current main.

rng1995

@kweinmeister Please address the reproduced KUBECONFIG relative-file regression in the linked review thread before re-review. The current merge output breaks valid certificate and key references after relocating the merged YAML.

chrisknvidia

Security review found two credential-isolation issues.

Signed-off-by: Karl Weinmeister <kweinmeister@google.com>

… KUBECONFIG path merging Signed-off-by: Karl Weinmeister <kweinmeister@google.com>

kweinmeister · 2026-09-19T11:47:09Z

Addressed feedback, ready for re-review. Also ran a successful eval on a live GKE cluster with the latest code. Thanks!

rng1995

@kweinmeister I verified the fixes for 16 of the 17 earlier concerns, including the constructor allowlists, explicit Workload Identity opt-in, kubeconfig path handling, and the original credential-isolation issues. The ADC lifetime concern remains incomplete; I have reopened that thread with the runtime-plan reproduction. Two additional configuration regressions are inline: benign MCP runtime variables are rejected, and the new generic OpenAI key fallback is not mapped for SkillSpector.

Local validation: 1,567 focused tests passed after installing the pinned Kubernetes dependency in an isolated test directory and allowing localhost integration fixtures. Separate reproductions exposed the token overwrite and both MCP validation paths; the SkillSpector environment mapping was checked independently. No live GKE/Vertex evaluation was performed.

GitHub currently reports no check runs or commit statuses for 3b798a1, so CI on this head is also unverified. Please address these blockers and obtain passing required checks before approval.

Limit the Gitleaks exception to the exact synthetic token, test path, and generic-api-key rule. This keeps the credential rejection regression test covered while allowing history scans of the PR to pass. Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>

…ite and MCP runtime env propagation - Prevent token overwrite in _resolve_agent_runtime_plan by propagating refreshed credentials via effective_provider - Reissue host ADC access token for Vertex endpoints before executing bounded Harbor jobs - Propagate harbor.runtime_env through _load_mcp_servers, _validate_mcp_toml, and SkillEvaluatorClaudeCode task MCP configuration - Ensure child_environment() exports OPENAI_API_KEY for OpenAI providers - Add comprehensive regression tests across runner, GKE mode, and local agent suites Signed-off-by: Karl Weinmeister <kweinmeister@google.com>

kweinmeister requested review from chrisknvidia, mosheabr, rmalani-nv and rng1995 as code owners September 13, 2026 23:09

kweinmeister changed the title ~~feat(tier3): support Claude Code on Vertex AI and add active preflight probe~~ feat(tier3): support Claude Code on Agent Platform and add active preflight probe Sep 13, 2026

kweinmeister force-pushed the fix/harbor-claude-vertex-routing branch from 300fcbb to 8fcc379 Compare September 14, 2026 20:02

kweinmeister added 2 commits September 14, 2026 15:17

kweinmeister force-pushed the fix/harbor-claude-vertex-routing branch from 8fcc379 to fea99b9 Compare September 14, 2026 20:18