feat(tier3): enforce structured JSON schemas and rate-limit retries in LLM judges

[kweinmeister]

Summary

When running Tier 3 evaluations with higher concurrency or across multi-case datasets, LLM judge grading (accuracy, goal_accuracy, and behavior_check) can fail in two ways:

1. Judge models with internal reasoning tokens or verbose output can emit preamble text or exhaust their output token budget before closing the JSON object, causing _parse_json_response to fail even with max_tokens=4096.
2. Parallel Harbor trial containers hitting the same judge endpoint at the end of a wave can trigger HTTP 429 rate limits or transient 502/503/504 gateway errors.

This change updates both the host judge path (src/skillevaluator/inference/client.py, src/skillevaluator/tier3/eval_core/llm_judge.py) and the container verifier (src/skillevaluator/tier3/harbor/templates/eval.py) to:

• Pass explicit response_schema and schema_name definitions (ACCURACY_JUDGE_SCHEMA, GOAL_ACCURACY_JUDGE_SCHEMA, BEHAVIOR_CHECK_JUDGE_SCHEMA) to the LLM client (response_format for OpenAI-compatible endpoints and output_config for Anthropic endpoints).
• Automatically downgrade once without the schema constraint (response_format={"type": "json_object"} for OpenAI-compatible, or prompt-based JSON for Anthropic) if a provider returns HTTP 400 or 422, logging a warning and memoizing the target so subsequent judge calls do not repeat failed schema requests.
• Retry HTTP 429 and transient 5xx/connection errors using exponential backoff with full jitter and Retry-After header parsing, configurable via SKILL_EVAL_LLM_MAX_RETRIES, SKILL_EVAL_LLM_RETRY_BASE_DELAY, and SKILL_EVAL_LLM_RETRY_MAX_DELAY.

Verification

• I am familiar with the Contributing Guidelines
• Added or updated focused tests
• Updated documentation for user-visible changes
• Ran make lint
• Ran make test
• Ran make build
• Did not add credentials, private datasets, or proprietary benchmark content

Release Impact

• No user-visible release note needed
• Updated CHANGELOG.md

[rng1995]

@kweinmeister Two issues remain in the new retry/schema behavior: SDK retries multiply the configured budget, and unrelated HTTP 400/422 responses permanently disable structured output for the target. Details and reproductions are inline.

Local validation: 879 focused tests passed. Separate reproductions using the real OpenAI/Anthropic SDKs with mock HTTP transports exposed these gaps. GitHub reports all 17 checks passing. Please address the two cases and add transport-level regressions before approval.

[kweinmeister]

@rng1995 Thanks for the review and the clear reproductions. Both issues have been resolved in commit 7280545:

1. Outer loop owns retry budget: Passed max_retries=0 to both OpenAI and Anthropic constructors and exposed http_client on LLMClient. Verified via mock HTTP transports that max_retries=0 sends 1 request and max_retries=3 sends 4 requests for both providers.
2. Selective schema fallback & deferred memoization: Restricted schema failure detection to genuine unsupported-option error payloads in both host and container template runtimes, and deferred _SCHEMA_UNSUPPORTED_TARGETS memoization until the prompt-only fallback confirms success. Verified that two context_length_exceeded errors followed by a valid prompt maintain [True, True, True] schema flags.

All 6,519 tests and make lint / package build pass cleanly. Ready for another look!

[rng1995]

@kweinmeister Thanks for addressing both earlier findings. I verified that SDK retries are disabled and that schema fallback only caches a genuine unsupported-option response after a successful downgrade. Both existing threads can remain resolved.

One integration issue remains: the normal Harbor runner drops the three retry environment variables before staging and launching the verifier, so the documented settings are ignored there. Details and the reproduction are inline. Please forward those controls through the runner and add an integration regression before approval.

Validation on 248d129: 764 focused tests passed locally, plus independent SDK/schema and environment-forwarding probes. GitHub reports all 17 checks passing; the PR has no merge conflicts.

[kweinmeister]

@rng1995 Thanks for catching that. I pushed an update to forward the three retry environment variables through the Harbor runner into both task staging and the Harbor subprocess environment, added an integration regression test covering the full path from os.environ through verifier config resolution, and verified it with an end-to-end Tier 3 smoke test. Ready for another look when you have a chance.

[rng1995]

@kweinmeister Thank you for your contribution and the updates. The original SDK retry and schema-fallback findings remain fixed, and host forwarding now works. Two existing blockers remain: native-task overrides can produce invalid TOML, and Bedrock ignores the configured judge retry policy. I confirmed both independently and followed up in the existing threads.

Local validation: 839 focused tests passed, one deselected; Ruff and diff checks passed. GitHub reports all 17 CI checks passing and no merge conflicts. Approval is pending the two remaining fixes.

[kweinmeister]

Pushed update in commit c7efb5a addressing the latest review feedback and code standards:

• Native Harbor TOML collisions: Implemented in-place replacement for colliding keys in [verifier.env] during native staging and normalized table header comment parsing (adapter.py).
• Bedrock Converse retries: Disabled internal botocore retries (max_attempts: 0) and applied the bounded judge retry loop and deadline enforcement to Converse calls (eval.py).
• Configurable judge wall-time budget: Added SKILL_EVAL_LLM_JUDGE_BUDGET_SEC (default 180s fallback) forwarded through Harbor and documented in docs/environment-variables.mdx.
• Data clumps refactored: Consolidated loose tuples into SchemaTargetKey and EvalRetryConfig NamedTuples, and unified shared test fixtures in conftest.py.
• Verification: All 7,117 tests, ruff check, ruff format, and ty check pass cleanly.

[rng1995]

@kweinmeister Thank you for your patience through several review rounds, and for your contribution to SkillEvaluator!

I re-reviewed c992b1f. Both remaining blockers are fixed:

• Native-task [verifier.env] collisions: colliding retry settings are now replaced by key, so staged task.toml files parse and resolve to the host value for all three retry settings in default and default_plus_custom. Authored values are kept when no host override is set.
• Bedrock judge retries: botocore retries are disabled, and Converse now follows the configured judge retry count and delay bounds. Against a loopback Converse endpoint, MAX_RETRIES=0 with AWS_MAX_ATTEMPTS=4 sends one request, and MAX_RETRIES=3 with AWS_MAX_ATTEMPTS=1 succeeds on the third request. The new transport-level tests cover both cases.

Validation: Ruff passes, as do 1,076 focused tests and the full local suite (9,696 passed, 18 skipped). All 17 CI checks are green.

A non-blocking follow-up that also exists on main, so not something this PR needs to fix: if an authored task spells the verifier env table another way (env.KEY = ... under [verifier], env = { ... }, or [ verifier.env ]), staging writes a duplicate [verifier.env] table. The staged file then fails to parse, and nothing reports the error. Parsing the staged task.toml after editing would surface it clearly.

Approved.