UAT release cells run main's test fixtures against a released binary

[mchmarny]

Summary

Nightly UAT release cells version-pin the aicr binary, the validator images, and the
snapshot-agent image to the release tag, but run the tests/uat/** fixtures and driver
scripts from main. Any breaking change to a lane's test config or harness therefore turns
that lane's release cell permanently red until the next tag - a compatibility artifact, not a
regression signal.

How it surfaced

UAT gcp-h100 training @ v0.21.1 failed six nights running, starting with the first release
cell after #2705 merged (2026-09-14). Example:
run 35451017035.

[cli] command failed: error=[INVALID_REQUEST] failed to parse config from
"/tmp/aicr-uat-tcpxo-config.aicr-uat-35451017035.yaml": yaml: unmarshal errors:
  line 60: field gke not found in type config.RecipeConfigurationSpec exitCode=2

#2705 added spec.recipe.configuration.gke.tcpxoInterfaces to the training config and the
GKE field to config.RecipeConfigurationSpec in the same commit. v0.21.1 has neither, and
pkg/config/loader.go
sets dec.KnownFields(true), so the config is rejected outright. The product behaved
correctly - the harness handed it a config from the future.

Note that stripping the offending field would not have been sufficient. The same commit also
made tests/uat/gcp/run export TRAINJOB_RUNTIME=torch-distributed-tcpxo unconditionally and
changed the shared tests/uat/lib/phases.sh, and v0.21.1 ships no such
ClusterTrainingRuntime. The fixture and the harness move together, so there is no clean
seam between "test data" and "test driver" in tests/uat/**.

Current mitigation (deliberately not a fix)

Commit b99c8e453 gates gcp-h100 training to >= v0.22.0 via
nightly-intent-min-versions. That silences this instance and self-resolves when v0.22.0
ships, but it is per-incident: the next breaking fixture change on any lane needs the same
manual gate, and until a human notices, that cell is red and teaching people to ignore red.

Blast radius today is one lane - tests/uat/gcp/tests/h100-training-config.yaml is the only
one of the ten lane configs carrying a spec.recipe.configuration block - but the structural
gap applies to all of them.

Options considered

1. Pin tests/uat/** to the tag for release cells via a second, scoped checkout, keeping
   .github/** from main. Gives release cells a coherent meaning (test what that release
   actually shipped) and keeps coverage through breaking changes. The blocker: there is no
   release/* branch on the remote, so a bug in a tag's UAT harness would be unfixable
   without cutting a new tag.
2. Repoint the existing actions/checkout ref: to the tag. Rejected - uses: ./.github/actions/*
   resolves against the checked-out workspace, and uat-gcp.yaml has six such references
   including install-aicr-release itself, which runs after checkout. This would silently
   swap the CI code that installs the release under test.
3. Auto-skip a release cell when its fixtures moved since the tag
   (git diff --quiet <tag>..HEAD -- tests/uat/<cloud>/ in Validate inputs). Self-maintaining
   and makes skew visible as a skip rather than a red run. Two traps: the checkout is shallow
   (fetch-depth defaults to 1, no tags present, so git diff <tag>..HEAD exits 128 not 1),
   and rc=1 (differs) must be distinguished from rc!=0 (error) or the guard skips on
   failure. Raising fetch-depth: 0 would slow every UAT job including the main cells.
4. Generalize nightly-intent-min-versions bookkeeping - keep the per-incident gate but
   detect the need automatically and fail the nightly with a clear "gate me" message instead
   of a red cell.

Option 1 is the most principled; option 3 is the cheapest that removes the false signal.
Option 1 becomes viable if release branches are introduced.

Acceptance criteria

• A breaking change to a lane's tests/uat/** fixtures or harness does not produce a
  red release cell for pre-change tags.
• Whatever the mechanism, the reduced coverage is visible (a skip, a notice, or an
  explicit gate) rather than silent.
• Release cells either test fixtures matching their tag, or do not run at all for the
  affected intent.
• docs/contributor/uat.md documents the chosen contract.
• The gcp-h100 training gate added by b99c8e453 is removed or subsumed once the
  general mechanism lands.

Notes

• UAT is not one of the seven release quality gates in RELEASE.md, so this never blocked a
  release. The cost is degraded signal and normalized red.
• If Monday's release is numbered anything other than v0.22.0, the interim gate's floor
  must be corrected - an over-high floor silently skips a good release cell, which is the
  bad direction.