dra-node-labeler health check suppressed on the DRA-eviction opt-in path

[yuanchen8911]

Summary

#2847 fixed the default-path NOT_FOUND regression from #2846, but suppresses the dra-node-labeler health check on the DRA-eviction opt-in path as well — where the component is deployed and the check was working before. The check now runs on no standard bundle-generated validation path.

This was a deliberate, disclosed tradeoff to unblock the nightly (see the #2847 description), not an oversight. This issue tracks restoring the coverage.

Mechanism

injectDRAEvictionLabel sets enabled: true only in the bundler's in-memory componentValues map. That value never reaches the recipe the validator reads:

• recipeResult.WithResolvedValues(componentValues) returns a shallow copy pinning an unexported resolvedValues field, used only for runComponentValidations; it is never assigned back.
• writeRecipeFile serializes recipeResult, whose ComponentRefs[].Overrides are untouched. No write-back to .Overrides exists in the bundler.

So with --dra-eviction-node-label set, the DaemonSet is deployed but the validator resolves enabled: false, sees an empty render, and suppresses via the new gatedHealthCheckSuppressed case.

The transitive mitigation is incomplete

#2847 notes the state stays covered by the DRA kubelet-plugin readiness probe. That holds only for total labeler failure:

• Total failure — no node labeled, plugin desiredNumberScheduled is 0, the > 0 guard in recipes/checks/nvidia-dra-driver-gpu/health-check.yaml fails. Covered.
• Partial failure — a newly joined gpu.present node never gets labeled. It is not counted in desiredNumberScheduled, so numberReady == desiredNumberScheduled still holds and the check passes while that node runs no kubelet plugin and publishes no ResourceSlices. Not covered.

The labeler's own check keys on the broader gpu.present cohort via nodeAffinity, which is why it catches the partial case and the plugin check cannot.

Why pointing at bundle/recipe.yaml is not sufficient on its own

The bundler already writes a post-filter bundle/recipe.yaml recording the effective component inventory, which is the right input for post-deployment validation. But after #2847 that alone does not restore the check: the serialized recipe's dra-node-labeler ref still resolves enabled: false from its values file, so the render is empty and the suppression fires regardless of which recipe is read.

Scope

• Define bundle/recipe.yaml as the canonical input for all post-deployment validation, not just UAT. Point the post-install readiness gate and conformance validations at it, and update the user-facing workflows that currently validate the original recipe after deployment — README.md (Quick Start, "After deploying the bundle, validate the running cluster against the recipe"), docs/user/tutorial.md, the post-deployment examples in docs/user/validation.md, and the demos under demos/. The pre-bundle --no-cluster dry run (UAT's phase_prep, and any documented pre-deploy check) must keep using the original recipe, because the bundle does not exist at that point.
• Either remove the default-off manifest gate and its validator suppression case, or persist the effective enabled: true into the serialized validation input.
• Replace the synthetic opted-in test in expected_resources_test.go, which hand-writes Overrides: {"enabled": true} — a state the bundler never produces — with coverage exercising the real bundle-to-validation path.
• Correct the claim in recipes/components/dra-node-labeler/values.yaml (repeated in fix(validator): suppress dra-node-labeler health check when eviction not opted in #2847's description) that the new key is "distinct from the ComponentRef enabled override that IsEnabled reads". They are the same key: componentEnabledOverrideKey = "enabled", and Overrides merge into component values in resolveComponentValues.
• Correct the suppression reason string, which asserts the bundler did not deploy the component — untrue on the opt-in path.

Acceptance

• Without DRA eviction: the labeler is absent from the validation input and its health check does not run.
• With DRA eviction enabled: the labeler is present and its health check runs and can fail.
• Following any documented post-deployment validation workflow uses bundle/recipe.yaml for the effective component inventory and evaluates health-check rendering consistently with the generated bundle.

Notes

Reachable through a supported, documented flag (--dra-eviction-node-label / scheduling.draEvictionNodeLabel). No UAT lane opts in today, which lowers present exposure but does not make the path unsupported.

Related: #2846, #2847, #2813, #2676