Skip to content

fix(bundler): stop dry-run previewing a stale cached chart archive - #2855

Merged
mchmarny merged 2 commits into
mainfrom
fix/2842-dry-run-stale-crd-chart
Sep 20, 2026
Merged

mchmarny merged 2 commits into
mainfrom
fix/2842-dry-run-stale-crd-chart

Conversation

@lockwobr

Copy link
Copy Markdown
Contributor

Summary

Gate the cached-chart reuse branch in install-upstream-helm.sh.tmpl on DRY_RUN_FLAG, so a dry run previews the chart the next real run will fetch instead of an archive left behind by an earlier deploy.

Motivation / Context

apply-crds.sh pulls the pinned chart once and leaves it at .aicr-chart.tgz, and install.sh installs from that exact file so both phases are bound to one artifact. That part is deliberate: re-resolving CHART/VERSION would be a second fetch that a mutable tag does not promise returns the same bytes.

But apply-crds.sh is skipped under --dry-run, because a dry run must not touch the cluster:

if [[ -z "${DRY_RUN_FLAG:-}" ]]; then
  bash ./apply-crds.sh
fi

while the reuse branch below it was unconditional:

if [[ -f "${SCRIPT_DIR}/.aicr-chart.tgz" ]]; then

So on a dry run no pull happens, and any archive on disk is by definition left over from a previous real deploy. If the pin has moved since, the preview shows the wrong chart. Worse than the ticket describes: the reuse branch also clears CHART_VERSION_ARGS and REPO, so the previewed command silently loses --version too. A dry run is the one operation whose entire purpose is to tell you what is about to happen.

Fixes: #2842
Related: #2853 (touches the same template; expect a golden conflict on whichever lands second)

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • Build/CI/tooling

Component(s) Affected

  • CLI (cmd/aicr, pkg/cli)
  • API server (cmd/aicrd, pkg/server)
  • Recipe engine / data (pkg/recipe)
  • Bundlers (pkg/bundler, pkg/component/*)
  • Collectors / snapshotter (pkg/collector, pkg/snapshotter)
  • Validator (pkg/validator)
  • Core libraries (pkg/errors, pkg/k8s)
  • Docs/examples (docs/, examples/)
  • Other: ____________

Implementation Notes

One condition, matching the one that already gates the pull, so the two halves of the handoff agree about when an archive exists.

install-local-helm.sh.tmpl was checked and has no equivalent branch: it installs from the vendored chart directory and never consults .aicr-chart.tgz, so the upstream template is the only one affected.

Note this was not closed by the private-mktemp -d rework that landed in #2725. That hardened which archive the pull selects; this is about the install trusting a file the current invocation never produced.

Testing

go test -race ./pkg/bundler/deployer/localformat/
go test ./pkg/bundler/ -run TestStockRenderParityGolden
golangci-lint run -c .golangci.yaml ./pkg/bundler/deployer/localformat/...

All pass; linter reports 0 issues.

The regression renders a bundle, plants a .aicr-chart.tgz, runs install.sh with DRY_RUN_FLAG=--dry-run under a stub helm that records argv, and asserts the upgrade resolved the pinned coordinates. It asserts positively on both the chart ref and --version, not merely on the absence of the archive path, because dropping --version is the second symptom. It also fails if the upgrade call was never reached, so it cannot pass vacuously.

Mutation-verified. Reverting the guard reproduces the defect exactly:

--- FAIL: TestInstallScript_DryRunIgnoresStaleChartArchive
    dry-run install used the stale cached archive instead of resolving the pinned
    chart coordinates: upgrade --install k8s-aibom .../001-k8s-aibom/.aicr-chart.tgz
    --namespace k8s-aibom-system --create-namespace -f values.yaml
    -f cluster-values.yaml --dry-run

Note the absent --version in that output: both symptoms in one line.

Risk Assessment

  • Low — Isolated change, well-tested, easy to revert
  • Medium — Touches multiple components or has broader impact
  • High — Breaking change, affects critical paths, or complex rollout

One shell condition. Real installs are unaffected, since DRY_RUN_FLAG is empty there and the branch behaves exactly as before. The only behavior that changes is the dry run, which now resolves CHART/VERSION the way the subsequent real run will.

Rollout notes: N/A.

Checklist

  • Tests pass locally (make test with -race)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

@lockwobr lockwobr added the theme/deployer Helm, ArgoCD, and deployment bundle generation label Sep 19, 2026
@lockwobr lockwobr self-assigned this Sep 19, 2026
@coderabbitai

coderabbitai Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration

Configuration used: Repository: NVIDIA/aicr/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 0cd6279b-2640-44c7-997b-ead892bff743

📥 Commits

Reviewing files that changed from the base of the PR and between 3e8ffd7 and 7b1c149.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: NVIDIA/aicr/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 3716917b-a2b3-4567-a9da-ed072bf1f114

📥 Commits

Reviewing files that changed from the base of the PR and between dd718ea and 3e8ffd7.

📒 Files selected for processing (1)
  • pkg/bundler/testdata/stock_render_golden.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.


📝 Walkthrough

Walkthrough

The Helm template and generated install scripts now reuse .aicr-chart.tgz only during non-dry-run installations. Dry runs retain the configured chart reference and version. A regression test verifies this behavior with a stale archive. The stock-render golden manifest contains updated artifact digests.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Low

Suggested reviewers: coffeepac

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main fix: preventing dry runs from using a stale cached chart archive.
Description check ✅ Passed The description directly explains the defect, the implementation, the regression test, and the expected behavior for dry runs and real installs.
Linked Issues check ✅ Passed For #2842, install-upstream-helm.sh.tmpl now uses .aicr-chart.tgz only when DRY_RUN_FLAG is unset and the archive exists. Dry runs therefore resolve the pinned chart coordinates and preserve the…
Out of Scope Changes check ✅ Passed The rendered install.sh fixtures and regenerated pkg/bundler/testdata/stock_render_golden.yaml reflect the template change. The new test supports #2842. No unrelated change is demonstrated.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@lockwobr
lockwobr force-pushed the fix/2842-dry-run-stale-crd-chart branch from 1fd1068 to c0166a6 Compare September 19, 2026 02:31
@github-actions

Copy link
Copy Markdown
Contributor

Recipe evidence check

No leaf overlays affected by this PR.

This gate is warning-only and never blocks merge.

@lockwobr
lockwobr marked this pull request as ready for review September 19, 2026 02:34
@lockwobr
lockwobr requested a review from a team as a code owner September 19, 2026 02:34
@github-actions

github-actions Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Coverage Report ✅

Metric Value
Coverage 84.9%
Threshold 83%
Status Pass
Coverage Badge
![Coverage](https://img.shields.io/badge/coverage-84.9%25-brightgreen)

No Go source files changed in this PR.

@lockwobr
lockwobr force-pushed the fix/2842-dry-run-stale-crd-chart branch 2 times, most recently from dd718ea to 3e8ffd7 Compare September 20, 2026 00:56
apply-crds.sh is the only thing that writes .aicr-chart.tgz, and it is
skipped under --dry-run so the cluster is not touched. The install script
still selected that archive on existence alone, so a dry run previewed
bytes left by an earlier real deploy rather than what the next real run
would fetch. The reuse branch also clears CHART_VERSION_ARGS and REPO, so
the preview silently dropped --version as well.

Gate the reuse on the same DRY_RUN_FLAG condition that gates the pull.

install-local-helm.sh.tmpl has no equivalent branch; it installs from the
vendored chart directory, so only the upstream template is affected.

Signed-off-by: Brian Lockwood <lockwobr@gmail.com>
@lockwobr
lockwobr force-pushed the fix/2842-dry-run-stale-crd-chart branch from 3e8ffd7 to 982d560 Compare September 20, 2026 04:40

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve: no findings against 982d560. Required checks and focused dry-run/install tests pass at the reviewed SHA.

@mchmarny
mchmarny enabled auto-merge (squash) September 20, 2026 11:47
@mchmarny
mchmarny disabled auto-merge September 20, 2026 11:47
@mchmarny
mchmarny enabled auto-merge (squash) September 20, 2026 11:52
@mchmarny
mchmarny merged commit b510a0f into main Sep 20, 2026
76 checks passed
@mchmarny
mchmarny deleted the fix/2842-dry-run-stale-crd-chart branch September 20, 2026 11:58
EronWright added a commit to turfbuild/aicr that referenced this pull request Sep 20, 2026
NVIDIA#2855 added the dry-run stale-archive guard to
install-upstream-helm.sh.tmpl after this branch captured its golden, so
TestBundleGolden_ReadinessGate compared freshly generated output against
the pre-NVIDIA#2855 install.sh and failed.

Regenerated with -update; the only file that moves is
testdata/readiness_gate/001-foo/install.sh, which now carries the same
DRY_RUN_FLAG guard and comment the other helm goldens already have. No
production code changes.

Signed-off-by: Eron Wright <eronwright@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/bundler size/XL theme/deployer Helm, ArgoCD, and deployment bundle generation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dry-run previews a stale cached chart in ownsCRDs bundles

2 participants