Skip to content

Add opt-in release pruning after switch - #2

Merged
y86 merged 5 commits into
masterfrom
agent/dicon-prune-old-releases-on-1101-1843
Nov 3, 2026
Merged

y86 merged 5 commits into
masterfrom
agent/dicon-prune-old-releases-on-1101-1843

Conversation

@y86

@y86 y86 commented Nov 1, 2026 •

Copy link
Copy Markdown

Push Cartel hosts have 16 GB disks at 85–93% utilization because every deploy retains another ~32 MB release. The 2026-11-01 inventory found 90 release directories on pushboy-1a, with about 2.68 GB removable. Add explicit retention so operators can reclaim that space and prevent further accumulation.

  • Add config :dicon, prune_on_switch: true to capture each host's current release before replacing the symlink, then prune with expected_current: new, keep_previous: old. Retain the exact previously selected release, whether stable, an RC, or newer than current after rollback. For example, switching 1.7.5 → 1.7.6 keeps 1.7.5 and deletes an unused 1.7.6-rc4. This preserves the old running release and rollback target throughout Push Cartel's switch → stop → start sequence. Rolling back retains the release rolled back from, so rolling forward requires no redeploy.
  • Add standalone mix dicon.prune [--dry-run], with the existing --only/--skip host filters and per-host retention/deletion output. Without switch context, keep current and the highest stable (non-pre-release) SemVer below it; fall back to the highest pre-release below current only when no lower stable exists. Use Version.compare/2, with deterministic directory-name ordering for build-metadata ties. Delete all other version directories, including unused RCs and newer experiments; preserve non-version entries, files and symlinks.
  • Default automatic cleanup to false because this library serves Push Cartel, admin, monitor-ops and other services with differing staging/retention workflows. Each consumer must opt in to bound disk growth. Deploy never prunes, preserving the upload → switch sequence. A first switch with no current keeps only the requested release. Switching to the already-current version skips pruning to preserve the rollback target on retries.
  • Refuse invalid existing current targets before switching, missing current in standalone prune, missing captured previous releases, and incomplete inventories. Quote deletion paths, recheck current and candidate directory types before each removal, and require explicit command-success output because the existing SSH executor ignores remote exit codes. Exercise the generated shell commands in disposable local directories through the existing fake connection/executor.

Serialize deploy, switch and prune on each host; there is no deployment lock. Standalone prune must not run between upload and switch. It has no switch history and can remove an RC or newer release retained by automatic pruning, so inspect its dry run and stop any process using a release it would delete first. A pruning error stops the task without undoing completed switches/deletions.

Cross-system impact: existing consumers retain their behavior until opting in. Rollout requires updating their dicon dependency and setting the new application config; several sibling repos currently constrain dicon to ~> 0.5. No service configuration, runtime protocol, database, or production host changes are included.

Validation on w2 with Elixir 1.18.4 / OTP 27.3.4.16:

  • Prune and switch suites with --warnings-as-errors: 35 tests, 0 failures. Cover the RC-on-disk switch, actual previous RC retention, rollback/roll-forward, stable-first standalone selection, pre-release fallback, first switch, retry, and capture failures.
  • mix format --check-formatted, mix compile --warnings-as-errors, mix deps.unlock --check-unused, and git diff --check: pass.
  • Full suite: 46 tests, 2 existing deploy-test failures. Reproduced both on untouched master (7414456: 15 tests, 2 failures): tests expect empty connection options but existing deploy forwards apps_env. Existing executor tests also emit charlist deprecation warnings on this runtime. These unrelated tests remain unchanged.
  • Initial mix deps.get could not start because the checkout pins uninstalled Elixir 1.12.3 / OTP 23.3.4.20. With installed runtime versions selected via environment variables, dependency resolution succeeded and reported the existing retired earmark 1.2.2. No dependency or tool-version changes.
  • No production access or writes; production execution and the pinned legacy runtime remain unverified. No database setup was needed.

y86 added 4 commits November 1, 2026 18:50
Keep current and the highest lower SemVer, including pre-releases, to stop release directories accumulating on small deployment disks. Prune only after a verified switch so staged uploads survive deploy, and leave automatic cleanup opt-in for shared consumers.

Add a standalone dry-run task and exercise guarded shell deletion through the fake executor against temporary release trees. Document rollback, serialization and rollout requirements.
Separate the command-success marker with NUL boundaries so a directory whose name ends with the marker cannot disguise an incomplete listing. Cover the false-success case through the fake executor.
Push Cartel switches before stopping the old BEAM. Capture current on each host before replacing it and retain that exact release so an unused RC cannot displace the running release or rollback target. Keep the rolled-back-from release available for rolling forward too.

Without switch context, prefer the highest lower stable release and fall back to a pre-release only when no stable exists. Preserve rollback targets on same-version retries, validate captured releases, and cover the updated policies in tests and documentation.
@y86
y86 marked this pull request as ready for review November 3, 2026 20:46
@y86
y86 merged commit ed951fc into master Nov 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant