Skip to content

fix(deployments): warn when a previous role container cannot be removed - #1263

Merged
Hydralerne merged 2 commits into
oblien:mainfrom
sudanese:fix/role-retire-warning
Oct 15, 2026
Merged

Hydralerne merged 2 commits into
oblien:mainfrom
sudanese:fix/role-retire-warning

Conversation

@sudanese

@sudanese sudanese commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When a single-app server deploy cannot remove a previous role container through RoleRollout.commit(), the deploy now finishes with a warning. The warning names each container by role, replica and short container ID.

Motivation

After the web container is live, commit() removes the previous release's role containers best-effort. A removal that rejected or timed out was only logged. The deploy finished clean, while an old worker could keep taking jobs with the previous release's code. Retention cleanup does not remove role containers, so nothing retried it either.

Related issue

Refs #1255 (item 4). This change makes the failure visible. Retrying the removal through retention cleanup is a separate follow-up.

Changes

packages/platform

  • src/engine/modules/deployments/role-rollout.ts: RoleRollout.commit() now resolves to string | null instead of void. It still never throws. The string names only the containers whose removal failed. Example: 2 previous role containers could not be removed: worker replica 1 (…), worker replica 2 (…). They may still be running the previous release's code. Remove them manually. A failed removal does not prove the container still runs, so the message says "may". The existing per-container warn log lines are unchanged.
  • src/engine/modules/deployments/build-pipeline.ts: executeServerDeploy adds that warning to the existing deployWarnings list, next to hostPortClaimWarning. From there it reaches onSuccess({ warningMessage }), the live warning banner in DeploymentProcessing, and the persisted Deployment completed with warnings: … log line.

Compose deployments retire a single-app predecessor's roles through retirePreviousRoleContainers. That separate path remains log-only.

The warning is deliberately not written to meta.deployWarning. That key marks routing trouble and offers "Retry routing", which would be the wrong fix here. As a result, the banner shows on the live deploy only, not after a cold page refresh.

apps/api

  • test/modules/deployments/role-rollout.test.ts:
    • The existing reject-or-hang commit test now asserts the exact warning. Two of three removals fail, one rejects and one times out, and only those two are named.
    • A new test covers a single failure with the singular wording and a 12-character container ID. A second commit() returns null.
    • Two clean-commit tests now assert null.
  • src/modules/deployments/build-pipeline.prebuilt-image.test.ts: a new pipeline-level case. The previous deployment recorded a role container that its runtime cannot remove, and the new release declares no roles. The deploy still succeeds. onSuccess receives the exact warning as warningMessage. metaPatch has no deployWarning and no edgeUnsynced.

Verification

bun run --cwd apps/api test -- role-rollout   # 48 passed (47 before, 1 new)
bun run --cwd apps/api test -- build-pipeline.prebuilt-image   # 73 passed (72 before, 1 new)
bun run --cwd apps/api test -- modules/deployments   # 92 files, 1535 passed
RUN_DOCKER_E2E=1 bun run --cwd apps/api test:e2e -- runtime-roles   # 8 passed, Docker Desktop
bun run --cwd packages/platform lint   # tsc --noEmit, clean
bun run --cwd apps/api lint            # tsc --noEmit, clean

Before the fix, the commit tests fail with expected undefined to be '2 previous role containers could not …'.

With build-pipeline.ts and role-rollout.ts reverted to main, the new pipeline test fails with expected undefined to be 'Previous role container queue replica…'.

Mutation checks on the rollout. Each mutant fails two rollout tests:

  • commit() always returns null. This mutant also fails the pipeline test.
  • The warning names every previous container, not only the failed ones.
  • The warning omits the container ID.

Mutation checks on the pipeline wiring. Each mutant fails the new pipeline test:

  • The warning is not added to deployWarnings.
  • The warning is written to metaPatch.deployWarning as well.

bun format: role-rollout.ts and role-rollout.test.ts are Prettier-clean. build-pipeline.ts and build-pipeline.prebuilt-image.test.ts were not Prettier-clean before this change. I did not reformat them, and Prettier leaves the changed lines unchanged.

Checklist

  • One change per PR — one bug, or one agreed feature, with nothing unrelated bundled in
  • The diff is scoped — no reformatting or lint fixes on lines I wasn't otherwise changing
  • A test fails without this change and passes with it, at the rollout level and at the pipeline level
  • Verification covers the API deployments tests, the runtime-roles Docker e2e suite, and the platform and API typechecks.
  • I understand every line of this diff and can explain it in review

commit() removes the previous release's role containers best-effort.
A failed or timed-out removal was only logged, so the deploy finished
clean while an old worker could keep taking jobs with the previous
release's code.

commit() now resolves to a warning that names each container it could
not remove by role, replica and short container ID, or null. The
server deploy adds it to the existing deployWarnings list, so it
reaches warningMessage, the live warning banner and the persisted
"Deployment completed with warnings" log line.

The warning stays out of meta.deployWarning. That key means routing
trouble and offers "Retry routing", which would be the wrong fix.
Retrying the removal through retention cleanup is a separate change.

Refs oblien#1255
…image pipeline

The rollout tests pin the warning that commit() returns. This test pins
how executeServerDeploy uses it. A previous deployment recorded a role
container that its runtime cannot remove, and the new release declares
no roles. The deploy still succeeds. onSuccess receives the warning as
warningMessage, and metaPatch carries no deployWarning or edgeUnsynced,
so the project is not flagged for "Retry routing".

Refs oblien#1255
@Hydralerne
Hydralerne merged commit e81786a into oblien:main Oct 15, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants