Repository navigation
fix(deployments): warn when a previous role container cannot be removed - #1263
Merged
Merged
Conversation
commit() removes the previous release's role containers best-effort. A failed or timed-out removal was only logged, so the deploy finished clean while an old worker could keep taking jobs with the previous release's code. commit() now resolves to a warning that names each container it could not remove by role, replica and short container ID, or null. The server deploy adds it to the existing deployWarnings list, so it reaches warningMessage, the live warning banner and the persisted "Deployment completed with warnings" log line. The warning stays out of meta.deployWarning. That key means routing trouble and offers "Retry routing", which would be the wrong fix. Retrying the removal through retention cleanup is a separate change. Refs oblien#1255
…image pipeline The rollout tests pin the warning that commit() returns. This test pins how executeServerDeploy uses it. A previous deployment recorded a role container that its runtime cannot remove, and the new release declares no roles. The deploy still succeeds. onSuccess receives the warning as warningMessage, and metaPatch carries no deployWarning or edgeUnsynced, so the project is not flagged for "Retry routing". Refs oblien#1255
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a single-app server deploy cannot remove a previous role container through
RoleRollout.commit(), the deploy now finishes with a warning. The warning names each container by role, replica and short container ID.Motivation
After the web container is live,
commit()removes the previous release's role containers best-effort. A removal that rejected or timed out was only logged. The deploy finished clean, while an old worker could keep taking jobs with the previous release's code. Retention cleanup does not remove role containers, so nothing retried it either.Related issue
Refs #1255 (item 4). This change makes the failure visible. Retrying the removal through retention cleanup is a separate follow-up.
Changes
packages/platform
src/engine/modules/deployments/role-rollout.ts:RoleRollout.commit()now resolves tostring | nullinstead ofvoid. It still never throws. The string names only the containers whose removal failed. Example:2 previous role containers could not be removed: worker replica 1 (…), worker replica 2 (…). They may still be running the previous release's code. Remove them manually.A failed removal does not prove the container still runs, so the message says "may". The existing per-container warn log lines are unchanged.src/engine/modules/deployments/build-pipeline.ts:executeServerDeployadds that warning to the existingdeployWarningslist, next tohostPortClaimWarning. From there it reachesonSuccess({ warningMessage }), the live warning banner inDeploymentProcessing, and the persistedDeployment completed with warnings: …log line.Compose deployments retire a single-app predecessor's roles through
retirePreviousRoleContainers. That separate path remains log-only.The warning is deliberately not written to
meta.deployWarning. That key marks routing trouble and offers "Retry routing", which would be the wrong fix here. As a result, the banner shows on the live deploy only, not after a cold page refresh.apps/api
test/modules/deployments/role-rollout.test.ts:commit()returnsnull.null.src/modules/deployments/build-pipeline.prebuilt-image.test.ts: a new pipeline-level case. The previous deployment recorded a role container that its runtime cannot remove, and the new release declares no roles. The deploy still succeeds.onSuccessreceives the exact warning aswarningMessage.metaPatchhas nodeployWarningand noedgeUnsynced.Verification
Before the fix, the commit tests fail with
expected undefined to be '2 previous role containers could not …'.With
build-pipeline.tsandrole-rollout.tsreverted tomain, the new pipeline test fails withexpected undefined to be 'Previous role container queue replica…'.Mutation checks on the rollout. Each mutant fails two rollout tests:
commit()always returnsnull. This mutant also fails the pipeline test.Mutation checks on the pipeline wiring. Each mutant fails the new pipeline test:
deployWarnings.metaPatch.deployWarningas well.bun format:role-rollout.tsandrole-rollout.test.tsare Prettier-clean.build-pipeline.tsandbuild-pipeline.prebuilt-image.test.tswere not Prettier-clean before this change. I did not reformat them, and Prettier leaves the changed lines unchanged.Checklist