fix(deployments): hand over a singleton by name, singleton status and command

[sudanese]

Summary

Singleton handover now decides which roles continue by name, effective singleton status and exact command text. A singleton whose workload no new role continues is stopped and retained before any added or changed role starts. On failure, it is restored only after every added or changed role is confirmed removed.

Motivation

Singleton handover matched previous containers by role name (role-rollout.ts). Two changes bypassed it:

• Rename. A scheduler renamed from scheduler to cron started beside the old container. The old one was removed only at commit(), after the web container was live.
• Kind change. A singleton that became a scalable worker with the same name started its replicas beside the old singleton.

In the run reported in #1253, the renamed cron and the old scheduler both ran for 23.3 s, and both fired the 06:34 tick.

A name alone cannot identify a workload. Two singletons can swap names, and an existing worker name can take over a scheduler's command. Both would still run one workload twice if continuity were decided by name and kind.

Related issue

Refs #1253. With the previous deployment's saved role definitions, this change handles each case the issue lists. Without them, the old same-name behaviour remains (see Limits).

Changes

packages/platform

• src/engine/modules/deployments/role-rollout.ts:
  • New planSingletonHandover(roles, previousRoles, previous). A new role continues a previous role only when the name, the singleton flag and the command all match. command is the only part of a role definition that says what it runs. The image and the environment come from the deployment and are shared by every role. Every other new role is changed.
  • A previous singleton that no new role continues is outgoing. When some role changed, the outgoing singleton is stopped and retained after the unchanged workers pass their startup check, and before any changed role starts. Changed workers start and are checked after that stop.
  • With no changed role, this comparison identifies no replacement, so outgoing singletons remain running until commit.
  • On failure, an outgoing singleton is restored only after every changed role is confirmed removed, whatever its name. A worker that continues unchanged does not block the restore. A continued singleton is still restored once its same-name replacement is confirmed removed.
  • The singleton phase skips a previous container that was already stopped as an outgoing singleton, so a swapped name is not stopped twice.
  • RoleRolloutDeps gains optional previousRoles. Without them, only same-name handover applies, exactly as before.
  • The singleton stop moves into one stopRetaining helper. Its cancellation check, timeout and error message are unchanged.
• src/engine/modules/deployments/role-runtime.ts: runtimeRoleRolloutDeps passes previousRoles to the rollout. It already received them for the teardown timeout.

apps/web

• content/docs/reference/openship-json.mdx: the singleton paragraphs now state the continuation rule, the outgoing order, the restore rule and the fallback when saved definitions are missing. They no longer limit stop-first handover to a role that keeps its name.

apps/api

• test/modules/deployments/role-rollout.test.ts: 18 new cases.
  • Rename: a renamed scheduler (scheduler to cron) is stopped after the unchanged worker is checked, and before cron starts.
  • Kind change: a scheduler that becomes a worker is stopped before any replica starts, both when it keeps its name and when it is renamed.
  • Swapped names: old a runs A and old b runs B, then new a runs B and new b runs A. Both old singletons stop before either new one starts. On abort, neither is restored while removal of either new singleton is uncertain. Both are restored when both are removed.
  • Workload moved into a worker name: old scheduler runs A and old queue runs B, then the new queue runs A with two replicas. The old scheduler stops before either replica starts. On abort, it is restored only when every replica is removed.
  • A same-name singleton with an edited command is stopped once, as an outgoing singleton.
  • A removed scheduler with no changed role keeps running until commit.
  • Without previous definitions, same-name handover applies as before.
  • A cancellation during the worker check is seen before the renamed scheduler is stopped.
  • After a failed start, the renamed scheduler is restored when cron is removed, stays stopped when cron cannot be removed, and is restored when only the unchanged worker cannot be removed.
  • A failed stop of the renamed scheduler fails the rollout and never starts cron.
• test/modules/deployments/role-runtime.test.ts: one new case. Through runtimeRoleRolloutDeps and the real rollout, a renamed scheduler's stop precedes its replacement's start.
• test/e2e/runtime-roles.e2e.test.ts: one new Docker case, the [Bug]: Renaming a singleton role runs the old and new containers together #1253 repro. Release f runs scheduler, and release g renames it to cron. Before commit, the old scheduler is exited and cron runs. The scheduler log ends start-f, stop-f, start-g. The test passes previousRoles to the rollout directly. The unit case above covers the forwarding in runtimeRoleRolloutDeps.

Verification

bun run --cwd apps/api test -- role-   # 3 files, 114 passed (95 before, 19 new)
bun run --cwd apps/api test -- modules/deployments   # 92 files, 1552 passed
RUN_DOCKER_E2E=1 bun run --cwd apps/api test:e2e -- runtime-roles   # 9 passed (8 before), Docker Desktop 29.8.2
bun run --cwd packages/platform lint   # tsc --noEmit, clean
bun run --cwd apps/api lint            # tsc --noEmit, clean

With role-rollout.ts from main, 14 of the new unit cases fail. The new Docker case fails with expected 'running' to be 'exited', because the old scheduler still runs beside cron.

Mutation checks. Each mutant fails at least one test:

Mutant	Failing tests
Continuation ignores command (name and singleton flag only)	6
Continuation ignores the singleton flag	1
Outgoing decided by same-name singleton, not by continuation	3
Singleton phase stops an already-retained outgoing container again	5
No outgoing singletons (same-name handover only)	14
Outgoing singletons stop even with no changed role	1
Restore checks same-name replacements only	4
Restore blocked by any replacement that could not be removed	3
Changed workers start before the outgoing stop	3
runtimeRoleRolloutDeps does not pass previousRoles	1

bun format: every changed file is Prettier-clean, except openship-json.mdx, which was not Prettier-clean before this change. Prettier reports no change on the lines I edited there.

Behaviour notes

• A singleton whose command changes under the same name is outgoing. It stops before the changed roles start, and its restore waits for every changed role to be removed.
• When a singleton is removed and another role is added or changed in the same deploy, the old singleton now stops before that role starts, not at commit. The new role could be the same workload under a new name. If the deploy fails, the old singleton is restored under the rule above.

Limits

• Continuation compares saved role fields. It does not detect changes in behavior caused by a new deployment image, environment, inherited image settings or application code behind an unchanged command.
• Name reuse and swaps are handled by the command comparison. Two roles that run the same workload under different commands, for example with an added flag, still count as different roles. The previous singleton is then outgoing, so it stops first, and its restore waits for every changed role.
• A previous deployment without saved role definitions keeps the old same-name behaviour. No normal deploy path loses those definitions.
• Compose deployments and Docker migrations do not use this rollout. This change does not alter how they retire or move role containers.
• A worker that is renamed into a singleton is not an outgoing singleton. Its old replicas keep running until commit, as before.
• Durable handover recovery is out of scope. A coordinator crash or a runtime that never answers can still leave containers unresolved, as the docs state.

Checklist

• One change per PR — one bug, or one agreed feature, with nothing unrelated bundled in
• The diff is scoped — no reformatting or lint fixes on lines I wasn't otherwise changing
• A test fails without this change and passes with it, including a Docker repro of the issue
• Verification covers the API role and deployment tests, the runtime-roles Docker e2e suite, and the platform and API typechecks.
• I understand every line of this diff and can explain it in review