Skip to content

fix(deployments): require a stable finding for every role replica - #1262

Open
sudanese wants to merge 1 commit into
oblien:mainfrom
sudanese:fix/role-startup-strict
Open

sudanese wants to merge 1 commit into
oblien:mainfrom
sudanese:fix/role-startup-strict

Conversation

@sudanese

@sudanese sudanese commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The role startup check now requires exactly one stable stability finding for each role container it started. A missing, duplicate or completed finding fails the rollout, so the predecessor is kept.

Motivation

The docs say "Every role container must stay up through the stabilization window". runtimeRoleRolloutDeps.verify checked only !verdict.ok, which let two cases through:

  • Missing finding. verifyDeployedContainers returns no finding for a replica it could not inspect, and none at all when sampling is unavailable. Without an exec health command, the rollout passed and could retire the predecessor without seeing its replacement run.
  • completed. The shared verdict treats an exit-0 process as ok when its restart policy leaves it stopped. Supported single-app role deployments use restart policy always, so this is an adapter-level case. A role is a long-running process, so it must not count as started.

Related issue

Refs #1255 (item 1).

Changes

packages/platform

  • src/engine/modules/deployments/role-runtime.ts: verify groups findings by target.containerId.
    • A finding that is not stable fails first, because its detail carries the exit code and log tail. A completed finding gets its own message: Role "worker" replica 2 exited with code 0 during the startup check. A role is a long-running process, so it must keep running.
    • Then a container with zero or several findings fails as unverified: Could not verify that role "worker" replica 1 started: the startup check returned no result for container …
    • Matching is by container ID, not by count, so duplicate findings cannot hide a missing replica.
    • exec health commands still run only after every replica passes.

The shared verifyDeployedContainers and the adapter's completed semantics are unchanged. Other callers keep their current behaviour.

apps/api

  • test/modules/deployments/role-runtime.test.ts:
    • The finding helper now takes a container ID and a status.
    • Four existing pass-path tests now supply one stable finding per container. Before, three of them passed with no findings, and one passed with a finding for a different container ID.
    • Six new cases: distinct stable findings pass, completed fails, a missing finding fails, a duplicate finding fails, a conclusive failure is reported before an unverified replica, and exec health does not run when a replica is unverified.
    • A seventh new case runs the issue's repro through the real verifyDeployedContainers: sampleStability throws, and the rollout now fails instead of passing.

Verification

bun run --cwd apps/api test -- role-runtime   # 47 passed (40 before, 7 new)
bun run --cwd apps/api test -- role- stability project-move   # 6 files, 153 passed
RUN_DOCKER_E2E=1 bun run --cwd apps/api test:e2e -- runtime-roles   # 8 passed, Docker Desktop 29.8.2
bun run --cwd packages/platform lint   # tsc --noEmit, clean
bun run --cwd apps/api lint            # tsc --noEmit, clean

Mutation checks. Each mutant fails at least one test:

Mutant Failing tests
Restore !verdict.ok (the code before this change) 5 (completed, missing, duplicate, real-watch repro, exec-not-run)
Match by count (findings.length === containers.length) instead of by container ID 2 (missing, duplicate)
Accept completed as started 1 (completed)
Check unverified replicas before conclusive failures 1 (conclusive failure first)

bun format: both changed files are Prettier-clean.

Behaviour change

A transient inspection error during the stabilization window now fails the role rollout. Before, it passed with a warning. The rollout's existing unwind runs, and the previous release keeps its role containers.

Checklist

  • One change per PR — one bug, or one agreed feature, with nothing unrelated bundled in
  • The diff is scoped — no reformatting or lint fixes on lines I wasn't otherwise changing
  • A test fails without this change and passes with it
  • Verification covers the role, stability and project-move API tests, the runtime-roles Docker e2e suite, and the platform and API typechecks.
  • I understand every line of this diff and can explain it in review

The role startup check rejected only findings with !verdict.ok. The
shared stability watch omits a replica it could not inspect, and it
accepts an exit-0 "completed" process. So a rollout could pass, and
retire its predecessor, without seeing the replacement run.

runtimeRoleRolloutDeps.verify now requires exactly one finding per
requested container ID, with verdict.status "stable". A completed
replica fails with a long-running-role message. A missing or duplicate
finding fails as unverified. Matching is by container ID, so duplicate
findings cannot hide a missing replica. A conclusive failure is
reported first because its detail carries the exit code and log tail.

The shared checker is unchanged. A transient inspection error now
fails the role rollout instead of passing it.

Refs oblien#1255
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant