feat(deployments): run worker and scheduler roles beside the web container

[Farahat612]

Run explicitly declared worker and scheduler processes as separate containers from the web app's built image. A Laravel queue worker or Rails Sidekiq process can roll out with its release and follow it through pause, resume, restart, rollback and deletion.

Part of #935: this implements the deployment and runtime support. Automatic preset roles are separate work.

Behavior

• Read roles from openship.json or the project API, validate them through the shared parser, and freeze them on the deployment.
• Start scalable workers beside the previous release and check them before changing any scheduler.
• Hand singleton roles over by stopping and retaining the previous container before starting its replacement. Retire the previous release's roles after the web deployment succeeds.
• On failure, attempt concurrent, bounded removal of the new containers. Restore a previous singleton only when every replacement for that role was successfully removed. A rejected or timed-out removal leaves that previous singleton stopped, logs both container IDs and the recovery action, and does not prevent independent roles from being restored. The original deployment error is preserved.
• Require a confirmed stop response before starting a replacement. Docker handles already-stopped and absent containers through the existing graceful-stop helper; inventory statuses such as unhealthy, paused or restarting cannot override a stop failure.
• Give role containers the app's environment without PORT, volumes, restart policy, resource limits and connected networks, with their own command and stop settings. They publish no ports or web network aliases. Disable the image's inherited web healthcheck; roles get process stabilization and an optional exec startup check.
• Include recorded role containers in lifecycle operations and cleanup keep sets. A subsequent Compose deployment retires the predecessor's roles. Project moves and copies are refused while roles are configured.

Migration 0192_runtime_roles adds nullable project.roles and deployment.role_containers columns after the existing lifecycle migration. Existing projects retain their behavior. Contracts, SDK, dashboard configuration plumbing and the configuration reference carry the new field.

Supported scope

Single-app Docker deployments. Cloud, bare, Kubernetes, static sites and multi-service deployments refuse configured roles before building. This does not add a role list/logs UI, steady-state role health monitoring or automatic role detection.

If replacement removal cannot be confirmed during rollback, recovery requires removing those replacements before restarting the previous singleton. A timeout is not treated as proof of shutdown.

Validation

• Local deployment and project-move tests: 136 passed.
• Local Docker adapter role, graceful-stop and portless-worker tests: 52 passed.
• Local migration-chain tests: 12 passed, including a populated upgrade from the lifecycle migration that preserves existing project/deployment rows, applied migration history and idempotence.
• API, adapter and platform typechecks passed; structured error-boundary audit passed.
• All 8 real-Docker role tests passed in the PR's Compose cutover/rollback CI job (12 deployment tests passed in that job overall). The role suite uses an image with a working HTTP healthcheck and covers portless role startup, worker overlap, scheduler handover, failed startup, rejected/timed-out rollback removal, late cleanup and retiring roles.
• All 17 CI checks passed on commit ee70abfc2d7cc932f5a08ea734b489bcd3653bf5, including both API shards, database, adapters, SDK/CLI, Cloud Docker workspaces and Linux/macOS Actions execution: https://github.com/oblien/openship/actions/runs/38698871550.

[Hydralerne]

perfect, amazing work thank you

[sudanese]

Thanks for this. I tested 7b2220ec6 locally with Laravel 12.70.0 on Docker Desktop. The app used a custom PHP CLI Dockerfile, a database queue worker with a 90s grace, a scheduler, and persisted SQLite. The app used the portless deployment path, so this did not exercise edge routing. The verification commands I ran passed.

The core path held up well. Deploys, env changes, rollback and the lifecycle actions all carried the roles. A failing release or exec health command kept the previous version running, and a long queue job drained cleanly within its grace. Below are the gaps I found, two reproduced and the rest from reading the code, each with a suggested fix. Happy to be told any are already planned.

Reproduced

1. An interrupted rollout leaves role containers running, and the next deploy runs two schedulers.

role_containers is written only after rolloutRoles returns (build-pipeline.ts:2507). On boot, sweepStaleInFlight (apps/api/src/app.ts:422) only marks the deployment cancelled.

Repro: deploy H (queue + scheduler) → start deploy I → kill the API once I's scheduler has started → restart the API. I is cancelled, but its queue and scheduler keep running unrecorded, and H's scheduler stays stopped. Deploy J then starts its own scheduler beside I's orphan, and two minute ticks each fired twice. The orphan worker also kept taking jobs with I's code.

Suggested fix: durable handover recovery needs to track container ownership and activation progress. Label-based discovery could provide an interim guard. But a deployment that differs from the active pointer may be reconciling, intentionally preserved on cancellation, or already serving before the pointer was updated. Such a guard needs an explicit reconciliation policy, and it is not a false-positive-free orphan check. Until recovery exists, the docs should explain how interrupted handovers can require reconciliation before redeploying.

2. Renaming a singleton runs old and new together.

Previous singletons are matched by role name (role-rollout.ts:208). After renaming scheduler to cron, the old scheduler wasn't stopped first and was only removed at commit(). That gave a 23.3s overlap, and that minute's tick fired twice.

Suggested fix: build a handover plan from both deployments' frozen role definitions. Stop and retain outgoing singletons before any replacement that could conflict with them starts, including renames and changes to a scalable role. Handle removal when there are no new singletons. On failure, restore an old singleton only after all potentially conflicting replacements are confirmed removed or missing, even when their names differ. This needs coordinated startup and unwind changes.

From reading the code (verified at 7b2220ec6)

• unwind restarts the old scheduler even if removing the new one failed (role-rollout.ts:153-178). Suggest restoring a predecessor only after every potentially conflicting replacement, including renamed roles, is removed or confirmed missing. If that is uncertain, leave the predecessor stopped and report the container IDs. Two tests (role-rollout.test.ts:548, :609) currently assert the restart.
• stopPrevious treats any status other than "running" as stopped (role-runtime.ts:145-160). The presentation status maps running-but-unhealthy to failed, restarting to deploying and paused to stopped. Suggest using the raw sampleStability state and accepting only created, exited, dead or missing.
• Startup check (role-runtime.ts:105-121): completed (exit 0) counts as ok, and a replica whose inspection fails is left out of the findings. Under restart policy no or on-failure, a role that exits 0 can pass this check. If the rest of the deploy succeeds, the old scheduler is retired. Suggest requiring exactly one stable finding per container ID for roles.
• Inherited healthcheck (docker.ts deployRole): Cmd is replaced but Healthcheck isn't. An inherited HTTP healthcheck can mark a worker unhealthy if it probes the web server the role no longer starts. Suggest Healthcheck: { Test: ["NONE"] } on role containers.
• Move/copy guard (project-move.ts:166-174): only project.roles is checked, so roles: [] lifts the guard before the redeploy that retires the running roles. Suggest checking the active deployment's frozen roles and container refs, plus unresolved role work, at both move/copy entry points. That includes reconciling and intentionally preserved cancelled deployments.
• PORT (docker.ts:1128): portless drops only the generated PORT, so a user-set PORT still reaches every role. Suggest filtering PORT=... and appending a bare "PORT" entry in deployRole, leaving other callers of the shared environment helper unchanged.
• Failed retirement: commit() removes the previous roles best-effort, and retention cleanup doesn't touch role containers. A failed removal may leave an old worker consuming jobs with the previous release's code. Suggest at least surfacing it in the deploy's existing warningMessage, naming the container.

Laravel-specific

• In the tested Laravel 12.70.0 image, schedule:work as PID 1 did not exit on SIGTERM. It kept ticking until SIGKILL at the grace deadline. With 90s configured, scheduler shutdown took about 90s. The docs should distinguish signal delivery via exec from whether the command handles that signal. They should also explain that scheduler grace depends on the command and version. (The 90s comes from the Laravel presets I contributed, and a short scheduler grace is probably the better preset value on 12.x.)
• when/unless copied from a preset into openship.json are dropped without a warning (the API rejects them with a 400). Copying both preset entries makes both roles unconditional, and horizon may instead fail if its package is absent. A parser warning would catch that.

Docs (openship-json.mdx)

• :444 suggested wording: "For a normal same-name handover, the recorded predecessor is stopped before its replacement starts. Runtime failures or an interrupted handover can leave containers unresolved. Reconcile them, and restart the scheduler if needed, before redeploying. This does not guarantee exactly-once scheduled execution." Still worth recommending application-level guards such as onOneServer() / withoutOverlapping().
• :402 stopSignal: when it is omitted, the image's STOPSIGNAL applies, and SIGTERM only when the image sets none.
• :403 stopGracePeriod: "the daemon's default" is more accurate than "Docker's 10 seconds". An unset value was killed after about 3s here.
• :447-453: worth saying that a role that dies after the deploy isn't detected or shown yet. A killed worker stayed down for minutes while the deployment showed ready.

Suggested priorities: the reproduced rename issue, coordinated unwind/stop/startup fixes, and accurate singleton documentation. Label-based admission checks need further design around reconciliation and preserved workloads.

[Farahat612]

@sudanese thank you for testing this on a real app, and for turning most of the findings into fixes the same day (#1262, #1263, #1264, #1265, #1266). I re-read main against your list: the role healthcheck, PORT, the strict stop and startup checks, restoring only after confirmed removal, the retire warning, the move guard, the rename handover and the docs wording are all in.

What I see left:

1. Recovery after an interrupted rollout, your first repro. Agreed it needs a reconciliation policy, not a label sweep. I'll write that up as an issue with a proposed policy rather than guess at it in code.
2. A role that dies after the deploy is not watched. I'd like to take this one: add the deployment's role containers to the health watch for single-app projects, so a crashed worker raises an incident the way the web container does.
3. A parser warning when preset-only when/unless appear in openship.json roles.

Are you already on any of these? If not, I'll start with 2.