Deployment task execution engine and lifecycle actions (ADR 0037/0038)

[aimeritething]

Authoritative decision records: ADR 0037 — Execute Deployment Tasks Under Leases and Guarded Transitions and ADR 0038 — Model Deployment Task Lifecycle Actions as Cancel, Redeploy, and Retention (docs/adr). New ubiquitous-language terms (Deployment Task Lease, Deployment Task Cancel Request, Redeploy, Deployment Task Retention) are in CONTEXT.md. Read both ADRs before implementing; this PRD adds the execution plan, user stories, and testing decisions on top of them. Related unchanged ADRs: 0023 (task model — including the status/phase split this PRD relies on), 0024 (canvas projections), 0025 (projection streams), 0028 (task-owned timelines — the readiness rule this work starts enforcing). Product is pre-launch: breaking schema and API changes are allowed, no backwards compatibility work.

Problem Statement

Deployments on the Project Canvas cannot be trusted end to end:

• A server restart or crash leaves a deployment showing "running" forever; nothing detects or resolves it.
• Cancelling a deployment only flips a database status. The runner keeps executing, keeps applying resources, and later overwrites the cancelled status with running or completed. The UI offers no cancel button at all.
• A failed deployment is a dead end: no retry affordance anywhere, so recovery means re-filling the deploy form from scratch. Retrying a template deployment (via the existing API) can deploy a second template instance because instance names are random-suffixed and regenerated per run.
• A deployment whose resources never became ready is still shown as completed, violating ADR 0028.
• Two concurrent actions (or a retry racing a live runner) can double-run the same task with interleaved writes.
• Deployment task state in the UI lives inside a large canvas snapshot hook rather than a store; status-to-presentation mapping is duplicated in at least three places; finished tasks linger in memory.

Grounding Correction (2026-07-06 implementation review)

An earlier revision of this PRD assumed a background worker pool claiming queued tasks. Grounding against the codebase falsified that: every Kubernetes operation a runner performs is authorized by the requesting user's kubeconfig, which exists only in request memory and is never persisted — the server holds no cluster credential (the Deploy Devbox API is the exception: the server mints namespace-scoped JWTs, so devbox pause/delete are server-executable). A worker that claimed a queued task would have no credentials to execute it, and persisting kubeconfigs is out of the question. Consequences threaded through this PRD:

• No worker pool. The lease is claimed inline by the credentialed request: create inserts queued and immediately claims queued → running in the same request, launching the runner in-process with the kubeconfig in memory; input submission claims blocked → running the same way. All safety properties (single execution right, epoch fencing, honest interruption, reaper resolution) are unchanged.
• queued is transient. A row still queued past a short start deadline (crash between insert and claim) is failed by the reaper with a never-started reason — never silently re-run.
• The reaper needs no cluster credentials for any of its duties, and it gains one: pausing devboxes of terminal tasks (via the devbox JWT) whose runner died before pausing.
• Readiness verification stays runner-owned under the lease (the runner has credentials in memory); there is no engine-side readiness observer. The existing read-path completion (snapshot/timeline GETs observing readiness and writing completed — an unfenced status write from the read path) is removed outright.

Implementation Amendments (2026-07-06, second pass)

A two-axis review of the first implementation pass surfaced gaps against this PRD; all were fixed in the same branch except one deliberate deferral. Deltas that refine the sections below:

• Secrets contract completed (US14/US15). The first pass only stopped submitted values from merging into persisted deploymentPlan.args. Now: template source.args are stripped of sensitive keys before insert (create requests may declare sensitiveKeys; a shared name/type heuristic covers undeclared callers; every stripped key name — never a value — is recorded back onto sensitiveKeys so clones know exactly what to re-ask); both the create route and the chat create tool hand the full values to the launched runner in process memory; the template runner fetches the template's input declarations, authoritatively re-scrubs the persisted source, and parks blocked when required inputs are missing or a recorded stripped key has no value in this run's memory — giving the template runner the blocking-input path US15 assumed, degrading safely (strip + task event + forced re-collection, never silently dropped values) when declarations cannot be fetched; every persisted copy of AI deploy output is sanitized — sensitive keys stripped from embedded args maps and known sensitive values scrubbed (raw, JSON-escaped, base64) from the whole output copy, wherever the gateway echoed them; and rendered resource YAML gets the same value scrub before it is persisted in artifactSummary.resourceYamls or completion event payloads. The blocking form always re-asks every sensitive input, not just missing ones. One sensitivity predicate is shared by persist-side stripping, DTO redaction, and form generation.
• Direct runner cancel granularity. The manifest loop now checks the abort signal before each document, implementing "the direct runner stops between resource applications".
• Identity copy is target-guarded. A clone copies result identities only when its effective target matches the predecessor's Project; an edited redeploy that retargets gets fresh identities.
• US10 delivered. "Edit & Redeploy" in the timeline pane header opens the kind-matched deployment pane (template/docker/database/GitHub) prefilled from the predecessor's source, target locked to the predecessor's Project, submitting through create with predecessorTaskId. Prompt-sourced tasks have no pane and stay one-click. The GitHub pane prefills no repo selection (repo is re-picked; submission still clones).
• US12 delivered. A confirm dialog warns before any redeploy whose predecessor recorded applied resources — on dock chips, the timeline pane header, and the template/docker/database pane submits; the GitHub pane shows the warning as a persistent subtitle instead (its submit actions are awaited internally, where a dialog gate would break pending states). No dialog when nothing was applied: one-click stays one-click.
• US26 deferred. The placeholder-node context menu is not built: the canvas has no context-menu infrastructure, and clicking a placeholder already opens the timeline pane where cancel/redeploy live. Revisit alongside canvas context-menu work.
• Status presentation unified. One shared status→hue mapping (running/applying blue, completed green, blocked amber, failed red, queued/cancelled neutral) at -500 tones across dock chips and the timeline pane; timeline step statuses share the semantics, while a result resource card's running stays green (it reports a healthy resource, not progress).

Solution

A deployment-owned execution engine makes every task's lifecycle honest and controllable, with four invariants: the Postgres task row is the only source of truth; a guarded transition is the only status write path; a Deployment Task Lease is the only execution right; a dedicated deployment-tasks store is the only UI read point.

From the user's perspective: statuses always tell the truth (interrupted runs become failed, runs that stop making progress fail after a bounded duration, unready deployments never show completed); Cancel works everywhere an active task exists and stops the workflow without deleting anything it already created; Redeploy recovers any failed or cancelled deployment in one click (optionally with edited settings) without duplicating resources; finished task records clean themselves up after 30 days. The same actions are available from the Deployment Task Dock chips, the Deployment Task Timeline pane, the Deployment Placeholder Node menu (cancel), and assistant chat tools.

User Stories

1. As a platform user, I want a deployment interrupted by a server restart to be marked failed with an "interrupted" reason, so that I act on it instead of watching a stuck spinner forever.
2. As a platform user, I want to cancel a queued or blocked deployment before it runs, so that a mistaken deploy does nothing at all.
3. As a platform user, I want to cancel a running deployment and see it actually stop within seconds, so that cancel is a real control and not a cosmetic status.
4. As a platform user, I want to cancel a deployment that is stuck waiting for resource readiness, so that I can stop waiting instead of sitting out a 10-minute timeout.
5. As a platform user, I want cancellation to never delete or roll back resources that were already applied, so that partial progress is preserved and inspectable.
6. As a platform user, I want the timeline of a cancelled deployment to itemize exactly which resources were applied and which were not, so that I can keep or clean up deliberately from the canvas.
7. As a platform user, I want a "cancelling…" indication between my click and the actual stop, so that I know my request registered.
8. As a platform user, I want a cancel request that the system cannot deliver promptly to still resolve to cancelled within a bounded deadline, so that "cancelling" can never hang forever.
9. As a platform user, I want one-click Redeploy on a failed deployment, so that transient failures (slow image pulls, devbox startup hiccups) do not force me to re-fill forms.
10. As a platform user, I want to adjust settings before redeploying, so that I can fix the cause of the failure in the same gesture.
11. As a platform user, I want a redeploy over preserved resources to converge on those same resources, so that retrying a template deployment can never create a duplicate second instance.
12. As a platform user, I want a warning when a redeploy would overwrite manual edits I made after the failed run, so that I never lose changes silently.
13. As a platform user, I want a deployment blocked on my input to wait indefinitely and keep its dock reminder, so that I can come back to it on my own schedule.
14. As a platform user, I want my submitted secrets (passwords, sensitive template inputs) never written to any database, so that secret hygiene survives the redesign.
15. As a platform user, I want to be re-asked for inputs on redeploy with non-sensitive fields prefilled from the plan's declared defaults, so that convenience never comes from stored submissions.
16. As a platform user, I want a deployment marked completed only when its required result resources are actually ready, so that green means green.
17. As a platform user, I want a readiness timeout to mark the deployment failed with the reason while preserving the created resources, so that I can distinguish "not ready yet" from "succeeded".
18. As a workspace member, I want simultaneous actions by teammates on the same task to resolve deterministically (one wins, the other sees the current state), so that races never corrupt a task.
19. As a platform user, I want double-clicking cancel to be harmless and silent, so that impatience never produces error toasts.
20. As a platform user, I want cancel on an already-finished task to tell me it already finished, so that the system never pretends to honor an impossible intent.
21. As a platform user, I want finished deployment records to purge themselves after 30 days, so that history does not accumulate unbounded.
22. As a platform user, I want a failed deployment to stay noticeable in the dock until I dismiss it or redeploy it, so that failures are never missed.
23. As a platform user, I want a deployment I cancelled myself to not demand my attention, so that reminders are reserved for surprises.
24. As a platform user, I want redeploying to automatically dismiss the predecessor's failure reminder, so that the dock never shows stale alarms for problems I am already fixing.
25. As a platform user, I want cancel and redeploy directly on dock chips and in the timeline pane header, so that recovery is one click from where I notice the problem.
26. As a platform user, I want the canvas placeholder's menu to offer cancel, so that I can stop a deployment from the canvas where I see it.
27. As an assistant chat user, I want chat to create, cancel, and explain deployments through the same lifecycle as the UI, so that chat and canvas never disagree about a task.
28. As a platform user, I want deployment creation responses to reflect the true stored state, so that the UI never renders a state the system is not in.
29. As an operator, I want every finished deployment's devbox paused rather than left running — including when the server died mid-run — so that runs are inspectable without idling compute until the retention purge deletes them.
30. As a maintainer, I want every status write to go through one guarded transition with lease-epoch fencing, so that illegal states and zombie-runner writes are structurally impossible rather than merely unlikely.
31. As a maintainer, I want runners to touch task state only through a fenced task handle, so that runner code cannot bypass the engine.
32. As a maintainer, I want progress streaming to work when the app runs more than one server process, so that scaling out does not silently break deployment progress.
33. As a platform user, I want a deployment that stops making progress to be failed after a bounded maximum duration, so that "running" always means progressing.
34. As a platform user, I want two teammates redeploying the same failure simultaneously to produce exactly one recovery task, so that identity reuse can never race itself onto the same resources.

Implementation Decisions

Governing model. ADR 0037 defines the engine; ADR 0038 defines the action semantics. Everything below is execution detail consistent with them.

Schema (deployment task table). Add cancel_requested_at, lease_owner, lease_expires_at, lease_epoch (bigint), lease_claimed_at (anchor for the maximum active-run duration, reset on every claim so parked blocked time never counts), retried_from_task_id (nullable, no foreign key — predecessors are purgeable). Drop heartbeat_at (lease expiry is the liveness fact). deploy_task_events.seq defaults from a global sequence (the advisory-lock max(seq)+1 allocation is removed; per-task ordering needs monotonicity, not density). Partial indexes cover the reaper's scans (leased statuses by lease expiry, queued by age, terminal by completion time) and a partial unique index on retried_from_task_id over active statuses enforces at most one active clone per predecessor. Allocated result identities are recorded inside artifact_summary as a first-class resultIdentities field (v1: templateInstanceName) — no separate column, replacing today's untyped burial inside the artifacts array; they are persisted with a guarded write at allocation time, before any external apply or provider call uses them, so an identity can never be allocated and then lost to a crash. One-time rollout migration: rows in active statuses are marked failed with an interrupted reason. Events and messages tables are otherwise unchanged.

Transition table (enforced by the single guarded-transition function; anything not listed is rejected):

From	To	Trigger
queued	running	create request claims inline (CAS on lease columns)
queued	cancelled	cancel request (nothing is running)
queued	failed	reaper: start deadline exceeded (crash between insert and claim)
running	blocked	runner requests Blocking Inputs; lease released
running	applying	artifacts generated, apply begins
blocked	running	input submission claims in place (inline claim)
blocked	cancelled	cancel request (immediate)
running, applying	failed	runner failure; readiness timeout; shutdown drain; reaper on expired lease; reaper on exceeded maximum active-run duration
running, applying	cancelled	checkpoint acknowledges cancel; reaper on expired lease with pending cancel; cancel-ack deadline exceeded
applying	completed	required Deployment Result Resource cards ready (ADR 0028)
completed, failed, cancelled	—	terminal; no outgoing transitions

Note: verify is a phase within the applying status (ADR 0023), not a status — cancelling during verify is the applying → cancelled row. Every engine-resolved transition (interrupted, never-started, timeout, forced cancel) also records a task event naming the verdict and reason, so timelines always explain system-decided outcomes.

Engine. An HMR-safe per-process engine runtime starts with the server boot hook (alongside migrations). Claiming and every status write are single-statement conditional updates carrying allowed source statuses and the expected lease_epoch; where a transition and its event must land together, one CTE statement does both. All deadline comparisons happen in SQL against the database clock. The launching process renews each held lease on a fenced timer while the run is in flight, so lease expiry means the owning process died — a slow integration call (slow devbox start, long gateway turn) is never misread as an interruption. A checkpoint() reads the pending cancel request; it is part of the runner's only interaction surface, the task handle: checkpoint, emitEvent, updateTimeline, setState/setArtifacts, requestInputs, complete, fail, acknowledgeCancel — every write epoch-fenced. Cadence constants (tuning, not contract): lease 60s renewed every 20s, checkpoint ≤20s, reaper sweep 30s, cancel-ack deadline 90s, maximum active run 30min, queued start deadline 120s. The reaper sweeps: queued rows past the start deadline → failed (never started); lease-holding statuses (running, applying) for expired leases (→ failed interrupted, or → cancelled when a cancel request is pending), pending-cancel deadlines (→ cancelled regardless of lease liveness), and exceeded run duration (→ failed timeout); terminal tasks whose devbox was never paused → pause via the server devbox JWT; terminal tasks past retention → purge. On shutdown (SIGTERM) the process stops launching, aborts in-flight runners, and resolves its own leases to failed (interrupted: shutdown) with guarded writes — honest state immediately rather than a lease-expiry wait. blocked is a leaseless parked state by design.

Blocked input. Submitted values are never persisted (events keep redacted key names only). The input submission request performs the blocked→running claim itself and hands values to the runner in process memory. The contract is row-level, not DTO-level: submitted values must never appear in the task row (source, deploymentPlan.args, artifact summary), task events, or timeline payloads — sensitive keys are stripped before any persist, and public-DTO redaction stays defense-in-depth rather than the primary guarantee. (Today the runner merges submitted args back into the persisted artifactSummary.deploymentPlan.args and relies on read-time redaction — a live secrets-at-rest defect this work fixes.) The current "failed at configure with pending inputs" resume special case is removed: blocked is the only waiting state. On redeploy, non-sensitive fields prefill from the deployment plan's declared defaults — never from a prior submission; sensitive fields are always re-entered. AI Runner resumption re-attaches to the persisted Deploy Devbox / Codex Gateway session identifiers.

Cancel. Two-phase: queued/blocked cancel immediately; running/applying record the cancel request, the UI shows "cancelling", the runner acknowledges at the next checkpoint with AbortSignal propagated into gateway turns, devbox operations, and readiness waits (when the cancel lands on the process owning the run, the abort fires immediately). The direct runner stops between resource applications; template application is one atomic provider call (cancel lands before or after, never mid-way). Cancel remains available during the verify phase ("stop waiting"). One cancelled terminal status; artifact summary and timeline record nothing/partially/fully applied with itemized resources. After a deadline-forced cancel that itemization is best-effort — a still-live runner's final apply can land after its recording writes are fenced off — so the canvas, not the task record, stays the source of truth for what exists. Devboxes are paused at every terminal outcome: the runner pauses on its own completion/failure/abort paths, the reaper pauses for engine-resolved terminals (server devbox JWT). Cancel never touches applied product resources. Two structural requirements make that hold: the cancel request is publicly readable — the snapshot DTO and the projection expose cancelRequestedAt, so "cancelling" is server-derived state that a refresh or a teammate sees too, with optimistic client state only bridging the request round-trip — and cancellation is a typed runner outcome distinct from failure: an aborted run never enters failure-cleanup paths (today's template failure cleanup deletes partial resources by label selector; an abort must never reach it), never resolves to failed, and reports its applied-resource evidence before exiting.

Redeploy. No retry on a task record. Recovery from failed or cancelled is a new task cloned from the predecessor's Deployment Source and Target. Creation accepts an optional predecessor reference and validates it: a failed/cancelled predecessor clones; an active or completed predecessor returns a conflict carrying its snapshot; a purged or unknown predecessor is a not-found — never a silent fresh create, which would re-allocate identities that preserved resources still hold. A concurrent clone of the same predecessor loses the insert itself (partial unique index) and receives the conflict with the winner's snapshot. The server records lineage and copies the predecessor's recorded result identities onto the clone's row at creation; because identities are recorded first-class at allocation time (notably the random-suffixed template instance name), a redeploy chain keeps them even when an intermediate task never reached artifact generation or the root was purged, and artifact generation reuses a recorded identity instead of allocating a fresh one. Creating a redeploy dismisses the predecessor's dock reminder — surfaces derive supersession from the clone's lineage; dismissal state itself stays client-local in v1. Redeploying onto existing resources shows a one-line overwrite warning. Verbatim clone and edited redeploy (deploy pane prefilled from predecessor source) are the same mechanism.

Retention. No user-facing delete (no endpoint, no affordance). The reaper purges terminal tasks older than 30 days via an internal delete: the per-task devbox first (server devbox JWT), then the row (events/messages cascade), emitting projection removal; any failing step retries on the next sweep (purge is idempotent), so a half-purged task is re-swept rather than stranded. Active and blocked tasks are never purged.

API surface (final). Create (also serving redeploy clones), list (cursor pagination on (createdAt, id) + status filter), snapshot, cancel (POST /:id/cancel), input, canvas-projection patch, and the three existing streams (project projections, per-task timeline, gateway events — shapes unchanged). Removed: the DELETE route (which actually cancelled), the retry route, and the synchronous template-deploy bypass route (confirmed no callers). Idempotency rule: intent already satisfied → success with current snapshot (repeated cancel); intent no longer satisfiable → conflict carrying the snapshot (cancel on a terminal task, clone of a non-terminal predecessor, clone losing the unique-index race); clone of a purged or unknown predecessor → not-found; surfaces reconcile from the snapshot without error toasts. Creation responses report the stored status truthfully.

Read side. One global Postgres LISTEN/NOTIFY channel replaces the in-process listener maps (a per-namespace channel would force workers and listeners into dynamic LISTEN bookkeeping; one channel with id-only payloads and in-process filtering carries the same information). The payload names namespace, project, and task ids only; subscribers re-read the row, and stream handlers subscribe before their bootstrap read so no update falls between snapshot and subscription. The LISTEN connection is dedicated (never a pool connection); on reconnect, subscribers re-bootstrap because notifications during the gap are lost. Retention-purge removals are the one exception to re-reading: the purge notification's ids are themselves the removal event. Project-scoped SSE streams (ADR 0025) are a filtered view; a future workspace-level subscriber (notification center) needs no schema change.

UI. A dedicated deployment-tasks store (bootstrap read + projection stream) becomes the single read point; the canvas snapshot hook is demoted to subscription lifecycle management; visibility/topology selectors move into the store with memoization (fixes stale-task retention and cross-project leakage). One shared status-to-presentation module replaces the three duplicated status switches. One client actions module implements cancel/redeploy with optimistic "cancelling" state bridging the round-trip, cancelRequestedAt from the stream as truth, buttons disabled while in flight. Affordances: dock chip (open timeline, cancel or redeploy, dismiss), timeline pane header (cancel / redeploy), placeholder node menu (open timeline, cancel). Cancelled is not attention-needed (brief dismissible notice); failed keeps attention until dismissed or superseded by a redeploy (supersession derived from lineage on the successor's projection; dismissals stay client-local in v1).

Decomposition. The deployment domain splits into: engine (transitions, lease, reaper, notify, events — the only status writer), runners (direct, template, ai — task handle only, never import storage), integrations (Kubernetes apply, devbox, gateway — stateless, abort-signal aware), storage (schema, db), read-side (projection and timeline builders, NOTIFY→SSE fanout). HTTP routes call engine/read-side. Decomposition is mechanical and lands last, after behavior is correct.

Testing Decisions

A good test exercises external behavior at the highest seam and never asserts on internals (no spying on transition calls, no reading private lease fields except through the read API).

Primary seam (the one new seam): the deployment engine and HTTP route handlers against a real test Postgres, with integrations faked at the integrations module boundary. The repo has no database-backed test harness today; introduce one. Embedded PGlite in-memory Postgres: drizzle ships a PGlite driver, and PGlite supports LISTEN/NOTIFY through its own listen API, so the listener connection sits behind a thin seam; it executes on a single connection, which covers single-statement conditional updates fine but cannot exercise true interleaving. Two guardrails make that sufficient: the engine constrains itself to single-statement conditional updates — no advisory locks and no multi-statement transactions in engine paths (the migrator's advisory lock stays outside the engine) — and adopting the harness starts with a smoke test proving the semantics the engine leans on (conditional-update guards, LISTEN/NOTIFY delivery, partial unique indexes). If either guardrail breaks, upgrade the harness to a real Postgres via testcontainers. Fakes for Kubernetes apply / devbox / gateway are injected at the integrations boundary and record calls + honor abort signals. Assertions go through the same reads production uses: route responses, task snapshot reads, and emitted stream events.

Behaviors that must be covered at this seam: exhaustive transition-table acceptance/rejection; inline claim and renewal; a slow faked integration call (longer than the lease) is not reaped while timer renewal runs; reaper outcomes (expired lease → failed interrupted; expired lease with pending cancel → cancelled; cancel-ack deadline → cancelled with live-runner starvation; exceeded maximum active-run duration → failed timeout; stale queued → failed never-started); fenced writes from a stale lease epoch are no-ops; cancel idempotency matrix (repeat cancel 200, cancel on terminal 409 with snapshot); clone validation matrix (active predecessor → 409 with snapshot, completed predecessor → 409, purged or unknown predecessor → 404, concurrent clone → unique violation surfaced as 409); blocked → inline-claim resume with submitted values never appearing in any row column (including deploymentPlan.args) or event payload; an abort during apply resolving to cancelled with no failure-cleanup deletion issued; redeploy clone (lineage recorded, template instance identity reused, predecessor dock reminder dismissed); chained redeploy A→B→C converging on A's template instance identity when B allocated nothing and A is purged; retention purge cascading events/messages and emitting projection removal; a purge whose devbox deletion fails retrying on the next sweep without deleting rows; readiness timeout ending in failed with resources preserved (fake apply reports created resources, fake readiness never turns ready); snapshot/timeline reads never writing status (read-path completion removal regression).

Secondary seam: runners against a fake task handle, for cooperative-cancel properties: after a checkpoint observes a cancel request, no further writes occur and abort signals reach in-flight fake operations. This seam exists precisely because the task handle is the runner's only capability surface.

UI: the deployment-tasks store against synthetic stream events (snapshot/upsert/remove sequences, grace-window expiry, project switching). Prior art: the existing colocated unit tests for projection/timeline/artifact logic (node:test with strict assert, some bun:test) — follow the same colocated style. Light component tests for action affordances follow the existing deployer component test pattern.

Module boundaries make everything below the primary seam unit-testable without mocks of our own code: integrations are faked as whole modules, never patched internally.

Out of Scope

• Notification center (platform-level surface; separate ADR — may absorb the dock and host deployment history).
• Any deployment history list surface.
• Resume of interrupted tasks (a pre-apply resume whitelist may come later).
• Rollback or deletion of applied resources on cancel.
• Diff detection before redeploying over manually edited resources (warning only).
• Workspace-level task streams, auto-retry policies, idempotency keys for create (the clone-uniqueness index covers redeploy; plain creates rely on button disabling).
• Moving execution to Go or adopting an external queue/workflow engine.
• Server-held cluster credentials of any kind (admin kubeconfig, impersonation) — execution stays attached to credentialed requests.
• Cancelling or purging tasks when their project or namespace is deleted (follow-up: project deletion should cancel its active tasks; a blocked task of a deleted project currently parks forever).
• Dock visual redesign beyond adding actions; CONTEXT.md dock definition unchanged.

Open Questions

• Whether to pause the per-task devbox while a task parks in blocked (an indefinitely parked AI task keeps its devbox running today's way; pausing may invalidate the gateway session, changing resume from re-attach to a new turn). Verify alongside the terminal-pause behavior in phase 2.

Further Notes

Suggested execution order — four independently mergeable phases:

1. Engine core (backend only): schema migration (new columns, heartbeat drop, event-seq sequence, partial + unique indexes, zombie-row cleanup), guarded transition function + transition table, task handle (runners' only write path), inline claim at create/input with timer-based lease renewal, reaper (never-started, interrupted, cancel-intent resolution, ack deadline, max duration, terminal devbox pause, retention), shutdown drain, engine-resolved transitions recording task events, readiness-timeout → failed, truthful create response, one-channel LISTEN/NOTIFY replacing in-process listeners, list pagination/filter. Phase 1 must also migrate every state-writing entry point, or the invariants do not hold — that is: the create route and chat create tool (inline claim + launch under lease), the input route and chat input tool (inline blocked→running claim), the legacy retry route (deleted outright; recovery returns as redeploy in phase 2), the legacy DELETE route (interim guarded immediate cancel until phase 2), and the read-path completion inside snapshot/timeline reads (deleted — it is an unfenced status write from the read path). Acceptance: primary-seam tests for the transition table, claim/reap/renewal (including the slow-integration case), and fencing pass; no code path launches a runner outside an inline claim; no read writes status; a killed dev server no longer leaves running rows (drained at shutdown, or reaped on the next sweep).
2. Lifecycle actions: cancel route (two-phase, typed cancellation outcome, AbortSignal propagation through runners and integrations, terminal devbox pause, cancelRequestedAt exposed on snapshot and projection) replacing the interim DELETE behavior, result identities recorded and persisted at allocation, redeploy clone in create (validation matrix, unique-index race, lineage, identity copy at creation, predecessor dismissal), the sensitive-args persistence fix (row-level secrets contract), chat tool updates, template-deploy bypass removal. Acceptance: cancel/redeploy/idempotency/clone-validation seam tests pass; template redeploy reuses the previous instance name, including across a chain with a purged root; an abort never resolves to failed and never triggers failure cleanup; submitted sensitive values appear in no row column or event payload.
3. UI: deployment-tasks store extraction, shared status module, actions with optimistic updates, affordances on dock chips / timeline header / placeholder menu, cancelled-not-attention behavior, lineage-derived supersession. Acceptance: store tests pass; cancel and redeploy reachable from all three surfaces.
4. Decomposition: engine/runners/integrations/storage/read-side split, dead-code sweep. Acceptance: no behavior change (seam tests still green), runners import no storage.

Known latent defects fixed by design: (1) retrying a template deployment while its resources were preserved allocates a fresh random-suffixed instance name and deploys a second instance; the redeploy identity-reuse rule eliminates it. (2) submitted template args — sensitive values included — are persisted today inside artifactSummary.deploymentPlan.args with only read-time redaction; the row-level secrets contract eliminates it. (3) snapshot/timeline reads complete tasks from the read path with an unfenced write; removed. All deserve dedicated regression tests.

Note on ADR availability: ADRs 0037/0038 and the CONTEXT.md term additions exist as working-tree changes in the local repo, amended 2026-07-06 during PRD review and again for the credential-grounding correction above (inline claims replacing the worker pool, transient queued with start deadline, reaper devbox duties, single notify channel, shutdown drain, event-seq sequence, read-path completion removal, clone-uniqueness index). Implementation sessions on this machine can read them; commit them as part of phase 1 or earlier.

This PRD can be split into per-phase tracker issues later if desired; the phases above are the natural cut lines.