feat(orchestration): admission queue with deferred pickup (#441)#544
Open
nizar-lahlali wants to merge 1 commit into
Open
feat(orchestration): admission queue with deferred pickup (#441)#544nizar-lahlali wants to merge 1 commit into
nizar-lahlali wants to merge 1 commit into
Conversation
When a task hits the per-user concurrency cap, park it in a new QUEUED state instead of failing it (the #331 mass-fail mode). A scheduled AdmissionQueuePickup Lambda drains the queue FIFO (by created_at) as slots free up, flipping QUEUED -> SUBMITTED and re-invoking the orchestrator, whose atomic admissionControl stays the single writer of the concurrency counter — a lost pickup race harmlessly re-queues without losing FIFO position. - task-status: new QUEUED state (pre-active — holds no slot), with SUBMITTED <-> QUEUED transitions plus QUEUED -> CANCELLED/FAILED - orchestrator: queueTask() replaces failTask on cap; queued_at is stamped once, admission_attempts increments per pass; Linear/Jira channel feedback now says 'queued' and fires only on first entry - reconcile-admission-queue: FIFO drain per user with read-only capacity pre-check, conditional flips (cancel-safe), 24h max-age backstop, and orchestrator re-invoke carrying a pickup nonce - reconcile-stranded-tasks: age by time-in-current-status so a task that waited in the queue longer than the stranded timeout is not killed right after pickup - API/CLI: GET /tasks/{id} computes read-time queue_position + estimated_wait_s (fail-open); bgagent status/detail render a 'Queue: position N (est. wait ~Xm)' line - cancel: QUEUED is cancellable everywhere (REST already generic; Slack cancel list extended); no concurrency release needed - idempotent replay returns the existing QUEUED task unchanged - tests: state-machine invariants, queueTask, pickup handler (incl. #331 fan-out-burst regression: burst above cap survives and drains FIFO with zero failures), get-task queue position, CLI rendering Closes #441
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #441. Successor to #331 (fan-out mass-fail against the concurrency cap).
What
When admission hits the per-user concurrency cap, the task is now queued, not failed: it transitions
SUBMITTED → QUEUED(a new pre-active state that holds no concurrency slot) and a scheduled AdmissionQueuePickup Lambda drains the queue in FIFO order (bycreated_at) as slots free up — flippingQUEUED → SUBMITTEDand re-invoking the orchestrator.Design highlights
admissionControlconditional-increment remains the only writer of the concurrency counter. A pickup that loses the admission race harmlessly re-queues (created_atnever changes, so FIFO position is preserved). This avoids introducing a second writer — the class of drift bug behind Fan-out orchestration mass-fails against the per-user concurrency cap (no queue/retry) #331.QUEUEDis cancellable via REST (already status-generic) and Slack (list extended). No compute or slot to release.Idempotency-Keyreturns the existingQUEUEDtask (200) — dedupe path is status-agnostic and untouched.QUEUE_MAX_AGE_SECONDS(default 24 h) is failed with a clear message. The stranded-task reconciler now ages tasks by time-in-current-status so a long queue wait isn't mistaken for a strandedSUBMITTEDtask right after pickup.API / CLI surface
TaskRecord:queued_at,admission_attempts(internal bookkeeping).GET /tasks/{id}:queued_at,queue_position(1-based FIFO rank among the caller's queued tasks, computed at read time, fail-open),estimated_wait_s(coarse heuristic).bgagent status/ detail view renderQueue: position N (est. wait ~Xm)whileQUEUED.check-types-syncpasses.#331 regression test
reconcile-admission-queue.test.tsincludes a fan-out-burst regression: 5 children queued above a cap of 3 all survive asQUEUED, drain FIFO (3 picked up, 2 remain), and zero tasks fail.Testing
mise run buildpasses end-to-end (agent quality, CDK compile + synth + 2265 tests across 125 suites, CLI 576 tests, docs build + sync, type/constants drift checks).queueTask(first-entry vs re-queue, race, no counter writes), pickup handler (FIFO, capacity bounds, per-user isolation, races, invoke-failure, max-age backstop, pagination), get-task queue position (rank, fail-open, left-queue race), CLI rendering.Notes for reviewers
ceil(position / cap) × avg-task-duration, default 600 s, env-tunable) — not a promise.AdmissionQueuePickupruns every minute; an empty-queue cycle is a single GSI query.FAILED+admission_rejectedevent as terminal will now seeQUEUED+admission_queuedinstead.