feat(pipeline): a durable job queue

[ralyodio]

Phase 1 of docs/url-first-pipeline.md, and the thing every later phase is blocked on.

JOB_KINDS has described this table since the first commit and nothing ever created it. Work either ran inline in the request — POST /prospects still runs the whole enrich → resolve → research → score → recommend → draft chain before responding — or the worker loop swept its own domain tables directly. Neither survives a batch: one URL fans out to a fetch, an extraction, an identity search per candidate person and a draft per recommendation.

What's here

• migrations/0007_jobs.sql — the table, three indexes.
• packages/pipeline/src/queue.ts — enqueue, claim, complete, fail, reclaim, drain, depth. It knows nothing about what a job does: drainQueue takes a handler, so the dispatcher lives in the server and tests pass a stub.
• apps/server/src/index.ts — an explicit runJob dispatcher and a drain at the end of each tick.
• packages/domain/src/ids.ts — a job id prefix; the domain had no kind for something the schema now stores.

Decisions worth reviewing

A table, not Redis. REDIS_URL sits empty in the vault and unreferenced in the code. Turso is already here and already backed up, and a queue that loses its contents on restart is not a queue.

The claim is atomic. One UPDATE … WHERE id = (SELECT …) RETURNING …, so two workers cannot both win — the second one's status = 'pending' predicate no longer matches. One replica runs today, so this is not yet load-bearing, but it costs nothing and a queue that quietly double-runs jobs is a bad thing to discover later.

Dedupe expires on its own. The unique index is partial, covering only pending and running rows, which makes it a "don't queue this twice right now" guard rather than "never do this again" — a site crawled last week is crawlable today. A duplicate is a reported outcome ({ queued: false }), not a thrown error: one URL pasted twice in a batch should not fail the batch.

Reclaim does not refund the attempt. Jobs abandoned by a dead worker return to the queue after a 15-minute lease, but keep their attempt count — otherwise one poisonous payload that kills its container retries forever and takes down the deployment.

Dead jobs are kept. A row that runs out of attempts becomes failed and holds its payload and last error. Deleting it makes "why did that URL never produce a card" unanswerable.

The drain runs serially, last in the tick. The work behind these jobs is rate-limited by other people's services — GitHub quota, crawl politeness, the model API — so concurrency here would only move the queue into someone else's 429s. limit (25) is the real throttle and bounds how long one tick holds the loop.

Verification

• bun run check — format, typecheck, 410 tests, 0 fail (13 new).
• Tests cover the atomic claim, dedupe suppression and its expiry after completion, run_after delay and ordering, retry-with-backoff, the dead-letter transition, reclaim with and without an expired lease, and that one throwing handler does not stop the rest of the tick.
• Separately verified the incremental migration path the container actually takes: applied 0000–0006 first, then 0007 on top — 7 skipped, 1 applied, all three indexes created, and a re-run applies 0.

Not in this PR

No new job kinds are enqueued yet — nothing calls enqueue in production paths. The dispatcher handles rescore_prospect and process_deletion, and throws by name for anything else. Making POST /prospects asynchronous is Phase 3 work and a behaviour change to an existing endpoint, so it is deliberately separate.

🤖 Generated with Claude Code