docs: an implementation plan for the URL-first pipeline

[ralyodio]

Adds docs/url-first-pipeline.md — the plan for taking OutreachGraph from "paste a GitHub handle" to "paste a company URL, or 100 of them".

Written against the code rather than against the idea, so every claim in it points at a file and a line.

The two findings that shape it

The blocker is not the crawler, it is the absence of a job queue. JOB_KINDS (packages/pipeline/src/jobs.ts:19) is a return-type enum with no table behind it; the worker loop (apps/server/src/index.ts:249) runs only expireSignals and processDeletion; and POST /prospects executes the whole pipeline synchronously inside the request. At 100 URLs that is not a slow request, it is an impossible one.

All the new work sits in front of the existing chain. Once a URL has produced a PersonCandidate, resolve → research → score → recommend → draft → approve already run. That is what makes this a tractable project rather than a rewrite.

Two supporting facts worth knowing before anyone argues about scope:

• website/observe and website/refresh_research are already research_only in the capability matrix, reason "Permitted public web retrieval" (capability-matrix.ts:175). Homepage crawling is a sanctioned capability with no adapter behind it, not a new policy category.
• enrichWithWaterfall is built and tested but unused — the API passes providers: [] (app.ts:365). Wiring providers in is configuration, not construction.

Shape

Four phases: job queue → site provider → company-first endpoints → network fan-out. Phase 2 is the high-risk one and needs a decision before code, because extraction quality is the product: deterministic parsing is cheap and brittle, an LLM pass is robust and costed per URL, and a hybrid is likely.

Docs only — no code changes.

🤖 Generated with Claude Code