Autonomous competitive intelligence. Scheduled agents watch competitor pages, detect pricing and positioning changes, resolve them into a temporal knowledge graph, and compile a weekly brief with every claim linked to the change that produced it.

Design in one page

Property	How
Exactly-once handoff	Jobs are Postgres rows claimed with select ... for update skip locked, so enqueue joins the transaction that caused it
Cost control	Two thresholds: bytes must change before diffing, an event must score 70 before a browser opens
Untrusted content	Never parsed in the cluster. Rendered in a hosted browser over CDP, or a per-run sandbox, both outside it
Web access	Every fetch can route through a Bright Data zone, with the zone and country carried in the proxy username
Blocking	A challenge page is detected and discarded before detection sees it, so a block never reads as a change
Agents	One loop, one AgentSpec. Adding an agent is a row, not a runtime
Crash safety	Every agent step checkpoints; the checkpoint carries a full spec snapshot so resume is self-contained
Degradation	No model key runs playbooks and a deterministic brief. No browser key runs in-process Chromium. No proxy fetches direct. No Langfuse logs stdout spans

Decisions are recorded in docs/adr/. Two have been superseded and say so: browsers moved off Modal to a pluggable backend, and per-domain pacing moved from Redis into Postgres.

robots.txt is advisory by default. ROBOTS_POLICY=enforce restores gating; either way the verdict and any Crawl-delay are fetched and recorded, because the delay is a useful signal for staying unblocked even when the directive is not obeyed. Per-domain pacing always applies.

Pipeline

Eleven job kinds do the work, plus a noop that proves the queue end to end. One dispatcher. A job that must wait for politeness is deferred to its earliest allowed time and given back the attempt it consumed, so pacing never eats a retry budget. A job left running by a dead worker is reclaimed by the ticker.

Job	Trigger	Does
robots_refresh	verdict older than a day	Fetches robots.txt, sets robots_ok and crawl_delay_s
sentinel_check	tier policy cadence	Conditional GET with stored ETag; escalates on content-hash change. A paused page is never scheduled
render_check	hash moved	Skeleton, diff, classify, persist evidence and events
explore	event scores 70+	Drives a real browser through a playbook, captures interactive states
index_event	each new event	Resolves facts, embeds for recall, queues alerts
deliver_alert	event scores 80+	Log and optional webhook, deduped per channel
compile_brief	weekly, per org	Deterministic render, or an editor agent with three read-only tools
resume_run	a parked run is due	Continues an agent from its checkpoint
geo_check	page has regions	Fetches the page from each country and diffs prices between them
serp_check	on demand, per query	Records search results and turns ad changes into campaign events
discover_competitors	on demand	Turns organic results into competitor candidates for review

Detection

Everything left of ChangeEvent is pure, which is why evals/detectors can score it. The gate holds precision and recall at 1.0 across seven cases, three of which expect no events: a reformatted price and a reordered feature list are the failures that matter.

Facts are bitemporal. Writing a value closes the previous one by stamping valid_to, so plan:Pro monthly_price reads as a history. That is also how a recovered outage is expressed: availability is an incident with an identity, and recovery writes available = true.

Web access

Fetching, rendering and search all route through Bright Data. The zone and any country target live in the proxy username, so one credential set serves every geo.

Path	Zone type	Used for
Proxy fetch	dc, res_static	Tier 0 and tier 1. Conditional GETs still return 304, so the cheap sentinel model survives
Hosted browser	browser_api	Tier 2, over CDP. No sandbox, image or shim of our own
SERP	serp	Paid-ad tracking and competitor discovery

Three things the API's behaviour forces, each of which cost a debugging session:

• Geo is a request field, not gl. Google answers a gl parameter from a proxy exit with a consent redirect, which the zone reports as a captcha. Passing country instead localises the search properly.
• The transport status is not the search status. A failed search still returns HTTP 200; the real outcome is the inner status_code, and the results arrive as a JSON string inside body.
• Country targeting is a zone capability. A residential zone without per-country IPs answers no IPs in selected country, so GEO_POOL points at the datacenter zone.

The SERP zone intermittently answers 502 captcha or 502 expect_body for any country, and 429 when pushed. Searches retry with backoff, and a rate limit waits considerably longer than a one-off captcha.

Dashboard

Sign in is Google through Supabase, using the PKCE flow so the session can live in httpOnly cookies rather than in client storage. The browser never holds a token: server components read the cookie, and anything the browser needs from the API goes through a same-origin proxy that attaches it. A Google account is not yet a Scout account, so the first sign-in provisions a workspace.

Surface	What it does
Changes	The feed. Rows banded by significance, grouped by day, every filter held in the URL so a view is shareable and the back button works
Evidence	Before and after for any change, the stored page rendered in a sandboxed frame, with the tier, content hash and captured browser states
Competitors	Cadence, fetch route, regions and mutes per page. Check now, pause, or delete with the consequences stated
Facts	Bitemporal history. A plan's price reads as a timeline, current value first, each superseded value showing when it closed
Briefs	The weekly brief, with every [evt:…] citation resolved into a link that opens the change behind it
Alerts	What was delivered and why, linked back to the change that raised it
Cost	Spend by agent, changes per day, queue depth and recent runs

Chart colours are not the interface colours. The teal that carries text fails the chroma floor as a data fill and reads gray, so marks use their own steps, validated for lightness, chroma, colour-vision separation and contrast against each surface in both themes.

Layout

packages/schemas     Pydantic models, enums, settings (dependency root)
packages/db          engine, queue primitive, Alembic migrations
packages/harness     agent loop, context assembler, tool registry, browser protocol
packages/detectors   skeleton, diff, classify, experiments (no IO)
packages/memory      vector store, fact resolution, price matrices
packages/storage     object storage behind one protocol, shared by api and worker
services/api         FastAPI: auth, onboarding, intel, briefs, evidence, captures, ops, settings
services/worker      queue consumer, ticker, fetchers, proxies, geo, serp, handlers
web                  Next.js dashboard: marketing pages, Google sign-in, the app shell
evals/detectors      precision gate, run in CI
infra                docker, daytona, modal, k8s manifests, terraform, RLS policies

Running it

cp .env.example .env
make up                                     # postgres, qdrant, api :8000, worker, web :3000
make db-migrate

Open localhost:3000, sign in with Google, and add a competitor. The workspace is created on first sign-in and everything after that is reachable from the dashboard. make ticker runs a scheduling pass immediately rather than waiting for the cadence.

For the API on its own, make dev-token seeds an org and prints a bearer token, and make demo-token prints one for the workspace most recently created in the UI, so curl and the browser act on the same org. make seed-pages URL=... adds a page without the UI.

make ticker                                 # robots first, then schedule
make test                                   # stop the worker first: it shares the database

Host-side checks without Docker: uv sync, then make lint, make typecheck, uv run python evals/detectors/gate.py. CI additionally builds the dashboard and runs terraform fmt, init and validate against the pinned provider.

Tier 2 needs a browser. With BRIGHTDATA_BROWSER_PUPPETER_PLAYWRIGHT_HOST set there is nothing to install; locally, uv sync --extra browser runs Chromium in process. The Daytona path is still supported (make daytona-snapshot, then make daytona-check URL=...), but its egress is default-deny on lower tiers, so a competitor domain can resolve and still be refused.

Configuration

Everything optional degrades rather than fails. Nothing below is required for local development.

Variable	Effect when unset
OPENAI_API_KEY	No model triage; playbook-only exploration, deterministic briefs
BRIGHTDATA_API_KEY	No SERP: ad tracking and discovery are unavailable
DATA_CENTER_PROXY_*, DOMESTIC_PROXY_*	Fetches go direct from the worker, unproxied and easier to block
BRIGHTDATA_BROWSER_PUPPETER_PLAYWRIGHT_HOST	Falls back to Daytona, then to in-process Chromium
GEO_REGIONS and a page's regions	Geo capture is off; no cross-country price comparison
DAYTONA_API_KEY	One fewer browser backend
SUPABASE_URL, SUPABASE_SERVICE_KEY	Evidence on local disk; the worker refuses to start outside local env
SUPABASE_JWT_SECRET	Local development secret; the API refuses to start outside local env
SUPABASE_ANON_KEY	Sign-in is unavailable; the dashboard says so rather than failing
APP_URL	The Google callback URL is inferred from the request, which is wrong behind a proxy
LANGFUSE_*	Structured stdout spans instead of traces
SENTRY_DSN_FASTAPI	No error reporting from the API, worker or ticker
SENTRY_DSN_NEXT	No error reporting from the dashboard
MIN_CRAWL_INTERVAL_S	No politeness floor beyond robots Crawl-delay
CORS_ORIGINS	Only localhost:3000 may call the API
KILL_SWITCH	Set true to stop the ticker and fail every agent run

Observability

Every agent step is a trace span. With Langfuse keys set they go to Langfuse as a trace with one observation per step and per model call; without them the same spans are written as structured stdout JSON, so the shape of what is recorded never depends on configuration.

Errors are split by surface: SENTRY_DSN_FASTAPI covers the API, worker and ticker, SENTRY_DSN_NEXT covers the dashboard, so a frontend error never lands in the worker's issue stream. Both send sendDefaultPii: false, because breadcrumbs here carry page URLs and job payloads. The dashboard DSN is inlined at build time like every NEXT_PUBLIC_ value, so it travels as a build arg rather than as deployment environment.

Security model

Org isolation is enforced twice. Every API route derives its org from the caller's membership row, never from client input, and row level security repeats the constraint in the database for all fifteen org-scoped tables, including those reached only through a competitor. The API holds the service role and bypasses RLS, so the policies exist for anything arriving with a user JWT.

tests/test_rls_policies.py applies the real policy file to a throwaway transaction and queries it as an unprivileged role, because a superuser bypasses row security no matter what the policies say.

Sessions are httpOnly cookies, never client-readable storage, so a script on the dashboard cannot read one. The API accepts both the asymmetric tokens Supabase issues and the symmetric development tokens, verifying the former against the project's key set rather than a shared secret.

Stored competitor HTML is served back with Content-Security-Policy: sandbox, so evidence can be viewed without executing against our origin. The proxy that serves it to the dashboard forwards that header rather than dropping it, which is what keeps hostile markup from running on our own origin: a planted snapshot that tries to write a cookie, navigate the top frame or read the API is blocked, and the frame reports an opaque origin. Playbooks are interpreted JSON, never generated code, and browser navigation is confined to the page's own host inside the invoker rather than in a prompt.