@@ -57,7 +57,7 @@ On-call escalation is explicitly a later phase.
5757## 2. Objectives
5858
5959### 2.1 Primary Goals
60- - Detect downtime for HTTP(S), TCP, PING, keyword, and SSL-expiry checks.
60+ - Detect downtime for HTTP(S), TCP, keyword, and SSL-expiry checks (ICMP PING is Phase 2 — see §12) .
6161- Alert within one confirmed check cycle across multiple channels; alert on recovery.
6262- Free tier of ** 20 monitors** at 60s interval, no credit card.
6363- Public status page per project (uptime %, response times, incidents).
@@ -84,8 +84,8 @@ On-call escalation is explicitly a later phase.
8484| ** HTTP(S)** | Status code, response time, redirect handling | URL, expected status, timeout, follow-redirects |
8585| ** Keyword** | HTTP body contains / omits a string | URL, keyword, match mode |
8686| ** SSL expiry** | Cert days-to-expiry warning | Host, warn-days (default 14) |
87- | ** TCP** | Port open | Host, port |
88- | ** PING (ICMP)** | Host reachability | Host / IP |
87+ | ** TCP** | Port open (outbound ` connect() ` , works from Railway) | Host, port |
88+ | ** PING (ICMP)** * (Phase 2) * | Host reachability — needs raw sockets, so runs on the §12 prober droplet, not Railway | Host / IP |
8989
9090Each monitor: name, type, target, interval (60s free / 30s paid), timeout,
9191expected-result config, channel(s), enabled flag, and optional link to the
@@ -212,8 +212,8 @@ All RLS-scoped to org/project, consistent with existing tables.
212212| ** M1 — Core loop** | HTTP + keyword + SSL monitors, due-time sweep in worker, state machine, email + webhook alerts, monitor CRUD UI |
213213| ** M2 — Channels + status page** | Slack, Discord, public status page, incident history, uptime % |
214214| ** M3 — Plans + limits** | Fold into existing plan tiers, 20-free enforcement, SMS (Twilio) + caps, custom domains |
215- | ** M4 — Polish** | TCP/PING checks, maintenance windows, schedule gates, weekly summary email |
216- | ** Phase 2** | On-call escalation policies, multi-region probing, PagerDuty/Teams/Jira, Playwright journeys |
215+ | ** M4 — Polish** | TCP checks, maintenance windows, schedule gates, weekly summary email |
216+ | ** Phase 2** | On-call escalation policies, multi-region probing, PagerDuty/Teams/Jira, Playwright journeys, ** ICMP PING checks (via §12 prober droplet) ** , ** exposed-services / port-drift check (§12) ** |
217217
218218---
219219
@@ -224,5 +224,92 @@ All RLS-scoped to org/project, consistent with existing tables.
224224 optional ` project_id ` , counts against org limit.)
2252253 . Fold into existing CrawlProof plans, or introduce an uptime add-on SKU?
2262264 . Status page: reuse existing public-report subdomain scheme, or new namespace?
227- 5 . Do TCP/PING (ICMP) checks work from the Railway runtime, or do they need an
228- external prober? (May push PING to Phase 2.)
227+ 5 . ** Resolved:** TCP works from Railway (outbound ` connect() ` ); ** ICMP PING does
228+ not** (needs raw sockets), so PING moves to ** Phase 2** and runs on the §12
229+ prober droplet alongside the port-drift scans. nmap is also Phase 2 (§12).
230+
231+ ---
232+
233+ ## 12. Phase 2 — Exposed-Services / Port-Drift Check
234+
235+ A lightweight attack-surface monitor: on a verified-owned host, detect ports that
236+ are open to the public internet but * shouldn't* be — e.g. a Redis (6379) or
237+ Postgres (5432) accidentally exposed. Framed as ** security drift detection** , not
238+ a scanner.
239+
240+ > ** Value framing:** "You told us to watch 80/443. We now also see ** 6379 (Redis)**
241+ > reachable from the public internet on your verified host — likely a
242+ > misconfiguration." Alerts fire when the open-port set * changes* from an accepted
243+ > baseline, not merely because a port is open.
244+
245+ ### 12.1 Hard constraints (why this is Phase 2, not V1)
246+ - ** Owned targets only.** Reuse CrawlProof's existing ** domain-ownership
247+ verification** as the gate. No arbitrary hostnames — this must never become
248+ port-scanning-as-a-service against third parties.
249+ - ** Not from the Railway app IP.** Scans run on a ** self-hosted prober (a
250+ DigitalOcean droplet)** , never from the Railway service, so an egress-abuse flag
251+ can't take down production. GCP and Railway AUPs prohibit network scanning;
252+ scanning from the app IP risks the service's IP. The droplet has its own IP
253+ reputation and raw-socket access. See §12.3 for the job-transport design.
254+ - ** Bounded + TCP-connect only.** Curated ** top-~ 100 common service ports** , never
255+ full 65535; TCP ` connect() ` only (containers lack ` CAP_NET_RAW ` for SYN scans
256+ anyway); ICMP discovery skipped (` -Pn ` -equivalent).
257+ - ** Rate-limited + infrequent.** Daily cadence, not per-minute; per-org caps.
258+
259+ ### 12.2 Behavior
260+ - Establish a ** baseline** open-port set per host on first scan (user confirms
261+ "these are expected").
262+ - On each subsequent scan, diff against baseline. ** New open port → alert**
263+ (down-alert-style, same channels). Closed expected port → optional info alert.
264+ - Record findings as incidents/events so they show in history and (optionally) a
265+ private security view — ** not** the public status page by default.
266+
267+ ### 12.3 Prober architecture — BullMQ over Redis, prober stays in the monorepo
268+
269+ Job transport is a ** BullMQ queue backed by Redis** , not synchronous RPC (scans
270+ take seconds-to-minutes and must survive restarts). BullMQ gives us retries +
271+ backoff, concurrency control, and ** repeatable jobs** for the daily cadence — so
272+ no hand-rolled leasing/reaper is needed.
273+
274+ ```
275+ Railway (producer) Redis (broker) DO droplet (prober)
276+ ────────────────── ────────────── ───────────────────
277+ API / worker enqueues ──▶ "prober" queue ◀── BullMQ Worker dials OUT
278+ port-scan jobs (rediss:// TLS) over rediss://, runs
279+ (repeatable = daily) nmap -sT -Pn --top-ports
280+ 100, returns result
281+ Railway QueueEvents ◀─── job "completed" ◀── (result = job return value)
282+ handler persists to
283+ Supabase, diffs
284+ baseline, fires alerts
285+ ```
286+
287+ - ** Redis is new infra.** CrawlProof has no Redis today (the worker uses
288+ in-container loopback HTTP). Add one (Railway Redis plugin or Upstash) exposed
289+ over ` rediss:// ` with auth. The droplet connects ** outbound only** to Redis —
290+ no inbound port on the prober box (firewall to egress 443/6380 only).
291+ - ** Prober lives in the monorepo** as a new workspace (mirrors the existing
292+ ` worker/ ` ), e.g. ` prober/ ` — its own ` package.json ` and entrypoint, sharing the
293+ ** job-payload types from ` lib/ ` ** so producer and prober can't drift.
294+ - ** Deploy is self-bootstrapping.** The ` deploy-prober ` GitHub Action fires on
295+ merges to ` master ` (droplet ` ubuntu@scan.crawlproof.com ` ): rsync ` prober/ ` +` lib/ `
296+ to the box, then run ` prober/deploy/provision.sh ` — an ** idempotent** script
297+ that installs nmap/Node/build tools, writes the Redis env file, installs the
298+ ` crawlproof-prober ` systemd unit, builds, and restarts. First run provisions a
299+ bare droplet; later runs update. ** No manual SSH/setup on the droplet, ever.**
300+ - ** Results flow back via the job return value** — the droplet writes no DB. A
301+ Railway-side ` QueueEvents ` (or a second in-process Worker) handles ` completed `
302+ /` failed ` , then persists results, diffs the baseline, and dispatches alerts.
303+ This keeps Supabase credentials off the droplet.
304+ - ** Credential blast radius:** the droplet holds only the ` rediss:// ` URL. Scope
305+ it with a ** dedicated Redis instance (or ACL user)** limited to the ` prober `
306+ queue keys, so a compromised droplet can't read other queues.
307+ - ** Repeatable jobs** drive the daily per-monitor scan schedule; per-org caps and
308+ ` attempts ` /backoff are BullMQ config, not custom code.
309+
310+ ### 12.4 Open items
311+ - Which port list ships as the default "watch" set.
312+ - Whether findings feed CrawlProof's existing audit/finding surface or a new one.
313+ - Redis host choice (Railway plugin vs. Upstash) and TLS/ACL setup.
314+ - One droplet = single vantage point + SPOF (fine for daily scans); the BullMQ
315+ Worker model scales to N droplets with zero config change if redundancy is wanted.
0 commit comments