Skip to content

Commit e2a65d5

Browse files
ralyodioclaude
andauthored
Uptime PRD §12: port-drift check + self-bootstrapping prober deploy (#76)
* docs: add Phase 2 exposed-services / port-drift check to uptime PRD Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: specify BullMQ/Redis prober transport, monorepo prober workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: add deploy-prober workflow + reference systemd unit Deploys the port-drift prober to the DO droplet on merges to master: SSH in, git reset to origin/master, npm ci + build the prober workspace, restart the crawlproof-prober systemd service. Path-filtered to prober/** and lib/**. Requires DROPLET_HOST/USER/SSH_KEY secrets. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: make prober deploy self-bootstrapping + idempotent deploy-prober now rsyncs prober/+lib/ to ubuntu@scan.crawlproof.com and runs an idempotent provision.sh that installs nmap/Node/build tools, writes the Redis env file, installs the systemd unit, builds, and restarts the service. First run provisions a bare droplet; no manual SSH setup ever. ubuntu user's default passwordless sudo does the privileged steps. Secrets: DROPLET_SSH_KEY, PROBER_REDIS_URL. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: resolve TCP vs PING placement — ICMP PING is Phase 2 (prober droplet) TCP connect checks work from Railway (V1/M4); ICMP PING needs raw sockets so it runs on the §12 prober droplet in Phase 2, alongside nmap port-drift scans. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent ddf1a69 commit e2a65d5

4 files changed

Lines changed: 236 additions & 7 deletions

File tree

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: deploy-prober
2+
3+
# Deploys the port-drift prober (docs/uptime-monitoring-prd.md §12) to the
4+
# DigitalOcean droplet ubuntu@scan.crawlproof.com on merges to master.
5+
#
6+
# FULLY SELF-BOOTSTRAPPING + IDEMPOTENT: the workflow installs every system
7+
# dependency (nmap, Node 20, build tools), writes the Redis env file, installs
8+
# the systemd unit, builds, and (re)starts the service on EVERY run. First run
9+
# provisions a bare droplet; later runs just update. You never SSH in to set up.
10+
#
11+
# The prober connects OUTBOUND to Redis — nothing here opens an inbound port on
12+
# the droplet (only sshd, which already exists).
13+
#
14+
# Required repo secrets:
15+
# DROPLET_SSH_KEY - private SSH key for ubuntu@scan.crawlproof.com
16+
# PROBER_REDIS_URL - rediss://... broker URL (scoped to the "prober" queue)
17+
18+
on:
19+
push:
20+
branches: [master]
21+
paths:
22+
- "prober/**"
23+
- "lib/**" # prober shares job-payload types from lib/
24+
- ".github/workflows/deploy-prober.yml"
25+
workflow_dispatch: # allow manual redeploys
26+
27+
# Never let two deploys race on the droplet.
28+
concurrency:
29+
group: deploy-prober
30+
cancel-in-progress: false
31+
32+
env:
33+
DEPLOY_HOST: scan.crawlproof.com
34+
DEPLOY_USER: ubuntu
35+
DEPLOY_PATH: /home/ubuntu/crawlproof.com
36+
37+
jobs:
38+
deploy:
39+
name: provision + deploy prober
40+
runs-on: ubuntu-latest
41+
steps:
42+
- uses: actions/checkout@v4
43+
44+
- name: Configure SSH
45+
run: |
46+
install -m 700 -d ~/.ssh
47+
printf '%s\n' "${{ secrets.DROPLET_SSH_KEY }}" > ~/.ssh/id_deploy
48+
chmod 600 ~/.ssh/id_deploy
49+
ssh-keyscan -H "$DEPLOY_HOST" >> ~/.ssh/known_hosts 2>/dev/null
50+
51+
- name: Ship prober + lib to droplet
52+
run: |
53+
ssh -i ~/.ssh/id_deploy "$DEPLOY_USER@$DEPLOY_HOST" "mkdir -p '$DEPLOY_PATH'"
54+
# --delete keeps the droplet in sync; node_modules/dist are excluded so
55+
# they are neither shipped nor deleted (rebuilt on the box).
56+
rsync -az --delete \
57+
-e "ssh -i ~/.ssh/id_deploy" \
58+
--exclude node_modules --exclude dist \
59+
prober lib \
60+
"$DEPLOY_USER@$DEPLOY_HOST:$DEPLOY_PATH/"
61+
62+
- name: Provision + restart (idempotent)
63+
run: |
64+
ssh -i ~/.ssh/id_deploy "$DEPLOY_USER@$DEPLOY_HOST" \
65+
"REDIS_URL='${{ secrets.PROBER_REDIS_URL }}' DEPLOY_PATH='$DEPLOY_PATH' bash '$DEPLOY_PATH/prober/deploy/provision.sh'"

‎docs/uptime-monitoring-prd.md‎

Lines changed: 94 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ On-call escalation is explicitly a later phase.
5757
## 2. Objectives
5858

5959
### 2.1 Primary Goals
60-
- Detect downtime for HTTP(S), TCP, PING, keyword, and SSL-expiry checks.
60+
- Detect downtime for HTTP(S), TCP, keyword, and SSL-expiry checks (ICMP PING is Phase 2 — see §12).
6161
- Alert within one confirmed check cycle across multiple channels; alert on recovery.
6262
- Free tier of **20 monitors** at 60s interval, no credit card.
6363
- Public status page per project (uptime %, response times, incidents).
@@ -84,8 +84,8 @@ On-call escalation is explicitly a later phase.
8484
| **HTTP(S)** | Status code, response time, redirect handling | URL, expected status, timeout, follow-redirects |
8585
| **Keyword** | HTTP body contains / omits a string | URL, keyword, match mode |
8686
| **SSL expiry** | Cert days-to-expiry warning | Host, warn-days (default 14) |
87-
| **TCP** | Port open | Host, port |
88-
| **PING (ICMP)** | Host reachability | Host / IP |
87+
| **TCP** | Port open (outbound `connect()`, works from Railway) | Host, port |
88+
| **PING (ICMP)** *(Phase 2)* | Host reachability — needs raw sockets, so runs on the §12 prober droplet, not Railway | Host / IP |
8989

9090
Each monitor: name, type, target, interval (60s free / 30s paid), timeout,
9191
expected-result config, channel(s), enabled flag, and optional link to the
@@ -212,8 +212,8 @@ All RLS-scoped to org/project, consistent with existing tables.
212212
| **M1 — Core loop** | HTTP + keyword + SSL monitors, due-time sweep in worker, state machine, email + webhook alerts, monitor CRUD UI |
213213
| **M2 — Channels + status page** | Slack, Discord, public status page, incident history, uptime % |
214214
| **M3 — Plans + limits** | Fold into existing plan tiers, 20-free enforcement, SMS (Twilio) + caps, custom domains |
215-
| **M4 — Polish** | TCP/PING checks, maintenance windows, schedule gates, weekly summary email |
216-
| **Phase 2** | On-call escalation policies, multi-region probing, PagerDuty/Teams/Jira, Playwright journeys |
215+
| **M4 — Polish** | TCP checks, maintenance windows, schedule gates, weekly summary email |
216+
| **Phase 2** | On-call escalation policies, multi-region probing, PagerDuty/Teams/Jira, Playwright journeys, **ICMP PING checks (via §12 prober droplet)**, **exposed-services / port-drift check (§12)** |
217217

218218
---
219219

@@ -224,5 +224,92 @@ All RLS-scoped to org/project, consistent with existing tables.
224224
optional `project_id`, counts against org limit.)
225225
3. Fold into existing CrawlProof plans, or introduce an uptime add-on SKU?
226226
4. Status page: reuse existing public-report subdomain scheme, or new namespace?
227-
5. Do TCP/PING (ICMP) checks work from the Railway runtime, or do they need an
228-
external prober? (May push PING to Phase 2.)
227+
5. **Resolved:** TCP works from Railway (outbound `connect()`); **ICMP PING does
228+
not** (needs raw sockets), so PING moves to **Phase 2** and runs on the §12
229+
prober droplet alongside the port-drift scans. nmap is also Phase 2 (§12).
230+
231+
---
232+
233+
## 12. Phase 2 — Exposed-Services / Port-Drift Check
234+
235+
A lightweight attack-surface monitor: on a verified-owned host, detect ports that
236+
are open to the public internet but *shouldn't* be — e.g. a Redis (6379) or
237+
Postgres (5432) accidentally exposed. Framed as **security drift detection**, not
238+
a scanner.
239+
240+
> **Value framing:** "You told us to watch 80/443. We now also see **6379 (Redis)**
241+
> reachable from the public internet on your verified host — likely a
242+
> misconfiguration." Alerts fire when the open-port set *changes* from an accepted
243+
> baseline, not merely because a port is open.
244+
245+
### 12.1 Hard constraints (why this is Phase 2, not V1)
246+
- **Owned targets only.** Reuse CrawlProof's existing **domain-ownership
247+
verification** as the gate. No arbitrary hostnames — this must never become
248+
port-scanning-as-a-service against third parties.
249+
- **Not from the Railway app IP.** Scans run on a **self-hosted prober (a
250+
DigitalOcean droplet)**, never from the Railway service, so an egress-abuse flag
251+
can't take down production. GCP and Railway AUPs prohibit network scanning;
252+
scanning from the app IP risks the service's IP. The droplet has its own IP
253+
reputation and raw-socket access. See §12.3 for the job-transport design.
254+
- **Bounded + TCP-connect only.** Curated **top-~100 common service ports**, never
255+
full 65535; TCP `connect()` only (containers lack `CAP_NET_RAW` for SYN scans
256+
anyway); ICMP discovery skipped (`-Pn`-equivalent).
257+
- **Rate-limited + infrequent.** Daily cadence, not per-minute; per-org caps.
258+
259+
### 12.2 Behavior
260+
- Establish a **baseline** open-port set per host on first scan (user confirms
261+
"these are expected").
262+
- On each subsequent scan, diff against baseline. **New open port → alert**
263+
(down-alert-style, same channels). Closed expected port → optional info alert.
264+
- Record findings as incidents/events so they show in history and (optionally) a
265+
private security view — **not** the public status page by default.
266+
267+
### 12.3 Prober architecture — BullMQ over Redis, prober stays in the monorepo
268+
269+
Job transport is a **BullMQ queue backed by Redis**, not synchronous RPC (scans
270+
take seconds-to-minutes and must survive restarts). BullMQ gives us retries +
271+
backoff, concurrency control, and **repeatable jobs** for the daily cadence — so
272+
no hand-rolled leasing/reaper is needed.
273+
274+
```
275+
Railway (producer) Redis (broker) DO droplet (prober)
276+
────────────────── ────────────── ───────────────────
277+
API / worker enqueues ──▶ "prober" queue ◀── BullMQ Worker dials OUT
278+
port-scan jobs (rediss:// TLS) over rediss://, runs
279+
(repeatable = daily) nmap -sT -Pn --top-ports
280+
100, returns result
281+
Railway QueueEvents ◀─── job "completed" ◀── (result = job return value)
282+
handler persists to
283+
Supabase, diffs
284+
baseline, fires alerts
285+
```
286+
287+
- **Redis is new infra.** CrawlProof has no Redis today (the worker uses
288+
in-container loopback HTTP). Add one (Railway Redis plugin or Upstash) exposed
289+
over `rediss://` with auth. The droplet connects **outbound only** to Redis —
290+
no inbound port on the prober box (firewall to egress 443/6380 only).
291+
- **Prober lives in the monorepo** as a new workspace (mirrors the existing
292+
`worker/`), e.g. `prober/` — its own `package.json` and entrypoint, sharing the
293+
**job-payload types from `lib/`** so producer and prober can't drift.
294+
- **Deploy is self-bootstrapping.** The `deploy-prober` GitHub Action fires on
295+
merges to `master` (droplet `ubuntu@scan.crawlproof.com`): rsync `prober/`+`lib/`
296+
to the box, then run `prober/deploy/provision.sh` — an **idempotent** script
297+
that installs nmap/Node/build tools, writes the Redis env file, installs the
298+
`crawlproof-prober` systemd unit, builds, and restarts. First run provisions a
299+
bare droplet; later runs update. **No manual SSH/setup on the droplet, ever.**
300+
- **Results flow back via the job return value** — the droplet writes no DB. A
301+
Railway-side `QueueEvents` (or a second in-process Worker) handles `completed`
302+
/`failed`, then persists results, diffs the baseline, and dispatches alerts.
303+
This keeps Supabase credentials off the droplet.
304+
- **Credential blast radius:** the droplet holds only the `rediss://` URL. Scope
305+
it with a **dedicated Redis instance (or ACL user)** limited to the `prober`
306+
queue keys, so a compromised droplet can't read other queues.
307+
- **Repeatable jobs** drive the daily per-monitor scan schedule; per-org caps and
308+
`attempts`/backoff are BullMQ config, not custom code.
309+
310+
### 12.4 Open items
311+
- Which port list ships as the default "watch" set.
312+
- Whether findings feed CrawlProof's existing audit/finding surface or a new one.
313+
- Redis host choice (Railway plugin vs. Upstash) and TLS/ACL setup.
314+
- One droplet = single vantage point + SPOF (fine for daily scans); the BullMQ
315+
Worker model scales to N droplets with zero config change if redundancy is wanted.
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# systemd unit for the CrawlProof port-drift prober (docs/uptime-monitoring-prd.md §12).
2+
# Installed and (re)started automatically by prober/deploy/provision.sh — you do
3+
# NOT install this by hand. Layout matches the deploy-prober GitHub Action:
4+
# repo checkout : /home/ubuntu/crawlproof.com
5+
# Redis env : /home/ubuntu/crawlproof-prober.env (0600, holds REDIS_URL)
6+
7+
[Unit]
8+
Description=CrawlProof port-drift prober (BullMQ worker)
9+
After=network-online.target
10+
Wants=network-online.target
11+
12+
[Service]
13+
Type=simple
14+
User=ubuntu
15+
WorkingDirectory=/home/ubuntu/crawlproof.com/prober
16+
EnvironmentFile=/home/ubuntu/crawlproof-prober.env
17+
ExecStart=/usr/bin/node dist/index.js
18+
Restart=always
19+
RestartSec=5
20+
# Hardening — the prober needs only outbound network + nmap.
21+
NoNewPrivileges=true
22+
ProtectSystem=strict
23+
PrivateTmp=true
24+
ReadWritePaths=/home/ubuntu/crawlproof.com/prober
25+
26+
[Install]
27+
WantedBy=multi-user.target

‎prober/deploy/provision.sh‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
#!/usr/bin/env bash
2+
# Idempotent provisioning for the CrawlProof port-drift prober droplet
3+
# (docs/uptime-monitoring-prd.md §12). Run remotely by the deploy-prober GitHub
4+
# Action on every deploy. Safe to re-run: performs first-time bootstrap of a
5+
# bare droplet AND routine updates. No interactive login is ever required.
6+
#
7+
# Inputs (env):
8+
# REDIS_URL - rediss://... broker URL (required)
9+
# DEPLOY_PATH - repo checkout on the droplet (default /home/ubuntu/crawlproof.com)
10+
set -euo pipefail
11+
12+
DEPLOY_PATH="${DEPLOY_PATH:-/home/ubuntu/crawlproof.com}"
13+
ENV_FILE="/home/ubuntu/crawlproof-prober.env"
14+
UNIT_SRC="${DEPLOY_PATH}/prober/deploy/crawlproof-prober.service"
15+
UNIT_DST="/etc/systemd/system/crawlproof-prober.service"
16+
export DEBIAN_FRONTEND=noninteractive
17+
18+
if [ -z "${REDIS_URL:-}" ]; then
19+
echo "[provision] ERROR: REDIS_URL is not set" >&2
20+
exit 1
21+
fi
22+
23+
echo "[provision] system packages (nmap, build tools)"
24+
sudo apt-get update -y
25+
sudo apt-get install -y --no-install-recommends nmap curl ca-certificates build-essential
26+
27+
echo "[provision] Node 20"
28+
if ! command -v node >/dev/null 2>&1 \
29+
|| [ "$(node -p 'process.versions.node.split(".")[0]' 2>/dev/null || echo 0)" != "20" ]; then
30+
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
31+
sudo apt-get install -y nodejs
32+
fi
33+
34+
echo "[provision] Redis env file (0600)"
35+
umask 077
36+
printf 'REDIS_URL=%s\n' "${REDIS_URL}" > "${ENV_FILE}"
37+
chmod 600 "${ENV_FILE}"
38+
39+
echo "[provision] build prober"
40+
cd "${DEPLOY_PATH}/prober"
41+
npm ci --no-audit --no-fund
42+
npm run build
43+
44+
echo "[provision] install + (re)start systemd service"
45+
sudo cp "${UNIT_SRC}" "${UNIT_DST}"
46+
sudo systemctl daemon-reload
47+
sudo systemctl enable crawlproof-prober
48+
sudo systemctl restart crawlproof-prober
49+
sudo systemctl --no-pager --lines=5 status crawlproof-prober || true
50+
echo "[provision] done"

0 commit comments

Comments
 (0)