From 219564d66f41ea1c3358b5a18288c71f3b6af91e Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Wed, 23 Sep 2026 20:48:18 +0000 Subject: [PATCH 01/12] ops: kit to move Postgres from Railway to self-hosted Supabase Logical replication from Railway (PG 18) into self-hosted Supabase (supabase/postgres 17.6.1.136, pinned self-hosted/v0.8.2) so the copy runs while production writes and the cutover costs minutes of failed writes, not hours of downtime. - server/setup-server.sh: official Supabase stack plus a nichedb overlay: Postgres on 5432 with TLS, pg_hba letting only `postgres` in from outside and only over TLS (compose subnet pinned so Docker's gateway counts as outside), anon/authenticated locked out of schema public, RAM-sized tuning, pooler on loopback, secrets kept off the terminal. - nichedb-db: provision, dns, vault, check, source-prep, schema, subscribe, status, indexes, verify, cutover, rollback, cleanup, apply-migration. - test/rehearse.sh: full local rehearsal on the production schema with a live writer; passes (checksums, generated column, sequences, read-only source, TLS/role/anon rules, the app's own Bun SQL pool). Co-Authored-By: Claude Opus 5.5 --- ops/selfhost-supabase/README.md | 148 ++++ ops/selfhost-supabase/lib/split_post_data.py | 57 ++ ops/selfhost-supabase/nichedb-db | 648 ++++++++++++++++++ ops/selfhost-supabase/server/setup-server.sh | 371 ++++++++++ .../test/app-driver-check.js | 9 + ops/selfhost-supabase/test/make-fixture.sh | 30 + ops/selfhost-supabase/test/rehearse.sh | 189 +++++ 7 files changed, 1452 insertions(+) create mode 100644 ops/selfhost-supabase/README.md create mode 100755 ops/selfhost-supabase/lib/split_post_data.py create mode 100755 ops/selfhost-supabase/nichedb-db create mode 100755 ops/selfhost-supabase/server/setup-server.sh create mode 100644 ops/selfhost-supabase/test/app-driver-check.js create mode 100755 ops/selfhost-supabase/test/make-fixture.sh create mode 100755 ops/selfhost-supabase/test/rehearse.sh diff --git a/ops/selfhost-supabase/README.md b/ops/selfhost-supabase/README.md new file mode 100644 index 0000000..725acc9 --- /dev/null +++ b/ops/selfhost-supabase/README.md @@ -0,0 +1,148 @@ +# nichedb Postgres: Railway to self-hosted Supabase + +This kit moves nichedb's production database off Railway's managed Postgres and +onto self-hosted Supabase (Docker) on our own server. It uses logical +replication, so the copy runs while production keeps writing, and the switch +costs a few minutes of failed writes instead of hours of downtime. + +## Why + +On 2026-09-23 the Railway database was 72 GB (`items` alone: 40 GB heap, +26 GB indexes, 5 GB TOAST, 35M rows). It grew about 6 GB a day from the dump +imports: books, law, podcasts. The volume is 100 GB. CPU is mostly idle, and +memory sits at the 24 GB cap with `shared_buffers` still at 128 MB. The problem +is disk, and it is cheaper to own. + +## What is where + +| Path | Role | +|---|---| +| `server/setup-server.sh` | Runs as root on the new box. Installs the official Supabase stack pinned to `self-hosted/v0.8.2` (Postgres 17.6.1.136), adds the nichedb overlay and starts it. Idempotent. | +| `nichedb-db` | Orchestrator, run from the dev box. Provisioning, DNS, vault, replication, cutover, rollback, cleanup. | +| `lib/split_post_data.py` | Splits the schema so primary keys go in before the copy, and the indexes, unique keys and foreign keys go in after. | +| `test/rehearse.sh` | Full local dress rehearsal on the real Supabase image. | +| `test/make-fixture.sh` | Builds the rehearsal fixture from production: schema, reference tables and newest items. No credentials, sessions, email or money tables. | + +What the overlay changes on stock Supabase: + +- **Postgres is published directly on 5432**, with TLS on and a self-signed + cert for `db.nichedb.dev`. The app already connects with + `sslmode=require` and does not verify the cert. +- **pg_hba**: from outside, only the `postgres` role gets in, and only over + TLS. The compose network is pinned to `172.31.250.0/24` so the Docker + gateway can be treated as outside. That covers clients Docker proxies: + localhost and IPv6. +- **anon/authenticated lose schema `public`**, including what PUBLIC grants + them by default. PostgREST can never serve nichedb's tables. Signups are + off. +- **Tuning** is sized from the box's RAM and CPUs: 25% `shared_buffers`, 16 GB + `max_wal_size`, parallel maintenance, and autovacuum tuned for `items`. +- **Supavisor** binds 127.0.0.1:6543 only. **Studio** sits behind Caddy with + HTTPS and basic auth at `https://supabase.nichedb.dev`. +- **Docker logs** rotate at 3 x 50 MB. **Supabase's `setup.sh` output**, + which prints every secret, goes to a root-only log, never the terminal. + +## Before the server arrives + +- **Keep Railway's volume ahead of the growth.** At about 6 GB a day, the 100 GB + volume fills around 2026-09-26. Railway's Postgres also has to hold the + replication slot's WAL during the copy, about 1.4 GB an hour, capped at + `SLOT_KEEP_GB=8`. +- **Server sizing.** Plan for 3x the database on the data disk, NVMe, + 32 GB+ RAM, Ubuntu 24.04+ or Debian 12+. Put it on the **US East coast**: + the app and its database both run in Railway's `us-east4` (northern + Virginia), and the app stays there and queries across. Every extra + millisecond of distance is paid on every query. +- **Freeze niche-db merges for the replication window.** Logical replication + does not copy DDL. A deploy that runs a migration mid-copy parks the + subscription. See "A migration landed mid-copy" below. + +## Runbook + +```sh +K=ops/selfhost-supabase/nichedb-db + +$K provision root@ # install + start Supabase, fetch the connection +$K dns # db.nichedb.dev + supabase.nichedb.dev -> server +$K vault # server .env -> vault nichedb-supabase--prod, + # SELFHOST_* merged into nichedb--prod (backup kept) +$K check # versions, sizes, free disk on both sides + +$K source-prep # Railway: wal_level=logical (RESTARTS Postgres, + # ~30-60 s app outage), TCP proxy, publication +$K schema # tables + primary keys on the target +$K subscribe # initial copy, then streaming +$K status # repeat until every table is state r +$K indexes # detached on the server; `status` tails the log +$K verify # schema must be identical + +$K cutover # the switch (below) +# ...a few days later, once nothing points back: +$K cleanup # drop publication/slot, remove the TCP proxy +``` + +`cutover` does this, in order, and refuses to start unless every table is +ready, the index counts match, the schemas are identical and lag is under +64 MB: + +1. Sets Railway's database `default_transaction_read_only = on` and closes + the app's sessions. Reads keep working. Writes fail from here. +2. Waits until the target has confirmed every WAL byte written so far. +3. Copies all 32 sequence positions. +4. Drops the subscription, which also drops the slot on Railway. +5. Sets the app's `DATABASE_URL` to the new server and waits for the deploy. +6. Smoke tests `https://nichedb.dev/healthz` and counts the app's connections + on the target. + +Railway's database is left read-only and intact as the fallback. + +**Rollback** with `$K rollback`. It restores `DATABASE_URL` to +`${{Postgres.DATABASE_URL}}` and makes Railway writable again. Anything +written to the new server after the cutover is not copied back. + +State and secrets live in `~/.local/state/nichedb-selfhost/` (mode 0700). + +### A migration landed mid-copy + +The subscription disables itself on the first apply error, and `status` shows +it disabled with errors. Apply the migration's DDL to the target, then +re-enable: + +```sh +$K apply-migration packages/db/migrations/00NN_name.sql +``` + +Its `schema_migrations` row arrives through replication, so the app will not +run it again. If the migration also changes data, apply only its DDL part. +The data changes come across from Railway anyway. + +## Rehearsal + +```sh +ops/selfhost-supabase/test/make-fixture.sh /tmp/nichedb-fixture # needs railway login +ops/selfhost-supabase/test/rehearse.sh /tmp/nichedb-fixture +``` + +Passed on 2026-09-23 with the production schema: 53 tables, 32 sequences, +84 indexes, 68 foreign keys. A writer inserted, updated and deleted items on +the source through the whole run. Checks: + +- **Data**: counts and an md5 over every item row are identical, and so is + the generated `search` column. +- **Sequences**: a fresh insert after the cutover takes a new id. +- **Cutover**: the source rejects writes afterwards, and the subscription and + slot are gone. +- **Access**: TLS is required, `supabase_admin` is refused from outside, and + anon gets "permission denied for schema public" over REST. +- **The app's own Bun SQL pool** reads the target with `sslmode=require`. +- **Migrations**: `migrate-cli` reports "up to date" against the target. +- **Operations**: `rollback` restores writes, and a second `setup-server.sh` + run keeps its secrets. + +## Not covered yet + +- **Backups** of the new server. Railway did this for us. Pick a destination + (S3/B2 bucket) and add WAL-G or pgBackRest before deleting Railway's copy. +- **Redis** stays on Railway. Only Postgres moves. +- **The TLS cert is self-signed.** Encrypted, but the app does not verify it, + same as today. diff --git a/ops/selfhost-supabase/lib/split_post_data.py b/ops/selfhost-supabase/lib/split_post_data.py new file mode 100755 index 0000000..7f79ca3 --- /dev/null +++ b/ops/selfhost-supabase/lib/split_post_data.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python3 +"""Split a `pg_dump --section=post-data` script into two files. + + pk.sql PRIMARY KEY constraints. Applied before the initial copy: logical + replication finds the row for every UPDATE/DELETE by its replica + identity, and without a key that is a sequential scan of items. + rest.sql Everything else (indexes, unique/foreign keys, triggers). Applied + after the copy, which is several times faster than maintaining + 26 GB of indexes row by row while 72 GB streams in. + +The session preamble (SET ..., set_config, \\restrict) is copied into both. + +usage: split_post_data.py post.sql pk.sql rest.sql +""" +import re +import sys + + +def statements(text): + """Yield top-level SQL statements. pg_dump post-data has no function bodies, + so a semicolon at the end of a line closes a statement.""" + buf = [] + for line in text.splitlines(keepends=True): + if not buf and (line.startswith('--') or not line.strip()): + continue + buf.append(line) + if line.rstrip().endswith(';') or line.startswith('\\'): + yield ''.join(buf) + buf = [] + if ''.join(buf).strip(): + yield ''.join(buf) + + +def main(src, pk_out, rest_out): + text = open(src, encoding='utf-8').read() + preamble, pk, rest = [], [], [] + for stmt in statements(text): + head = stmt.lstrip() + if head.startswith('\\') or re.match(r'SET |SELECT pg_catalog\.set_config', head): + preamble.append(stmt) + elif re.search(r'\bPRIMARY KEY\b', stmt): + pk.append(stmt) + else: + rest.append(stmt) + # \unrestrict closes \restrict: keep it at the end of both files. + tail = [s for s in preamble if s.lstrip().startswith('\\unrestrict')] + head = [s for s in preamble if not s.lstrip().startswith('\\unrestrict')] + for path, body in ((pk_out, pk), (rest_out, rest)): + with open(path, 'w', encoding='utf-8') as f: + f.write(''.join(head) + '\n' + '\n'.join(body) + '\n' + ''.join(tail)) + print(f'{len(pk)} primary keys, {len(rest)} other post-data statements') + + +if __name__ == '__main__': + if len(sys.argv) != 4: + sys.exit(__doc__) + main(*sys.argv[1:]) diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db new file mode 100755 index 0000000..693ce1f --- /dev/null +++ b/ops/selfhost-supabase/nichedb-db @@ -0,0 +1,648 @@ +#!/usr/bin/env bash +# +# nichedb-db: move nichedb's Postgres off Railway onto self-hosted Supabase +# with logical replication, so the cutover costs minutes of failed writes +# instead of hours of downtime. Run from the dev box. See README.md for the +# runbook; `nichedb-db help` for the commands. +# +set -euo pipefail + +HERE=$(cd "$(dirname "$(readlink -f "$0")")" && pwd) +STATE_DIR=${STATE_DIR:-$HOME/.local/state/nichedb-selfhost} +STATE=$STATE_DIR/state.env +mkdir -p "$STATE_DIR" +chmod 700 "$STATE_DIR" +# shellcheck disable=SC1090 +[ -f "$STATE" ] && . "$STATE" + +# ------------------------------------------------------------------ config +# Railway: project nichedb.dev, environment production. +RW_PROJECT=${RW_PROJECT:-fb518b20-ed8b-4066-a485-85c1b627063f} +RW_ENV=${RW_ENV:-dc6d3af4-c0b6-48bf-b93e-4ba0808209f1} +RW_PG_SERVICE=${RW_PG_SERVICE:-0e3bac56-c390-4afd-b9a2-6a98f5b66321} +RW_APP_SERVICE=${RW_APP_SERVICE:-fd8ab4c1-3787-461a-a9a6-3564603deee3} +RW_PG_SERVICE_NAME=${RW_PG_SERVICE_NAME:-Postgres} +SOURCE_DB=${SOURCE_DB:-railway} +# SOURCE_MODE=railway (production) or docker (rehearsal: SOURCE_CONTAINER). +SOURCE_MODE=${SOURCE_MODE:-railway} +SOURCE_CONTAINER=${SOURCE_CONTAINER:-} +# TARGET_SSH=user@host of the new server, or `local` for a rehearsal. +TARGET_SSH=${TARGET_SSH:-} +TARGET_DIR=${TARGET_DIR:-/opt/nichedb-supabase} +TARGET_DB=${TARGET_DB:-postgres} +PUB=${PUB:-nichedb_migrate} +SUB=${SUB:-nichedb_migrate} +APP_HEALTH_URL=${APP_HEALTH_URL:-https://nichedb.dev/healthz} +DB_DOMAIN=${DB_DOMAIN:-db.nichedb.dev} +STUDIO_DOMAIN=${STUDIO_DOMAIN:-supabase.nichedb.dev} +DNS_ZONE=${DNS_ZONE:-nichedb.dev} +VAULT_TEAM=${VAULT_TEAM:-profullstack} +SSH_KEY=${SSH_KEY:-$HOME/.ssh/id_ed25519} +# Cap on WAL the source keeps for a stalled subscriber. Past it the slot is +# invalidated (re-run `subscribe`) instead of the Railway volume filling up. +SLOT_KEEP_GB=${SLOT_KEEP_GB:-8} + +log() { printf '\n===> %s\n' "$*" >&2; } +note() { printf ' %s\n' "$*" >&2; } +warn() { printf 'WARNING: %s\n' "$*" >&2; } +die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } + +save_state() { # save_state KEY VALUE + touch "$STATE" + chmod 600 "$STATE" + grep -v "^$1=" "$STATE" > "$STATE.tmp" || true + printf '%s=%q\n' "$1" "$2" >> "$STATE.tmp" + mv "$STATE.tmp" "$STATE" + eval "$1=\$2" +} + +confirm() { # confirm "question" (skipped by --yes) + [ "${YES:-0}" = 1 ] && return 0 + [ -t 0 ] || die "$1 -- rerun with --yes to proceed non-interactively" + printf '%s [y/N] ' "$1" >&2 + local a + read -r a + [ "$a" = y ] || [ "$a" = Y ] || die "aborted" +} + +# --------------------------------------------------------------- railway API +rw_token() { jq -r '.user.accessToken' "$HOME/.railway/config.json"; } + +gql() { # gql QUERY [VARIABLES_JSON] + local body + body=$(jq -nc --arg q "$1" --argjson v "${2:-{\}}" '{query:$q, variables:$v}') + local out + out=$(curl -fsS https://backboard.railway.com/graphql/v2 \ + -H "Authorization: Bearer $(rw_token)" -H 'Content-Type: application/json' \ + -H 'User-Agent: nichedb-db' --data "$body") + if [ "$(jq '.errors // [] | length' <<<"$out")" != 0 ]; then + printf '%s\n' "$out" >&2 + return 1 + fi + printf '%s' "$out" +} + +rw_vars() { # rw_vars SERVICE_ID [unrendered] + gql 'query($p:String!,$e:String!,$s:String!,$u:Boolean){variables(projectId:$p,environmentId:$e,serviceId:$s,unrendered:$u)}' \ + "$(jq -nc --arg p "$RW_PROJECT" --arg e "$RW_ENV" --arg s "$1" --argjson u "${2:-false}" '{p:$p,e:$e,s:$s,u:$u}')" | + jq '.data.variables' +} + +rw_set_var() { # rw_set_var SERVICE_ID NAME VALUE (triggers a deploy) + gql 'mutation($i:VariableUpsertInput!){variableUpsert(input:$i)}' \ + "$(jq -nc --arg p "$RW_PROJECT" --arg e "$RW_ENV" --arg s "$1" --arg n "$2" --arg v "$3" \ + '{i:{projectId:$p,environmentId:$e,serviceId:$s,name:$n,value:$v}}')" >/dev/null +} + +rw_latest_deployment() { # rw_latest_deployment SERVICE_ID -> "id status createdAt" + gql 'query($p:String!,$e:String!,$s:String!){deployments(first:1,input:{projectId:$p,environmentId:$e,serviceId:$s}){edges{node{id status createdAt}}}}' \ + "$(jq -nc --arg p "$RW_PROJECT" --arg e "$RW_ENV" --arg s "$1" '{p:$p,e:$e,s:$s}')" | + jq -r '.data.deployments.edges[0].node | "\(.id) \(.status) \(.createdAt)"' +} + +rw_tcp_proxy() { # -> "domain port id" or empty + gql 'query($s:String!,$e:String!){tcpProxies(serviceId:$s,environmentId:$e){id domain proxyPort applicationPort}}' \ + "$(jq -nc --arg s "$RW_PG_SERVICE" --arg e "$RW_ENV" '{s:$s,e:$e}')" | + jq -r '.data.tcpProxies[] | select(.applicationPort==5432) | "\(.domain) \(.proxyPort) \(.id)"' | head -n1 +} + +# ------------------------------------------------------------ SQL transports +railway_ssh_user() { + if [ -z "${RW_PG_SSH_USER:-}" ]; then + local u + u=$(railway ssh config --dry-run -i "$SSH_KEY" --project "$RW_PROJECT" --environment "$RW_ENV" \ + --service "$RW_PG_SERVICE_NAME" 2>/dev/null | awk '$1=="User" {print $2; exit}') + [ -n "$u" ] || die "could not read the Railway ssh user for $RW_PG_SERVICE_NAME (railway login?)" + save_state RW_PG_SSH_USER "$u" + fi + printf '%s' "$RW_PG_SSH_USER" +} + +src_exec() { # run a command inside the source Postgres container, stdin passed through + case "$SOURCE_MODE" in + docker) docker exec -i "$SOURCE_CONTAINER" "$@" ;; + railway) + # `railway ssh` itself fails host-key verification; plain ssh works. + ssh -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile="$HOME/.ssh/known_hosts_railway" \ + -o ServerAliveInterval=30 -o LogLevel=ERROR -i "$SSH_KEY" -l "$(railway_ssh_user)" ssh.railway.com "$@" + ;; + *) die "SOURCE_MODE must be railway or docker" ;; + esac +} + +# Source SQL always runs with read-only off: after `cutover` the database +# default is read-only, and rollback/cleanup still have to write. +src_sql() { # SQL on stdin + { echo "set default_transaction_read_only = off;"; cat; } | + src_exec psql -U postgres -d "$SOURCE_DB" -v ON_ERROR_STOP=1 -X -q -At +} +src_q() { src_sql <<<"$1"; } + +tgt_exec() { # run a shell command string on the target host + if [ "$TARGET_SSH" = local ]; then + bash -c "$1" + else + [ -n "$TARGET_SSH" ] || die "no target: run 'nichedb-db provision user@host' first (or set TARGET_SSH)" + ssh -o BatchMode=yes -o ServerAliveInterval=30 "$TARGET_SSH" "$1" + fi +} +tgt_sudo() { if [ "$TARGET_SSH" = local ]; then printf ''; else printf 'sudo -n '; fi; } + +tgt_sql() { # tgt_sql ROLE (SQL on stdin) - ROLE is supabase_admin or postgres + tgt_exec "$(tgt_sudo)docker exec -i supabase-db psql -h localhost -U $1 -d $TARGET_DB -v ON_ERROR_STOP=1 -X -q -At" +} +tgt_q() { tgt_sql supabase_admin <<<"$1"; } + +tgt_copy() { # tgt_copy LOCAL_FILE REMOTE_PATH + if [ "$TARGET_SSH" = local ]; then cp "$1" "$2"; else scp -q "$1" "$TARGET_SSH:$2"; fi +} + +# --------------------------------------------------------------- helpers +source_conninfo() { # libpq conninfo the subscriber uses to reach the source + if [ "$SOURCE_MODE" = docker ]; then + printf 'host=%s port=5432 dbname=%s user=postgres password=%s' \ + "${SOURCE_HOST:-$SOURCE_CONTAINER}" "$SOURCE_DB" "${SOURCE_PASSWORD:?set SOURCE_PASSWORD}" + return + fi + local proxy pw + proxy=$(rw_tcp_proxy) + [ -n "$proxy" ] || die "the Railway Postgres has no TCP proxy yet: run 'nichedb-db source-prep'" + pw=$(rw_vars "$RW_PG_SERVICE" | jq -r '.PGPASSWORD // .POSTGRES_PASSWORD') + set -- $proxy + printf 'host=%s port=%s dbname=%s user=postgres password=%s sslmode=require' "$1" "$2" "$SOURCE_DB" "$pw" +} + +target_url() { + [ -n "${SELFHOST_DATABASE_URL:-}" ] || die "no SELFHOST_DATABASE_URL in state: run 'nichedb-db provision' (or fetch-connection)" + printf '%s' "$SELFHOST_DATABASE_URL" +} + +slot_lag_bytes() { + src_q "select coalesce(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn), -1)::bigint from pg_replication_slots where slot_name = '$SUB'" +} + +sync_states() { # "r 53" etc. + tgt_q "select srsubstate, count(*) from pg_subscription_rel r join pg_subscription s on s.oid = r.srsubid where s.subname = '$SUB' group by 1 order by 1" | tr '|' ' ' +} + +all_ready() { + local notready + notready=$(tgt_q "select count(*) from pg_subscription_rel r join pg_subscription s on s.oid = r.srsubid where s.subname = '$SUB' and srsubstate <> 'r'") + [ "$notready" = 0 ] +} + +schema_fingerprint() { # normalised public schema of one side: src|tgt + local dump + if [ "$1" = src ]; then + dump=$(src_exec pg_dump -U postgres -d "$SOURCE_DB" --schema-only --no-owner --no-privileges -n public) + else + dump=$(tgt_exec "$(tgt_sudo)docker exec supabase-db pg_dump -h localhost -U supabase_admin -d $TARGET_DB --schema-only --no-owner --no-privileges -n public") + fi + # Drop comments, session SETs, \restrict keys and version banners. + grep -Ev '^(--|SET |SELECT pg_catalog.set_config|\\(un)?restrict|CREATE SCHEMA public;|COMMENT ON SCHEMA public)' <<<"$dump" | + sed -e 's/[[:space:]]*$//' | grep -v '^$' +} + +# ---------------------------------------------------------------- commands +cmd_provision() { # provision user@host + local host=${1:-$TARGET_SSH} + [ -n "$host" ] || die "usage: nichedb-db provision user@host" + save_state TARGET_SSH "$host" + log "Provisioning $host" + tgt_exec 'mkdir -p /tmp/nichedb-selfhost' + tgt_copy "$HERE/server/setup-server.sh" /tmp/nichedb-selfhost/setup-server.sh + local envs="DB_DOMAIN=$DB_DOMAIN STUDIO_DOMAIN=$STUDIO_DOMAIN" + local k + for k in SUPABASE_REF INSTALL_ROOT PROJECT SITE_URL DB_PORT PGDATA_DIR MEM_MB CPUS WITH_PROXY; do + [ -n "${!k:-}" ] && envs="$envs $k=${!k}" + done + tgt_exec "sudo -n env $envs bash /tmp/nichedb-selfhost/setup-server.sh" + cmd_fetch_connection +} + +cmd_fetch_connection() { + log "Fetching the connection details from the server" + local f=$STATE_DIR/connection.env + umask 077 + tgt_exec "$(tgt_sudo)cat $TARGET_DIR/nichedb-connection.env" > "$f" + local line + while IFS= read -r line; do + case "$line" in SELFHOST_*=*) save_state "${line%%=*}" "${line#*=}" ;; esac + done < "$f" + local ip + ip=$(tgt_exec "curl -fsS4 https://api.ipify.org" 2>/dev/null || true) + [ -n "$ip" ] && save_state TARGET_IP "$ip" + note "target ${TARGET_SSH} (${TARGET_IP:-ip unknown}); secrets saved in $STATE (0600)" +} + +cmd_vault() { + log "Saving the Supabase secrets to the team vault" + local work + work=$(mktemp -d) + trap 'rm -rf "$work"' RETURN + # 1) the server's whole Supabase .env as its own vault + tgt_exec "$(tgt_sudo)cat $TARGET_DIR/.env" > "$work/supabase.env" + logicsrc teams push "$VAULT_TEAM" nichedb-supabase prod --env "$work/supabase.env" + # 2) SELFHOST_* next to the app's own secrets, merged, never replacing them + if logicsrc teams pull "$VAULT_TEAM" nichedb prod --env "$work/app.env" >/dev/null 2>&1 && [ -s "$work/app.env" ]; then + local n=1 + while [ -e "$STATE_DIR/nichedb--prod.bak-$(printf %03d $n).env" ]; do n=$((n + 1)); done + cp "$work/app.env" "$STATE_DIR/nichedb--prod.bak-$(printf %03d $n).env" + chmod 600 "$STATE_DIR"/nichedb--prod.bak-*.env + grep -v '^SELFHOST_' "$work/app.env" > "$work/merged.env" || true + grep '^SELFHOST_' "$STATE_DIR/connection.env" >> "$work/merged.env" + logicsrc teams push "$VAULT_TEAM" nichedb prod --env "$work/merged.env" + note "nichedb--prod backup: $STATE_DIR/nichedb--prod.bak-$(printf %03d $n).env" + else + die "could not pull vault nichedb--prod; nothing pushed to it (the nichedb-supabase vault was saved)" + fi +} + +cmd_dns() { + local ip=${1:-${TARGET_IP:-}} + [ -n "$ip" ] || die "usage: nichedb-db dns " + log "DNS: $DB_DOMAIN and $STUDIO_DOMAIN -> $ip" + porkbun set "$DNS_ZONE" "${DB_DOMAIN%%.$DNS_ZONE}" A "$ip" --ttl 600 + porkbun set "$DNS_ZONE" "${STUDIO_DOMAIN%%.$DNS_ZONE}" A "$ip" --ttl 600 + [ -n "${TARGET_IPV6:-}" ] && porkbun set "$DNS_ZONE" "${DB_DOMAIN%%.$DNS_ZONE}" AAAA "$TARGET_IPV6" --ttl 600 + [ -n "${TARGET_IPV6:-}" ] && porkbun set "$DNS_ZONE" "${STUDIO_DOMAIN%%.$DNS_ZONE}" AAAA "$TARGET_IPV6" --ttl 600 + return 0 +} + +cmd_check() { + log "Source" + src_q "select 'version '||current_setting('server_version')||', wal_level '||current_setting('wal_level')||', size '||pg_size_pretty(pg_database_size(current_database()))||', max_slot_wal_keep_size '||current_setting('max_slot_wal_keep_size')" | sed 's/^/ /' + src_exec df -h /var/lib/postgresql/data 2>/dev/null | tail -1 | awk '{print " disk: "$3" used, "$4" free of "$2}' || true + log "Target" + tgt_q "select 'version '||current_setting('server_version')||', ssl '||current_setting('ssl')||', shared_buffers '||current_setting('shared_buffers')||', size '||pg_size_pretty(pg_database_size(current_database()))" | sed 's/^/ /' + tgt_exec "df -h $TARGET_DIR | tail -1" | awk '{print " disk: "$3" used, "$4" free of "$2}' + local srcsize tgtfree + srcsize=$(src_q "select pg_database_size(current_database())") + tgtfree=$(tgt_exec "df -B1 --output=avail $TARGET_DIR | tail -1") + if [ "$tgtfree" -lt $((srcsize * 2)) ]; then + warn "target has less than 2x the source size free; the copy plus index builds may not fit" + fi +} + +cmd_source_prep() { + [ "$SOURCE_MODE" = railway ] || { note "SOURCE_MODE=$SOURCE_MODE: only the publication is created"; cmd_publication; return; } + log "Preparing the Railway source" + local wal + wal=$(src_q "show wal_level") + if [ "$wal" != logical ]; then + confirm "Set wal_level=logical and RESTART Railway's Postgres (the app loses its DB for ~30-60 s)?" + src_q "alter system set wal_level = logical; alter system set max_slot_wal_keep_size = '${SLOT_KEEP_GB}GB';" + local dep + dep=$(rw_latest_deployment "$RW_PG_SERVICE" | awk '{print $1}') + gql 'mutation($id:String!){deploymentRestart(id:$id)}' "$(jq -nc --arg id "$dep" '{id:$id}')" >/dev/null + note "restart requested for deployment $dep; waiting for Postgres" + + for _ in $(seq 1 60); do + sleep 5 + wal=$(src_q "show wal_level" 2>/dev/null || true) + [ "$wal" = logical ] && break + done + [ "$wal" = logical ] || die "wal_level is still '$wal' after the restart" + note "wal_level=logical" + else + note "wal_level already logical" + src_q "alter system set max_slot_wal_keep_size = '${SLOT_KEEP_GB}GB'; select pg_reload_conf();" >/dev/null + fi + + local proxy + proxy=$(rw_tcp_proxy) + if [ -z "$proxy" ]; then + confirm "Expose Railway's Postgres on a public TCP proxy (password + TLS) for the replication?" + if ! gql 'mutation($i:TCPProxyCreateInput!){tcpProxyCreate(input:$i){id domain proxyPort}}' \ + "$(jq -nc --arg s "$RW_PG_SERVICE" --arg e "$RW_ENV" '{i:{serviceId:$s,environmentId:$e,applicationPort:5432}}')" >/dev/null; then + die "tcpProxyCreate failed. Add it by hand: Railway > nichedb.dev > Postgres > Settings > Networking > TCP Proxy, port 5432; then rerun." + fi + for _ in $(seq 1 30); do proxy=$(rw_tcp_proxy); [ -n "$proxy" ] && break; sleep 5; done + fi + note "TCP proxy: $(awk '{print $1":"$2}' <<<"$proxy")" + cmd_publication +} + +cmd_publication() { + local missing + missing=$(src_q "select string_agg(c.relname, ', ') from pg_class c join pg_namespace n on n.oid = c.relnamespace where n.nspname = 'public' and c.relkind in ('r','p') and c.relreplident = 'd' and not exists (select 1 from pg_index i where i.indrelid = c.oid and i.indisprimary)") + # With FOR ALL TABLES, an UPDATE on a table with no replica identity FAILS on + # the source. Every table has a primary key today; refuse if that changes. + [ -z "$missing" ] || die "tables without a primary key would break writes on the source: $missing" + if [ "$(src_q "select count(*) from pg_publication where pubname = '$PUB'")" = 0 ]; then + src_q "create publication $PUB for all tables" + note "publication $PUB created (all tables)" + else + note "publication $PUB already exists" + fi +} + +cmd_schema() { + log "Copying the schema (tables + primary keys; indexes come after the data)" + local n + n=$(tgt_q "select count(*) from pg_tables where schemaname = 'public'") + if [ "$n" != 0 ] && [ "${FORCE:-0}" != 1 ]; then + die "target already has $n tables in public; FORCE=1 to apply on top anyway" + fi + local d=$STATE_DIR/schema + mkdir -p "$d" + src_exec pg_dump -U postgres -d "$SOURCE_DB" --schema-only --no-owner --no-privileges --section=pre-data > "$d/pre.sql" + src_exec pg_dump -U postgres -d "$SOURCE_DB" --schema-only --no-owner --no-privileges --section=post-data > "$d/post.sql" + python3 "$HERE/lib/split_post_data.py" "$d/post.sql" "$d/pk.sql" "$d/rest.sql" >&2 + note "pre-data $(grep -c 'CREATE TABLE' "$d/pre.sql") tables; saved in $d" + # Extensions need supabase_admin; everything else is owned by postgres, the + # role the app connects as and runs its migrations with. + grep -E '^CREATE EXTENSION' "$d/pre.sql" | tgt_sql supabase_admin + tgt_sql postgres < "$d/pre.sql" + tgt_sql postgres < "$d/pk.sql" + tgt_sql supabase_admin <<'EOF' +revoke all on all tables in schema public from anon, authenticated; +revoke all on all sequences in schema public from anon, authenticated; +revoke all on all functions in schema public from anon, authenticated; +revoke usage on schema public from public, anon, authenticated; +grant usage on schema public to postgres, service_role; +EOF + note "target now has $(tgt_q "select count(*) from pg_tables where schemaname = 'public'") tables in public" +} + +cmd_subscribe() { + log "Subscribing the target to the source" + if [ "$(tgt_q "select count(*) from pg_subscription where subname = '$SUB'")" != 0 ]; then + note "subscription $SUB already exists" + return + fi + local conn + conn=$(source_conninfo) + # disable_on_error: a schema mismatch (a migration deployed mid-copy) parks + # the subscription instead of retrying forever; `status` shows it. + tgt_sql supabase_admin </dev/null || echo none)" + src_q "select 'source slot '||slot_name||' wal_status='||wal_status||' safe_wal_size='||coalesce(pg_size_pretty(safe_wal_size),'unlimited') from pg_replication_slots where slot_name='$SUB'" | sed 's/^/ /' + note "sizes: source $(src_q "select pg_size_pretty(pg_database_size(current_database()))"), target $(tgt_q "select pg_size_pretty(pg_database_size(current_database()))")" + if [ -n "$(tgt_exec "test -f $TARGET_DIR/indexes.log && echo y" 2>/dev/null || true)" ]; then + note "index build: $(tgt_exec "$(tgt_sudo)tail -n 2 $TARGET_DIR/indexes.log" | tr '\n' ' ')" + fi +} + +cmd_indexes() { + local d=$STATE_DIR/schema + [ -s "$d/rest.sql" ] || die "no $d/rest.sql: run 'nichedb-db schema' first" + if ! all_ready && [ "${FORCE:-0}" != 1 ]; then + die "the initial copy is not finished ($(sync_states | tr '\n' ' ')); FORCE=1 to build now" + fi + log "Building indexes, unique and foreign keys on the target (detached; replication keeps streaming)" + local mwm work + mwm=$(tgt_q "show maintenance_work_mem") + work=$(mktemp -d) + { + printf "set maintenance_work_mem = '%s';\n" "$mwm" + cat "$d/rest.sql" + printf "select 'INDEXES DONE';\n" + } > "$work/indexes.sql" + cat > "$work/indexes.sh" < "$TARGET_DIR/indexes.log" 2>&1 || { tail -n 5 "$TARGET_DIR/indexes.log" >&2; die "index build failed"; } + tail -n 1 "$TARGET_DIR/indexes.log" >&2 + else + # Detached: 26 GB of indexes outlives an ssh session. + tgt_exec "$(tgt_sudo)nohup sh /tmp/nichedb-indexes.sh > /tmp/nichedb-indexes.log 2>&1 < /dev/null & sleep 1; $(tgt_sudo)ln -sf /tmp/nichedb-indexes.log $TARGET_DIR/indexes.log" + note "running on the server; progress in $TARGET_DIR/indexes.log (nichedb-db status shows the tail)" + fi +} + +cmd_verify() { + log "Schema diff (public), source vs target" + local a b + a=$(mktemp) + b=$(mktemp) + schema_fingerprint src > "$a" + schema_fingerprint tgt > "$b" + if diff -u "$a" "$b" > "$STATE_DIR/schema.diff"; then + note "identical" + else + warn "schemas differ (full diff: $STATE_DIR/schema.diff):" + head -n 40 "$STATE_DIR/schema.diff" >&2 + SCHEMA_DIFFERS=1 + fi + rm -f "$a" "$b" + + log "Row counts (source, target)" + # One query per side. The big tables compare max(id) unless FULL=1: an exact + # count of 35M rows is a minute of I/O on each side. + local big=${BIG_TABLES:-items} q + q=$(src_q "select string_agg(format('select %L, %s from public.%I', tablename, case when tablename = any(string_to_array('$big', ' ')) and ${FULL:-0} = 0 then 'coalesce(max(id),0)::text' else 'count(*)::text' end, tablename), ' union all ' order by tablename) from pg_tables where schemaname = 'public'") + local s g + s=$(src_q "$q") + g=$(tgt_q "$q") + local bad + bad=$(join -t'|' -a1 -a2 -e MISSING -o 0,1.2,2.2 <(sort <<<"$s") <(sort <<<"$g") | + awk -F'|' '{st = ($2 == $3) ? "ok" : "DIFF"; if (st != "ok") n++; printf " %-30s %12s %12s %s\n", $1, $2, $3, st} END {print n + 0 > "/dev/fd/3"}' 3>&1 1>&2) + note "$bad table(s) differ (expected while writes are still flowing; items compares max(id))" + [ "${SCHEMA_DIFFERS:-0}" = 0 ] +} + +sync_sequences() { + log "Copying sequence positions" + local sql + sql=$(src_q "select format('select setval(%L, %s, %L);', quote_ident(schemaname)||'.'||quote_ident(sequencename), coalesce(last_value, start_value), last_value is not null) from pg_sequences where schemaname = 'public'") + tgt_sql supabase_admin <<<"$sql" >/dev/null + note "$(wc -l <<<"$sql") sequences set" +} + +cmd_cutover() { + log "Preflight" + all_ready || die "not every table is in state r: $(sync_states | tr '\n' ' ')" + [ "$(tgt_q "select subenabled from pg_subscription where subname = '$SUB'")" = t ] || die "subscription is disabled: see 'nichedb-db status'" + local si ti + si=$(src_q "select count(*) from pg_indexes where schemaname = 'public'") + ti=$(tgt_q "select count(*) from pg_indexes where schemaname = 'public'") + [ "$si" = "$ti" ] || die "indexes: source $si, target $ti. Run/finish 'nichedb-db indexes' first" + cmd_verify >/dev/null 2>&1 || die "schemas differ: run 'nichedb-db verify' and reconcile first" + local lag + lag=$(slot_lag_bytes) + [ "$lag" -lt $((64 * 1024 * 1024)) ] || die "slot lag is $(numfmt --to=iec "$lag"); wait for it to drain" + [ "$SOURCE_MODE" = docker ] || target_url >/dev/null + confirm "Cut over now? Writes fail for a few minutes; reads keep working from the old DB." + + log "1/6 Source read-only, sessions closed" + src_q "alter database $SOURCE_DB set default_transaction_read_only = on" + src_q "select count(pg_terminate_backend(pid)) from pg_stat_activity where datname = '$SOURCE_DB' and backend_type = 'client backend' and pid <> pg_backend_pid()" >/dev/null + save_state CUTOVER_STARTED "$(date -u +%FT%TZ)" + + log "2/6 Waiting for the target to confirm every source write" + local lsn done_=0 + lsn=$(src_q "select pg_current_wal_lsn()") + for _ in $(seq 1 120); do + if [ "$(src_q "select confirmed_flush_lsn >= '$lsn' from pg_replication_slots where slot_name = '$SUB'")" = t ]; then done_=1; break; fi + sleep 2 + done + [ "$done_" = 1 ] || die "target did not catch up to $lsn in 4 minutes. Source is still READ-ONLY: fix, or 'nichedb-db rollback'" + note "caught up to $lsn" + + sync_sequences + log "3/6 Final counts" + cmd_verify || warn "verify reported differences; check before trusting the target" + + log "4/6 Dropping the subscription (also drops the slot on the source)" + tgt_q "drop subscription $SUB" + + if [ "$SOURCE_MODE" = railway ]; then + log "5/6 Pointing the app at the new database" + local old + old=$(rw_vars "$RW_APP_SERVICE" true | jq -r '.DATABASE_URL') + [ "${OLD_DATABASE_URL:-}" ] || save_state OLD_DATABASE_URL "$old" + rw_set_var "$RW_APP_SERVICE" DATABASE_URL "$(target_url)" + note "DATABASE_URL updated (was $old); Railway is deploying" + wait_for_deploy "$RW_APP_SERVICE" + log "6/6 Smoke test" + curl -fsS --max-time 20 "$APP_HEALTH_URL" && echo >&2 || warn "$APP_HEALTH_URL did not answer ok" + note "app connections on the target: $(tgt_q "select count(*) from pg_stat_activity where usename = 'postgres' and client_addr is not null and client_addr <> '127.0.0.1'")" + else + note "5/6 and 6/6 skipped (rehearsal: no app to repoint)" + fi + save_state CUTOVER_DONE "$(date -u +%FT%TZ)" + log "Cut over. The Railway DB stays READ-ONLY as the fallback; 'nichedb-db rollback' undoes the switch." +} + +wait_for_deploy() { + local before status id _ + before=$(date -u +%FT%TZ) + for _ in $(seq 1 90); do + sleep 10 + read -r id status _ <<<"$(rw_latest_deployment "$1")" + note "deployment $id: $status" + case "$status" in + SUCCESS) return 0 ;; + FAILED | CRASHED) die "deployment $id $status: consider 'nichedb-db rollback'" ;; + esac + done + die "deployment still not SUCCESS after 15 minutes (started $before)" +} + +cmd_rollback() { + warn "Rollback points the app back at Railway. Writes made on the new server since the cutover are NOT copied back." + confirm "Roll back to the Railway database?" + if [ "$SOURCE_MODE" = railway ]; then + rw_set_var "$RW_APP_SERVICE" DATABASE_URL "${OLD_DATABASE_URL:-\${{Postgres.DATABASE_URL\}}}" + note "DATABASE_URL restored" + fi + src_q "alter database $SOURCE_DB reset default_transaction_read_only" + src_q "select count(pg_terminate_backend(pid)) from pg_stat_activity where datname = '$SOURCE_DB' and backend_type = 'client backend' and pid <> pg_backend_pid()" >/dev/null + note "source is writable again" + [ "$SOURCE_MODE" = railway ] && wait_for_deploy "$RW_APP_SERVICE" + return 0 +} + +cmd_cleanup() { + confirm "Drop the publication and any leftover slot on the source, and remove the TCP proxy?" + src_q "drop publication if exists $PUB" + src_q "select pg_drop_replication_slot(slot_name) from pg_replication_slots where slot_name = '$SUB' and not active" >/dev/null + if [ "$SOURCE_MODE" = railway ]; then + local proxy + proxy=$(rw_tcp_proxy) + if [ -n "$proxy" ]; then + gql 'mutation($id:String!){tcpProxyDelete(id:$id)}' "$(jq -nc --arg id "$(awk '{print $3}' <<<"$proxy")" '{id:$id}')" >/dev/null + note "TCP proxy removed" + fi + fi + note "The Railway Postgres service itself is left in place (read-only). Delete it by hand once you no longer want the fallback." +} + +cmd_apply_migration() { # apply-migration packages/db/migrations/00xx_name.sql + local f=${1:?usage: nichedb-db apply-migration } + log "Applying $(basename "$f") to the target (schema only; its schema_migrations row arrives through replication)" + tgt_sql postgres < "$f" + tgt_q "alter subscription $SUB enable" 2>/dev/null || true + note "subscription re-enabled" +} + +usage() { + cat <<'EOF' +nichedb-db: move nichedb's Postgres from Railway to self-hosted Supabase. + +In order: + provision user@host install Supabase on the new server (setup-server.sh) + dns [ipv4] db.nichedb.dev + supabase.nichedb.dev A records (Porkbun) + vault push the server's secrets to the logicsrc team vaults + check versions, sizes and free disk on both sides + source-prep Railway: wal_level=logical (RESTART), TCP proxy, publication + schema copy tables + primary keys to the target + subscribe start logical replication (initial copy, then streaming) + status copy progress, slot lag, errors + indexes build indexes/unique/foreign keys once the copy is done + verify schema diff + row counts (FULL=1 counts items too) + cutover source read-only, drain, sequences, switch DATABASE_URL + rollback point the app back at Railway and make it writable + cleanup drop publication/slot and the TCP proxy (after you're sure) + +Other: + fetch-connection re-read nichedb-connection.env from the server + apply-migration FILE a migration shipped mid-copy: apply its SQL to the target + and re-enable the subscription + +Flags: --yes skips confirmations. State lives in ~/.local/state/nichedb-selfhost. +EOF +} + +main() { + local args=() + while [ $# -gt 0 ]; do + case "$1" in + --yes | -y) YES=1 ;; + *) args+=("$1") ;; + esac + shift + done + set -- "${args[@]:-help}" + local cmd=$1 + shift || true + case "$cmd" in + provision) cmd_provision "$@" ;; + fetch-connection) cmd_fetch_connection ;; + vault) cmd_vault ;; + dns) cmd_dns "$@" ;; + check) cmd_check ;; + source-prep) cmd_source_prep ;; + schema) cmd_schema ;; + subscribe) cmd_subscribe ;; + status) cmd_status ;; + indexes) cmd_indexes ;; + verify) cmd_verify ;; + cutover) cmd_cutover ;; + rollback) cmd_rollback ;; + cleanup) cmd_cleanup ;; + apply-migration) cmd_apply_migration "$@" ;; + help | -h | --help) usage ;; + *) usage; exit 1 ;; + esac +} + +# NICHEDB_DB_LIB=1 lets test/make-fixture.sh source the helpers without main. +[ "${NICHEDB_DB_LIB:-0}" = 1 ] || main "$@" diff --git a/ops/selfhost-supabase/server/setup-server.sh b/ops/selfhost-supabase/server/setup-server.sh new file mode 100755 index 0000000..f42e313 --- /dev/null +++ b/ops/selfhost-supabase/server/setup-server.sh @@ -0,0 +1,371 @@ +#!/usr/bin/env bash +# +# Stand up self-hosted Supabase for nichedb on a fresh Ubuntu/Debian box. +# Run as root ON THE DATABASE SERVER. Idempotent: a second run keeps the +# generated secrets, certificate and data, and only rewrites the nichedb +# overlay (tuning, pg_hba, compose override) before restarting the stack. +# +# What it does: +# 1. System: base packages, Docker log rotation, ufw (ssh + 80/443 + Postgres) +# 2. Supabase: official setup.sh at a pinned self-hosted release tag +# 3. .env: public URLs, signups off, compose overrides (Caddy + nichedb) +# 4. Postgres: self-signed TLS cert, pg_hba that only lets `postgres` in from +# outside and only over TLS, tuning sized from this box's RAM and CPUs +# 5. Start the stack, then take nichedb's tables away from the anon and +# authenticated roles so PostgREST can never serve them +# 6. Write nichedb-connection.env (0600) with the app's DATABASE_URL +# +# Knobs (environment variables): +# SUPABASE_REF self-hosted/v0.8.2 pinned release of supabase/docker +# INSTALL_ROOT /opt parent of the project directory +# PROJECT nichedb-supabase project directory name +# DB_DOMAIN db.nichedb.dev Postgres host name (cert SAN) +# STUDIO_DOMAIN supabase.nichedb.dev Studio + API over Caddy HTTPS +# SITE_URL https://nichedb.dev +# DB_PORT 5432 public Postgres port on the host +# PGDATA_DIR (project)/volumes/db/data put PGDATA on a separate disk +# MEM_MB / CPUS detected override the tuning inputs +# WITH_PROXY 1 0 = no Caddy (no 80/443) +# SKIP_SYSTEM 0 1 = no apt/ufw/docker (rehearsal) +# ONLY_DB 0 1 = start only the db service +# +set -euo pipefail + +SUPABASE_REF=${SUPABASE_REF:-self-hosted/v0.8.2} +INSTALL_ROOT=${INSTALL_ROOT:-/opt} +PROJECT=${PROJECT:-nichedb-supabase} +DB_DOMAIN=${DB_DOMAIN:-db.nichedb.dev} +STUDIO_DOMAIN=${STUDIO_DOMAIN:-supabase.nichedb.dev} +SITE_URL=${SITE_URL:-https://nichedb.dev} +DB_PORT=${DB_PORT:-5432} +PGDATA_DIR=${PGDATA_DIR:-} +# Fixed subnet for the compose network, so pg_hba can tell the gateway (where +# Docker's userland proxy makes outside clients appear) from real services. +DOCKER_SUBNET=${DOCKER_SUBNET:-172.31.250.0/24} +DOCKER_GATEWAY=${DOCKER_GATEWAY:-172.31.250.1} +WITH_PROXY=${WITH_PROXY:-1} +SKIP_SYSTEM=${SKIP_SYSTEM:-0} +ONLY_DB=${ONLY_DB:-0} + +DIR="$INSTALL_ROOT/$PROJECT" +PG_UID=100 # postgres inside supabase/postgres:17.6.1.x +PG_GID=101 + +log() { printf '\n===> %s\n' "$*"; } +warn() { printf 'WARNING: %s\n' "$*" >&2; } +die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; } + +[ "$SKIP_SYSTEM" = 1 ] || [ "$(id -u)" = 0 ] || die "run as root (or SKIP_SYSTEM=1 for a rehearsal)" + +# ---------------------------------------------------------------- 1. system +backup() { # never overwrite without a numbered copy beside the original + local f=$1 n=1 dir name base ext + [ -e "$f" ] || return 0 + dir=$(dirname "$f") + name=$(basename "$f") + if [[ "${name#.}" == *.* ]]; then base=${name%.*} ext=".${name##*.}"; else base=$name ext=""; fi + while [ -e "$dir/$base.bak-$(printf %03d $n)$ext" ]; do n=$((n + 1)); done + cp -a "$f" "$dir/$base.bak-$(printf %03d $n)$ext" +} + +ssh_ports() { + ss -ltnpH 2>/dev/null | awk '/"sshd"/ {n=split($4,a,":"); print a[n]}' | sort -u +} + +system_setup() { + log "Base packages" + export DEBIAN_FRONTEND=noninteractive + apt-get update -qq + apt-get install -qq -y curl ca-certificates openssl jq ufw >/dev/null + + log "Docker log rotation" + mkdir -p /etc/docker + if [ ! -s /etc/docker/daemon.json ]; then + printf '{\n "log-driver": "json-file",\n "log-opts": { "max-size": "50m", "max-file": "3" }\n}\n' > /etc/docker/daemon.json + elif ! grep -q max-size /etc/docker/daemon.json; then + warn "/etc/docker/daemon.json exists without log rotation; left untouched" + fi + + log "Firewall" + local p ports + ports=$(ssh_ports) + [ -n "$ports" ] || ports=22 + for p in $ports; do ufw allow "$p/tcp" >/dev/null; done + ufw allow "$DB_PORT/tcp" >/dev/null + if [ "$WITH_PROXY" = 1 ]; then + ufw allow 80/tcp >/dev/null + ufw allow 443/tcp >/dev/null + ufw allow 443/udp >/dev/null + fi + ufw --force enable >/dev/null + # Docker publishes ports through its own iptables chains, ahead of ufw, so + # the rules above document intent; the real gate for 5432 is pg_hba + TLS. + ufw status | sed 's/^/ /' +} + +# ------------------------------------------------------------- 2. supabase +supabase_setup() { + mkdir -p "$INSTALL_ROOT" + if [ -f "$DIR/.env" ] && [ -f "$DIR/docker-compose.yml" ]; then + log "Supabase project already at $DIR; keeping its secrets" + return + fi + log "Supabase $SUPABASE_REF into $DIR" + local tmp + tmp=$(mktemp -d) + curl -fsSL "https://raw.githubusercontent.com/supabase/supabase/$SUPABASE_REF/docker/setup.sh" -o "$tmp/setup.sh" + local flags=(--ref "$SUPABASE_REF" -p "$PROJECT" -y) + [ "$SKIP_SYSTEM" = 1 ] && flags+=(--skip-deps) + # setup.sh prints every generated secret; keep that off the terminal (and + # out of whatever ssh session is watching) in a root-only log instead. + local slog="$INSTALL_ROOT/$PROJECT-setup.log" + (umask 077 && : > "$slog") + if ! (cd "$INSTALL_ROOT" && sh "$tmp/setup.sh" "${flags[@]}") >> "$slog" 2>&1; then + grep -E '^(===>|ERROR|WARNING)' "$slog" | tail -n 20 >&2 + die "Supabase setup.sh failed; full log (contains secrets): $slog" + fi + grep -E '^===>' "$slog" | grep -v -i 'key\|secret' | sed 's/^/ /' || true + rm -rf "$tmp" +} + +set_env() { # set_env KEY VALUE -> replace or append in $DIR/.env + local k=$1 v=$2 + if grep -q "^$k=" "$DIR/.env"; then + sed -i "s|^$k=.*$|$k=$v|" "$DIR/.env" + else + printf '%s=%s\n' "$k" "$v" >> "$DIR/.env" + fi +} +get_env() { grep "^$1=" "$DIR/.env" | head -n1 | cut -d= -f2-; } + +configure_env() { + log "Configuring .env" + backup "$DIR/.env" + set_env SUPABASE_PUBLIC_URL "https://$STUDIO_DOMAIN" + set_env API_EXTERNAL_URL "https://$STUDIO_DOMAIN/auth/v1" + set_env SITE_URL "$SITE_URL" + set_env PROXY_DOMAIN "$STUDIO_DOMAIN" + set_env CERTBOT_EMAIL "admin@${DB_DOMAIN#*.}" + set_env POOLER_TENANT_ID nichedb + set_env STUDIO_DEFAULT_ORGANIZATION "Profullstack" + set_env STUDIO_DEFAULT_PROJECT "nichedb" + # nichedb authenticates its own users; GoTrue must not hand out accounts. + set_env DISABLE_SIGNUP true + set_env ENABLE_ANONYMOUS_USERS false + local files=docker-compose.yml + [ "$WITH_PROXY" = 1 ] && files="$files:docker-compose.caddy.yml" + files="$files:docker-compose.nichedb.yml" + set_env COMPOSE_FILE "$files" + chmod 600 "$DIR/.env" +} + +# --------------------------------------------------------------- 4. postgres +detect_resources() { + MEM_MB=${MEM_MB:-$(awk '/MemTotal/ {print int($2/1024)}' /proc/meminfo)} + CPUS=${CPUS:-$(nproc)} +} + +write_tls() { + local tls="$DIR/volumes/nichedb/tls" + mkdir -p "$tls" + if [ ! -s "$tls/server.key" ]; then + log "Self-signed TLS certificate for $DB_DOMAIN (10 years)" + openssl req -x509 -nodes -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \ + -days 3650 -subj "/CN=$DB_DOMAIN" -addext "subjectAltName=DNS:$DB_DOMAIN" \ + -keyout "$tls/server.key" -out "$tls/server.crt" 2>/dev/null + fi + if [ -O "$tls/server.key" ]; then + chmod 600 "$tls/server.key" + chmod 644 "$tls/server.crt" + fi + if [ "$(id -u)" = 0 ]; then + chown "$PG_UID:$PG_GID" "$tls/server.key" "$tls/server.crt" + elif [ -O "$tls/server.key" ]; then + # Rehearsal as a normal user: the key must still belong to the container's postgres. + docker run --rm -v "$tls:/tls" --entrypoint /bin/chown alpine:3 "$PG_UID:$PG_GID" /tls/server.key /tls/server.crt + fi +} + +write_pg_hba() { + cat > "$DIR/volumes/nichedb/pg_hba.conf" < "$DIR/volumes/nichedb/nichedb.conf" < "$DIR/docker-compose.nichedb.yml" + if [ -n "$PGDATA_DIR" ]; then + mkdir -p "$PGDATA_DIR" + fi +} + +# ------------------------------------------------------------------ 5. start +compose() { (cd "$DIR" && docker compose "$@"); } + +start_stack() { + log "Starting the stack" + if [ "$ONLY_DB" = 1 ]; then + compose up -d --wait db + else + compose up -d --wait + fi + # A config change on an already-running db needs a restart to take effect. + compose restart db >/dev/null + + for _ in $(seq 1 60); do + docker exec supabase-db pg_isready -U postgres -h localhost >/dev/null 2>&1 && break + sleep 2 + done +} + +sql_admin() { docker exec -i supabase-db psql -U supabase_admin -h localhost -d postgres -v ON_ERROR_STOP=1 -X -q -At "$@"; } + +lock_down_public() { + log "Keeping nichedb's tables away from PostgREST's anon/authenticated roles" + sql_admin <<'EOF' +-- Supabase grants anon/authenticated everything new in public by default. +-- nichedb serves its own API and never uses PostgREST, so take it all back. +alter default privileges for role postgres in schema public revoke all on tables from anon, authenticated; +alter default privileges for role postgres in schema public revoke all on sequences from anon, authenticated; +alter default privileges for role postgres in schema public revoke all on functions from anon, authenticated; +alter default privileges for role supabase_admin in schema public revoke all on tables from anon, authenticated; +alter default privileges for role supabase_admin in schema public revoke all on sequences from anon, authenticated; +alter default privileges for role supabase_admin in schema public revoke all on functions from anon, authenticated; +revoke all on all tables in schema public from anon, authenticated; +revoke all on all sequences in schema public from anon, authenticated; +revoke all on all functions in schema public from anon, authenticated; +-- USAGE (and EXECUTE on functions) also reach anon through PUBLIC. Only the +-- app's role and service_role get into the schema at all. +revoke usage on schema public from public, anon, authenticated; +grant usage on schema public to postgres, service_role; +EOF +} + +check_postgres() { + log "Checks" + sql_admin -c "select 'ssl='||current_setting('ssl')||' shared_buffers='||current_setting('shared_buffers')||' hba='||current_setting('hba_file')||' version='||current_setting('server_version')" + local anon + anon=$(sql_admin -c "select has_schema_privilege('anon','public','usage') or has_schema_privilege('authenticated','public','usage')") + [ "$anon" = f ] || die "anon/authenticated can still use schema public" + echo "anon/authenticated locked out of schema public" + df -h "${PGDATA_DIR:-$DIR/volumes/db/data}" | tail -1 | awk '{print "data disk: "$4" free of "$2}' +} + +write_connection() { + local pw + pw=$(get_env POSTGRES_PASSWORD) + umask 077 + cat > "$DIR/nichedb-connection.env" < [ITEMS=20000] +# +set -euo pipefail +HERE=$(cd "$(dirname "$(readlink -f "$0")")" && pwd) +OUT=${1:?usage: make-fixture.sh } +ITEMS=${ITEMS:-20000} +mkdir -p "$OUT" +# Reuse the orchestrator's Railway transport (src_exec) without running main. +# shellcheck disable=SC1091 +NICHEDB_DB_LIB=1 . "$HERE/../nichedb-db" + +SMALL="collections sources feeds runs collection_stats data_dumps niches schema_migrations api_usage auto_place_cache contribution_tiers" + +echo "schema" >&2 +src_exec pg_dump -U postgres -d "$SOURCE_DB" --schema-only --no-owner --no-privileges > "$OUT/schema.sql" +echo "reference tables: $SMALL" >&2 +args=() +for t in $SMALL; do args+=(-t "public.$t"); done +src_exec pg_dump -U postgres -d "$SOURCE_DB" --data-only --no-owner "${args[@]}" > "$OUT/small.sql" +echo "newest $ITEMS items" >&2 +cols=$(src_q "select string_agg(quote_ident(column_name), ',' order by ordinal_position) from information_schema.columns where table_schema = 'public' and table_name = 'items' and is_generated = 'NEVER'") +printf '%s\n' "$cols" > "$OUT/items.columns" +src_exec psql -U postgres -d "$SOURCE_DB" -X -q -c "copy (select $cols from items order by id desc limit $ITEMS) to stdout" > "$OUT/items.copy" +wc -c "$OUT"/* >&2 diff --git a/ops/selfhost-supabase/test/rehearse.sh b/ops/selfhost-supabase/test/rehearse.sh new file mode 100755 index 0000000..28d2669 --- /dev/null +++ b/ops/selfhost-supabase/test/rehearse.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash +# +# Full dress rehearsal on this machine, no Railway involved: +# source postgres:18 + wal_level=logical, loaded from a fixture +# (make-fixture.sh: production schema + sample rows) +# target the real server/setup-server.sh (SKIP_SYSTEM=1, db only, port 15432) +# run schema -> subscribe -> copy -> indexes -> verify -> cutover +# while a writer inserts/updates/deletes items on the source, +# then asserts counts match, sequences continue, and the security +# rules hold (TLS only, postgres only, anon locked out). +# +# usage: rehearse.sh [workdir] KEEP=1 leaves it running +# +set -euo pipefail +HERE=$(cd "$(dirname "$(readlink -f "$0")")" && pwd) +FIX=$(cd "${1:?usage: rehearse.sh [workdir]}" && pwd) +WORK=${2:-$(mktemp -d)} +mkdir -p "$WORK" +WORK=$(cd "$WORK" && pwd) +NET=nichedb-rehearsal +SRC=nichedb-rehearsal-src +SRCPW=rehearsal-source-pw +PORT=${PORT:-15432} + +step() { printf '\n######## %s\n' "$*"; } +fail() { printf '\nREHEARSAL FAILED: %s\n' "$*" >&2; exit 1; } +src() { docker exec -i "$SRC" psql -U postgres -d railway -v ON_ERROR_STOP=1 -X -q -At "$@"; } +tgt() { docker exec -i supabase-db psql -h localhost -U supabase_admin -d postgres -v ON_ERROR_STOP=1 -X -q -At "$@"; } + +cleanup() { + [ -n "${WRITER:-}" ] && kill "$WRITER" 2>/dev/null || true + [ "${KEEP:-0}" = 1 ] && { echo "KEEP=1: left running (workdir $WORK)"; return; } + (cd "$WORK/nichedb-supabase" 2>/dev/null && docker compose down -v >/dev/null 2>&1) || true + docker rm -f "$SRC" >/dev/null 2>&1 || true + docker network rm "$NET" >/dev/null 2>&1 || true + # Postgres wrote the data dir as its own uid: remove it through Docker. + docker run --rm -v "$WORK:/w" alpine:3 rm -rf /w/nichedb-supabase /w/state >/dev/null 2>&1 || true +} +trap cleanup EXIT + +step "source: postgres:18, wal_level=logical" +docker network create "$NET" >/dev/null 2>&1 || true +docker rm -f "$SRC" >/dev/null 2>&1 || true +docker run -d --name "$SRC" --network "$NET" -e POSTGRES_PASSWORD="$SRCPW" -e POSTGRES_DB=railway \ + postgres:18 -c wal_level=logical -c max_slot_wal_keep_size=1GB >/dev/null +for _ in $(seq 1 60); do docker exec "$SRC" pg_isready -U postgres -d railway >/dev/null 2>&1 && break; sleep 1; done +sleep 2 + +step "source: load fixture" +src < "$FIX/schema.sql" >/dev/null +docker exec -i -e PGOPTIONS='-c session_replication_role=replica' "$SRC" \ + psql -U postgres -d railway -v ON_ERROR_STOP=1 -X -q < "$FIX/small.sql" >/dev/null +docker exec -i -e PGOPTIONS='-c session_replication_role=replica' "$SRC" \ + psql -U postgres -d railway -v ON_ERROR_STOP=1 -X -q -c "copy items ($(cat "$FIX/items.columns")) from stdin" < "$FIX/items.copy" +# The fixture is a slice: drop rows whose FK parents were not copied, so the +# target can validate its foreign keys exactly as production will. +src <<'EOF' +do $$ +declare r record; n bigint; total bigint; +begin + loop + total := 0; + for r in + select c.conrelid::regclass as child, c.confrelid::regclass as parent, + (select string_agg('t.'||quote_ident(a.attname), ',' order by k.o) from unnest(c.conkey) with ordinality k(n, o) join pg_attribute a on a.attrelid = c.conrelid and a.attnum = k.n) as ccols, + (select string_agg('p.'||quote_ident(a.attname), ',' order by k.o) from unnest(c.confkey) with ordinality k(n, o) join pg_attribute a on a.attrelid = c.confrelid and a.attnum = k.n) as pcols + from pg_constraint c where c.contype = 'f' and c.connamespace = 'public'::regnamespace + loop + execute format('delete from %s t where row(%s) is not null and not exists (select 1 from %s p where row(%s) = row(%s))', + r.child, r.ccols, r.parent, r.pcols, r.ccols); + get diagnostics n = row_count; + total := total + n; + end loop; + exit when total = 0; + end loop; +end $$; +select setval(pg_get_serial_sequence('items', 'id'), (select max(id) from items)); +EOF +echo "source items: $(src -c 'select count(*) from items')" + +step "target: setup-server.sh (rehearsal mode)" +# setup.sh pulls every Supabase image; the rehearsal only needs the db, so a +# PATH shim turns `docker compose pull` into a no-op. +mkdir -p "$WORK/bin" +REAL_DOCKER=$(command -v docker) +cat > "$WORK/bin/docker" </dev/null || true + +export SOURCE_MODE=docker SOURCE_CONTAINER=$SRC SOURCE_PASSWORD=$SRCPW +export TARGET_SSH=local TARGET_DIR=$WORK/nichedb-supabase STATE_DIR=$WORK/state +DB="$HERE/../nichedb-db" + +step "orchestrator: fetch-connection, check, source-prep" +"$DB" fetch-connection +"$DB" check +"$DB" source-prep + +step "orchestrator: schema" +"$DB" schema +[ "$(tgt -c "select count(*) from pg_tables where schemaname='public'")" = "$(src -c "select count(*) from pg_tables where schemaname='public'")" ] || fail "table count differs after schema" + +step "writer: inserts/updates/deletes on the source for the whole run" +( + while :; do + src >/dev/null 2>&1 < 'r'")" = 0 ] && break ;; esac + sleep 3 +done +"$DB" status +[ "$(tgt -c "select count(*) from pg_subscription_rel where srsubstate <> 'r'")" = 0 ] || fail "initial copy never finished" + +step "orchestrator: indexes (while the writer keeps going)" +"$DB" indexes +[ "$(tgt -c "select count(*) from pg_indexes where schemaname='public'")" = "$(src -c "select count(*) from pg_indexes where schemaname='public'")" ] || fail "index count differs" + +step "orchestrator: verify (schema must be identical)" +"$DB" verify || fail "schema diff between source and target" + +step "orchestrator: cutover (writer still running: it must start failing)" +"$DB" cutover --yes +sleep 2 +kill "$WRITER" 2>/dev/null || true +WRITER= +[ "$(src -c "show default_transaction_read_only")" = on ] || fail "source is not read-only after cutover" +r=$(src -c "update items set title = title where id = (select max(id) from items)" 2>&1) && fail "source still accepts writes after cutover" +echo " source write after cutover: rejected ($(cut -c1-70 <<<"$r"))" + +step "assertions" +FULL=1 "$DB" verify 2>&1 | tee "$WORK/verify.txt" +grep -q DIFF "$WORK/verify.txt" && fail "row counts differ after cutover" +s=$(src -c "select count(*)||' '||max(id)||' '||md5(string_agg(id||title||updated_at, ',' order by id)) from items") +t=$(tgt -c "select count(*)||' '||max(id)||' '||md5(string_agg(id||title||updated_at, ',' order by id)) from items") +echo " items source: $s" +echo " items target: $t" +[ "$s" = "$t" ] || fail "items content differs" +[ "$(src -c "select md5(string_agg(search::text, ',' order by id)) from items")" = "$(tgt -c "select md5(string_agg(search::text, ',' order by id)) from items")" ] || fail "generated search column differs" +echo " generated search column: identical" +# Sequences: a fresh insert on the target must not collide with a copied id. +tgt -c "insert into items (collection_id, source_id, external_id, title) select collection_id, source_id, 'rehearsal-after-cutover', 'x' from items limit 1 returning id" >/dev/null || + fail "insert after cutover failed (sequence not advanced?)" +echo " insert after cutover: ok" +[ "$(tgt -c "select count(*) from pg_subscription")" = 0 ] || fail "subscription still present" +[ "$(src -c "select count(*) from pg_replication_slots")" = 0 ] || fail "slot left on the source" +echo " subscription and slot gone" + +step "security" +pg() { docker run --rm --network host -e PGCONNECT_TIMEOUT=5 postgres:18 psql "$1" -X -At -c "$2" 2>&1; } +PW=$(grep '^POSTGRES_PASSWORD=' "$WORK/nichedb-supabase/.env" | cut -d= -f2-) +r=$(pg "postgres://postgres:$PW@127.0.0.1:$PORT/postgres?sslmode=require" "select ssl from pg_stat_ssl where pid = pg_backend_pid()") +[ "$r" = t ] || fail "postgres over TLS: $r" +echo " postgres over TLS: allowed" +r=$(pg "postgres://postgres:$PW@127.0.0.1:$PORT/postgres?sslmode=disable" "select 1") && fail "postgres without TLS was allowed" +echo " postgres without TLS: rejected ($(tail -n1 <<<"$r" | cut -c1-90))" +r=$(pg "postgres://supabase_admin:$PW@127.0.0.1:$PORT/postgres?sslmode=require" "select 1") && fail "supabase_admin from outside was allowed" +echo " supabase_admin from outside: rejected" +[ "$(tgt -c "select has_table_privilege('anon','public.items','select') or has_schema_privilege('anon','public','usage') or has_table_privilege('authenticated','public.items','select')")" = f ] || + fail "anon/authenticated can read nichedb tables" +echo " anon/authenticated: no access to public" +# The app's own driver, with the URL shape the cutover sets. +APPURL="postgres://postgres:$PW@127.0.0.1:$PORT/postgres?sslmode=require" +r=$(DATABASE_URL="$APPURL" bun "$HERE/app-driver-check.js" 2>&1) || true +[ "$r" = "$(tgt -c 'select count(*) from items') true" ] || fail "app driver could not read the target over TLS: $r" +echo " app driver (Bun SQL, sslmode=require): read ${r% *} items over TLS" + +printf '\nREHEARSAL PASSED\n' From 5b95e807e8c153e180a1a956f18df973c5a298ee Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Wed, 23 Sep 2026 22:19:36 +0000 Subject: [PATCH 02/12] ops: refresh the Railway CLI token before API calls The access token lives about an hour, after which every GraphQL call is 'Not Authorized'. A mid-cutover expiry would stop the DATABASE_URL switch. Co-Authored-By: Claude Opus 5.5 --- ops/selfhost-supabase/nichedb-db | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index 693ce1f..411868e 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -66,7 +66,14 @@ confirm() { # confirm "question" (skipped by --yes) } # --------------------------------------------------------------- railway API -rw_token() { jq -r '.user.accessToken' "$HOME/.railway/config.json"; } +rw_token() { + # The CLI's access token lives about an hour; every API call then answers + # "Not Authorized". Any CLI command refreshes it, so refresh when close. + local exp + exp=$(jq -r '.user.tokenExpiresAt // 0' "$HOME/.railway/config.json") + if [ "$exp" -lt $(($(date +%s) + 300)) ]; then railway whoami >/dev/null 2>&1 || die "railway login expired: run 'railway login'"; fi + jq -r '.user.accessToken' "$HOME/.railway/config.json" +} gql() { # gql QUERY [VARIABLES_JSON] local body From bee8f2d9bfa9a46372787986abe6172f1df33cae Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 19:37:53 +0000 Subject: [PATCH 03/12] ops: shared-cluster mode, nichedb as a database in an existing Supabase prepare-db creates the database inside a cluster another app already runs on the box (one Supabase stack per host), locks anon/authenticated out of that database only, and writes the connection file. dns/vault skip what belongs to the cluster's owner. Index log lives in /tmp, not their dir. Co-Authored-By: Claude Fable 5.1 --- ops/selfhost-supabase/README.md | 22 ++++++++ ops/selfhost-supabase/nichedb-db | 96 +++++++++++++++++++++++++++----- 2 files changed, 103 insertions(+), 15 deletions(-) diff --git a/ops/selfhost-supabase/README.md b/ops/selfhost-supabase/README.md index 725acc9..86557ed 100644 --- a/ops/selfhost-supabase/README.md +++ b/ops/selfhost-supabase/README.md @@ -81,6 +81,28 @@ $K cutover # the switch (below) $K cleanup # drop publication/slot, remove the TCP proxy ``` +### Sharing a cluster another app already owns + +A box runs one Supabase stack (fixed container names, one 5432). When another +app's stack is already there, nichedb becomes a database inside it instead: + +```sh +$K prepare-db root@ /path/to/that/supabase-project nichedb +``` + +instead of `provision`. It creates the database, enables citext + pg_trgm, +locks anon/authenticated out of that database's `public` (the cluster's own +`postgres` database is untouched), and writes `nichedb-connection.env` into +the project directory. The rest of the runbook is unchanged; `dns` then sets +only `db.nichedb.dev`, and `vault` leaves the cluster's `.env` to its owner. +The cluster needs `wal_level=logical`, a free replication slot, and a pg_hba +rule that lets `postgres` in over TLS. What this costs: one restart, bad +config or OOM takes both apps' databases down together. + +This is how it went live: dev2 (23.95.228.174, Buffalo NY, 16-core Ryzen, +91 GB, 1.8 TB NVMe) already ran crawlproof.com's cluster at +`/home/anthony/www/crawlproof.com/supabase`. + `cutover` does this, in order, and refuses to start unless every table is ready, the index counts match, the schemas are identical and lag is under 64 MB: diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index 411868e..9e212f3 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -30,6 +30,9 @@ SOURCE_CONTAINER=${SOURCE_CONTAINER:-} TARGET_SSH=${TARGET_SSH:-} TARGET_DIR=${TARGET_DIR:-/opt/nichedb-supabase} TARGET_DB=${TARGET_DB:-postgres} +# SHARED_CLUSTER=1: the target is a database inside a Supabase cluster that +# another app already owns on the box (`prepare-db` instead of `provision`). +SHARED_CLUSTER=${SHARED_CLUSTER:-0} PUB=${PUB:-nichedb_migrate} SUB=${SUB:-nichedb_migrate} APP_HEALTH_URL=${APP_HEALTH_URL:-https://nichedb.dev/healthz} @@ -153,11 +156,12 @@ tgt_exec() { # run a shell command string on the target host ssh -o BatchMode=yes -o ServerAliveInterval=30 "$TARGET_SSH" "$1" fi } -tgt_sudo() { if [ "$TARGET_SSH" = local ]; then printf ''; else printf 'sudo -n '; fi; } +tgt_sudo() { case "$TARGET_SSH" in local | root@*) printf '' ;; *) printf 'sudo -n ' ;; esac; } -tgt_sql() { # tgt_sql ROLE (SQL on stdin) - ROLE is supabase_admin or postgres - tgt_exec "$(tgt_sudo)docker exec -i supabase-db psql -h localhost -U $1 -d $TARGET_DB -v ON_ERROR_STOP=1 -X -q -At" +tgt_sql_in() { # tgt_sql_in DB ROLE (SQL on stdin) + tgt_exec "$(tgt_sudo)docker exec -i supabase-db psql -h localhost -U $2 -d $1 -v ON_ERROR_STOP=1 -X -q -At" } +tgt_sql() { tgt_sql_in "$TARGET_DB" "$1"; } # tgt_sql ROLE - supabase_admin or postgres tgt_q() { tgt_sql supabase_admin <<<"$1"; } tgt_copy() { # tgt_copy LOCAL_FILE REMOTE_PATH @@ -227,6 +231,58 @@ cmd_provision() { # provision user@host cmd_fetch_connection } +cmd_prepare_db() { # prepare-db user@host /path/to/supabase-project [dbname] + local host=${1:-$TARGET_SSH} dir=${2:-$TARGET_DIR} db=${3:-${TARGET_DB_SHARED:-nichedb}} + [ -n "$host" ] && [ -n "$dir" ] || die "usage: nichedb-db prepare-db user@host /path/to/supabase-project [dbname]" + save_state TARGET_SSH "$host" + save_state TARGET_DIR "$dir" + save_state TARGET_DB "$db" + save_state SHARED_CLUSTER 1 + log "Preparing database $db in the Supabase cluster at $host:$dir" + tgt_exec "test -f $dir/.env && $(tgt_sudo)docker exec supabase-db pg_isready -h localhost -U postgres" >/dev/null || + die "no running Supabase cluster with $dir/.env on $host" + local wal + wal=$(tgt_sql_in postgres supabase_admin <<<"show wal_level") + [ "$wal" = logical ] || die "cluster wal_level is $wal, not logical" + if [ "$(tgt_sql_in postgres supabase_admin <<<"select count(*) from pg_database where datname = '$db'")" = 0 ]; then + tgt_sql_in postgres supabase_admin <<<"create database $db owner postgres" + note "database $db created" + else + note "database $db already exists" + fi + tgt_sql_in "$db" supabase_admin <<'EOF' +create extension if not exists citext with schema public; +create extension if not exists pg_trgm with schema public; +-- This database is the app's alone: PostgREST/anon never get into it. (Only +-- this database; the cluster's own `postgres` database is not touched.) +alter default privileges for role postgres in schema public revoke all on tables from anon, authenticated; +alter default privileges for role postgres in schema public revoke all on sequences from anon, authenticated; +alter default privileges for role postgres in schema public revoke all on functions from anon, authenticated; +revoke all on all tables in schema public from anon, authenticated; +revoke all on all sequences in schema public from anon, authenticated; +revoke all on all functions in schema public from anon, authenticated; +revoke usage on schema public from public, anon, authenticated; +grant usage on schema public to postgres, service_role; +EOF + note "extensions citext + pg_trgm; anon/authenticated locked out of $db" + local pw + pw=$(tgt_exec "$(tgt_sudo)grep '^POSTGRES_PASSWORD=' $dir/.env | cut -d= -f2-") + [ -n "$pw" ] || die "no POSTGRES_PASSWORD in $dir/.env" + local port + port=$(tgt_exec "$(tgt_sudo)grep '^POSTGRES_PORT=' $dir/.env | cut -d= -f2-") + # Written by root into the shared project dir, 0600, same shape provision makes. + tgt_exec "umask 077; $(tgt_sudo)tee $dir/nichedb-connection.env >/dev/null" < "$work/supabase.env" - logicsrc teams push "$VAULT_TEAM" nichedb-supabase prod --env "$work/supabase.env" + # 1) the server's whole Supabase .env as its own vault (not when the cluster + # belongs to another app: its owner keeps that vault) + if [ "$SHARED_CLUSTER" != 1 ]; then + tgt_exec "$(tgt_sudo)cat $TARGET_DIR/.env" > "$work/supabase.env" + logicsrc teams push "$VAULT_TEAM" nichedb-supabase prod --env "$work/supabase.env" + fi # 2) SELFHOST_* next to the app's own secrets, merged, never replacing them if logicsrc teams pull "$VAULT_TEAM" nichedb prod --env "$work/app.env" >/dev/null 2>&1 && [ -s "$work/app.env" ]; then local n=1 @@ -268,11 +327,14 @@ cmd_vault() { cmd_dns() { local ip=${1:-${TARGET_IP:-}} [ -n "$ip" ] || die "usage: nichedb-db dns " - log "DNS: $DB_DOMAIN and $STUDIO_DOMAIN -> $ip" + log "DNS: $DB_DOMAIN -> $ip" porkbun set "$DNS_ZONE" "${DB_DOMAIN%%.$DNS_ZONE}" A "$ip" --ttl 600 - porkbun set "$DNS_ZONE" "${STUDIO_DOMAIN%%.$DNS_ZONE}" A "$ip" --ttl 600 [ -n "${TARGET_IPV6:-}" ] && porkbun set "$DNS_ZONE" "${DB_DOMAIN%%.$DNS_ZONE}" AAAA "$TARGET_IPV6" --ttl 600 - [ -n "${TARGET_IPV6:-}" ] && porkbun set "$DNS_ZONE" "${STUDIO_DOMAIN%%.$DNS_ZONE}" AAAA "$TARGET_IPV6" --ttl 600 + # A shared cluster's Studio belongs to the app that owns it. + if [ "$SHARED_CLUSTER" != 1 ]; then + porkbun set "$DNS_ZONE" "${STUDIO_DOMAIN%%.$DNS_ZONE}" A "$ip" --ttl 600 + [ -n "${TARGET_IPV6:-}" ] && porkbun set "$DNS_ZONE" "${STUDIO_DOMAIN%%.$DNS_ZONE}" AAAA "$TARGET_IPV6" --ttl 600 + fi return 0 } @@ -407,8 +469,8 @@ cmd_status() { note "source slot lag: $( [ -n "$lag" ] && numfmt --to=iec "$lag" 2>/dev/null || echo none)" src_q "select 'source slot '||slot_name||' wal_status='||wal_status||' safe_wal_size='||coalesce(pg_size_pretty(safe_wal_size),'unlimited') from pg_replication_slots where slot_name='$SUB'" | sed 's/^/ /' note "sizes: source $(src_q "select pg_size_pretty(pg_database_size(current_database()))"), target $(tgt_q "select pg_size_pretty(pg_database_size(current_database()))")" - if [ -n "$(tgt_exec "test -f $TARGET_DIR/indexes.log && echo y" 2>/dev/null || true)" ]; then - note "index build: $(tgt_exec "$(tgt_sudo)tail -n 2 $TARGET_DIR/indexes.log" | tr '\n' ' ')" + if [ -n "$(tgt_exec "test -f /tmp/nichedb-indexes.log && echo y" 2>/dev/null || true)" ]; then + note "index build: $(tgt_exec "tail -n 2 /tmp/nichedb-indexes.log" | tr '\n' ' ')" fi } @@ -435,12 +497,12 @@ EOF tgt_copy "$work/indexes.sh" /tmp/nichedb-indexes.sh rm -rf "$work" if [ "$TARGET_SSH" = local ]; then - sh /tmp/nichedb-indexes.sh > "$TARGET_DIR/indexes.log" 2>&1 || { tail -n 5 "$TARGET_DIR/indexes.log" >&2; die "index build failed"; } - tail -n 1 "$TARGET_DIR/indexes.log" >&2 + sh /tmp/nichedb-indexes.sh > /tmp/nichedb-indexes.log 2>&1 || { tail -n 5 /tmp/nichedb-indexes.log >&2; die "index build failed"; } + tail -n 1 /tmp/nichedb-indexes.log >&2 else # Detached: 26 GB of indexes outlives an ssh session. - tgt_exec "$(tgt_sudo)nohup sh /tmp/nichedb-indexes.sh > /tmp/nichedb-indexes.log 2>&1 < /dev/null & sleep 1; $(tgt_sudo)ln -sf /tmp/nichedb-indexes.log $TARGET_DIR/indexes.log" - note "running on the server; progress in $TARGET_DIR/indexes.log (nichedb-db status shows the tail)" + tgt_exec "$(tgt_sudo)nohup sh /tmp/nichedb-indexes.sh > /tmp/nichedb-indexes.log 2>&1 < /dev/null & sleep 1" + note "running on the server; progress in /tmp/nichedb-indexes.log (nichedb-db status shows the tail)" fi } @@ -596,6 +658,9 @@ nichedb-db: move nichedb's Postgres from Railway to self-hosted Supabase. In order: provision user@host install Supabase on the new server (setup-server.sh) + prepare-db user@host /path/to/supabase [db] + INSTEAD of provision when the box already runs a Supabase + cluster for another app: nichedb becomes a database in it dns [ipv4] db.nichedb.dev + supabase.nichedb.dev A records (Porkbun) vault push the server's secrets to the logicsrc team vaults check versions, sizes and free disk on both sides @@ -632,6 +697,7 @@ main() { shift || true case "$cmd" in provision) cmd_provision "$@" ;; + prepare-db) cmd_prepare_db "$@" ;; fetch-connection) cmd_fetch_connection ;; vault) cmd_vault ;; dns) cmd_dns "$@" ;; From 04c091fb1173f79609c8a3f1083eb497511d8cba Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 19:41:00 +0000 Subject: [PATCH 04/12] ops: no new DNS names; the app connects by the server's own hostname db.nichedb.dev was invented and is gone (record deleted). DB_DOMAIN now defaults to the target host, dns is opt-in, setup-server uses hostname -f. Co-Authored-By: Claude Fable 5.1 --- ops/selfhost-supabase/README.md | 11 +++++------ ops/selfhost-supabase/nichedb-db | 12 ++++++++---- ops/selfhost-supabase/server/setup-server.sh | 9 +++++---- 3 files changed, 18 insertions(+), 14 deletions(-) diff --git a/ops/selfhost-supabase/README.md b/ops/selfhost-supabase/README.md index 86557ed..9de36d3 100644 --- a/ops/selfhost-supabase/README.md +++ b/ops/selfhost-supabase/README.md @@ -26,8 +26,8 @@ is disk, and it is cheaper to own. What the overlay changes on stock Supabase: - **Postgres is published directly on 5432**, with TLS on and a self-signed - cert for `db.nichedb.dev`. The app already connects with - `sslmode=require` and does not verify the cert. + cert. The app connects by the server's existing hostname with + `sslmode=require` and does not verify the cert. No new DNS names. - **pg_hba**: from outside, only the `postgres` role gets in, and only over TLS. The compose network is pinned to `172.31.250.0/24` so the Docker gateway can be treated as outside. That covers clients Docker proxies: @@ -38,7 +38,7 @@ What the overlay changes on stock Supabase: - **Tuning** is sized from the box's RAM and CPUs: 25% `shared_buffers`, 16 GB `max_wal_size`, parallel maintenance, and autovacuum tuned for `items`. - **Supavisor** binds 127.0.0.1:6543 only. **Studio** sits behind Caddy with - HTTPS and basic auth at `https://supabase.nichedb.dev`. + HTTPS and basic auth on the server's hostname. - **Docker logs** rotate at 3 x 50 MB. **Supabase's `setup.sh` output**, which prints every secret, goes to a root-only log, never the terminal. @@ -63,7 +63,6 @@ What the overlay changes on stock Supabase: K=ops/selfhost-supabase/nichedb-db $K provision root@ # install + start Supabase, fetch the connection -$K dns # db.nichedb.dev + supabase.nichedb.dev -> server $K vault # server .env -> vault nichedb-supabase--prod, # SELFHOST_* merged into nichedb--prod (backup kept) $K check # versions, sizes, free disk on both sides @@ -93,8 +92,8 @@ $K prepare-db root@ /path/to/that/supabase-project nichedb instead of `provision`. It creates the database, enables citext + pg_trgm, locks anon/authenticated out of that database's `public` (the cluster's own `postgres` database is untouched), and writes `nichedb-connection.env` into -the project directory. The rest of the runbook is unchanged; `dns` then sets -only `db.nichedb.dev`, and `vault` leaves the cluster's `.env` to its owner. +the project directory. The rest of the runbook is unchanged, and `vault` +leaves the cluster's `.env` to its owner. The cluster needs `wal_level=logical`, a free replication slot, and a pg_hba rule that lets `postgres` in over TLS. What this costs: one restart, bad config or OOM takes both apps' databases down together. diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index 9e212f3..d4f1019 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -36,8 +36,10 @@ SHARED_CLUSTER=${SHARED_CLUSTER:-0} PUB=${PUB:-nichedb_migrate} SUB=${SUB:-nichedb_migrate} APP_HEALTH_URL=${APP_HEALTH_URL:-https://nichedb.dev/healthz} -DB_DOMAIN=${DB_DOMAIN:-db.nichedb.dev} -STUDIO_DOMAIN=${STUDIO_DOMAIN:-supabase.nichedb.dev} +# The app reaches the database by the server's existing hostname. No new DNS +# names: `dns` only runs when you pass DB_DOMAIN yourself. +DB_DOMAIN=${DB_DOMAIN:-${TARGET_SSH#*@}} +STUDIO_DOMAIN=${STUDIO_DOMAIN:-} DNS_ZONE=${DNS_ZONE:-nichedb.dev} VAULT_TEAM=${VAULT_TEAM:-profullstack} SSH_KEY=${SSH_KEY:-$HOME/.ssh/id_ed25519} @@ -327,11 +329,13 @@ cmd_vault() { cmd_dns() { local ip=${1:-${TARGET_IP:-}} [ -n "$ip" ] || die "usage: nichedb-db dns " + [ "$DB_DOMAIN" != "${TARGET_SSH#*@}" ] || die "DB_DOMAIN is the server's own name; nothing to add" + [[ "$DB_DOMAIN" == *.$DNS_ZONE ]] || die "DB_DOMAIN $DB_DOMAIN is not under $DNS_ZONE" log "DNS: $DB_DOMAIN -> $ip" porkbun set "$DNS_ZONE" "${DB_DOMAIN%%.$DNS_ZONE}" A "$ip" --ttl 600 [ -n "${TARGET_IPV6:-}" ] && porkbun set "$DNS_ZONE" "${DB_DOMAIN%%.$DNS_ZONE}" AAAA "$TARGET_IPV6" --ttl 600 # A shared cluster's Studio belongs to the app that owns it. - if [ "$SHARED_CLUSTER" != 1 ]; then + if [ "$SHARED_CLUSTER" != 1 ] && [ -n "$STUDIO_DOMAIN" ]; then porkbun set "$DNS_ZONE" "${STUDIO_DOMAIN%%.$DNS_ZONE}" A "$ip" --ttl 600 [ -n "${TARGET_IPV6:-}" ] && porkbun set "$DNS_ZONE" "${STUDIO_DOMAIN%%.$DNS_ZONE}" AAAA "$TARGET_IPV6" --ttl 600 fi @@ -661,7 +665,6 @@ In order: prepare-db user@host /path/to/supabase [db] INSTEAD of provision when the box already runs a Supabase cluster for another app: nichedb becomes a database in it - dns [ipv4] db.nichedb.dev + supabase.nichedb.dev A records (Porkbun) vault push the server's secrets to the logicsrc team vaults check versions, sizes and free disk on both sides source-prep Railway: wal_level=logical (RESTART), TCP proxy, publication @@ -675,6 +678,7 @@ In order: cleanup drop publication/slot and the TCP proxy (after you're sure) Other: + dns [ipv4] only with DB_DOMAIN=: add its A record fetch-connection re-read nichedb-connection.env from the server apply-migration FILE a migration shipped mid-copy: apply its SQL to the target and re-enable the subscription diff --git a/ops/selfhost-supabase/server/setup-server.sh b/ops/selfhost-supabase/server/setup-server.sh index f42e313..dff76d5 100755 --- a/ops/selfhost-supabase/server/setup-server.sh +++ b/ops/selfhost-supabase/server/setup-server.sh @@ -19,8 +19,8 @@ # SUPABASE_REF self-hosted/v0.8.2 pinned release of supabase/docker # INSTALL_ROOT /opt parent of the project directory # PROJECT nichedb-supabase project directory name -# DB_DOMAIN db.nichedb.dev Postgres host name (cert SAN) -# STUDIO_DOMAIN supabase.nichedb.dev Studio + API over Caddy HTTPS +# DB_DOMAIN $(hostname -f) Postgres host name (cert SAN) +# STUDIO_DOMAIN = DB_DOMAIN Studio + API over Caddy HTTPS # SITE_URL https://nichedb.dev # DB_PORT 5432 public Postgres port on the host # PGDATA_DIR (project)/volumes/db/data put PGDATA on a separate disk @@ -34,8 +34,9 @@ set -euo pipefail SUPABASE_REF=${SUPABASE_REF:-self-hosted/v0.8.2} INSTALL_ROOT=${INSTALL_ROOT:-/opt} PROJECT=${PROJECT:-nichedb-supabase} -DB_DOMAIN=${DB_DOMAIN:-db.nichedb.dev} -STUDIO_DOMAIN=${STUDIO_DOMAIN:-supabase.nichedb.dev} +# The server's own name: the app connects to the box it already knows. +DB_DOMAIN=${DB_DOMAIN:-$(hostname -f 2>/dev/null || hostname)} +STUDIO_DOMAIN=${STUDIO_DOMAIN:-$DB_DOMAIN} SITE_URL=${SITE_URL:-https://nichedb.dev} DB_PORT=${DB_PORT:-5432} PGDATA_DIR=${PGDATA_DIR:-} From 4b258987a25c1843e021c437b3fa4295d7cbec44 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 19:48:06 +0000 Subject: [PATCH 05/12] ops: move the app with its database, the fleet way app-deploy/app-cert/app-vhost stage nichedb.dev under ~/www/nichedb.dev (checkout, docker-compose.app.yml with Redis, app.env from Railway's variables, nginx vhost, cert pre-issued over Porkbun DNS-01 so the flip has no TLS gap). cutover then points Railway's app at the new DB as web only, starts the app on the box, and moves the A records at Porkbun. Co-Authored-By: Claude Fable 5.1 --- .../app/docker-compose.app.yml | 41 ++++++ ops/selfhost-supabase/app/nginx-vhost.conf | 44 ++++++ ops/selfhost-supabase/nichedb-db | 129 +++++++++++++++++- 3 files changed, 208 insertions(+), 6 deletions(-) create mode 100644 ops/selfhost-supabase/app/docker-compose.app.yml create mode 100644 ops/selfhost-supabase/app/nginx-vhost.conf diff --git a/ops/selfhost-supabase/app/docker-compose.app.yml b/ops/selfhost-supabase/app/docker-compose.app.yml new file mode 100644 index 0000000..6ef8d12 --- /dev/null +++ b/ops/selfhost-supabase/app/docker-compose.app.yml @@ -0,0 +1,41 @@ +# nichedb.dev on a fleet box: the app image built from the checked-out repo, +# plus its Redis. Postgres is the box's Supabase cluster (DATABASE_URL in .env). +# nginx on the host terminates TLS and proxies to 127.0.0.1:${APP_PORT}. +# +# Layout on the box (same shape as every other site under ~/www): +# ~/www/nichedb.dev/app git checkout of profullstack/niche-db +# ~/www/nichedb.dev/app.env the app's environment (0600) +# ~/www/nichedb.dev/deploy.env APP_PORT for this file +# ~/www/nichedb.dev/docker-compose.app.yml this file +services: + app: + build: + context: ./app + image: nichedb:latest + restart: unless-stopped + env_file: app.env + environment: + NODE_ENV: production + PORT: "3000" + REDIS_URL: redis://redis:6379 + ports: + - "127.0.0.1:${APP_PORT:-3010}:3000" + depends_on: + redis: + condition: service_started + logging: + driver: json-file + options: { max-size: "50m", max-file: "3" } + + redis: + image: redis:8-alpine + restart: unless-stopped + command: ["redis-server", "--appendonly", "yes", "--maxmemory", "2gb", "--maxmemory-policy", "noeviction"] + volumes: + - redisdata:/data + logging: + driver: json-file + options: { max-size: "20m", max-file: "2" } + +volumes: + redisdata: diff --git a/ops/selfhost-supabase/app/nginx-vhost.conf b/ops/selfhost-supabase/app/nginx-vhost.conf new file mode 100644 index 0000000..cedf7f0 --- /dev/null +++ b/ops/selfhost-supabase/app/nginx-vhost.conf @@ -0,0 +1,44 @@ +# nichedb.dev: managed by ops/selfhost-supabase/nichedb-db app-vhost +# The app runs in Docker on 127.0.0.1:__APP_PORT__; this is the only public face. + +server { + listen 80; + listen [::]:80; + server_name __DOMAIN__ www.__DOMAIN__; + + # HTTP-01 renewals (the first cert is issued over DNS-01 before DNS moves). + location ^~ /.well-known/acme-challenge/ { + root /var/www/acme; + default_type "text/plain"; + } + location / { + return 301 https://$host$request_uri; + } +} + +server { + listen 443 ssl; + listen [::]:443 ssl; + http2 on; + server_name __DOMAIN__ www.__DOMAIN__; + + ssl_certificate /etc/nginx/ssl/__DOMAIN__/fullchain.pem; + ssl_certificate_key /etc/nginx/ssl/__DOMAIN__/privkey.pem; + ssl_protocols TLSv1.2 TLSv1.3; + ssl_prefer_server_ciphers off; + + client_max_body_size 32m; + + location / { + proxy_pass http://127.0.0.1:__APP_PORT__; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_read_timeout 300s; + proxy_buffering off; + } +} diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index d4f1019..9dd6940 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -562,6 +562,10 @@ cmd_cutover() { lag=$(slot_lag_bytes) [ "$lag" -lt $((64 * 1024 * 1024)) ] || die "slot lag is $(numfmt --to=iec "$lag"); wait for it to drain" [ "$SOURCE_MODE" = docker ] || target_url >/dev/null + if [ "$SOURCE_MODE" = railway ]; then + tgt_exec "test -s $CERT_DIR/fullchain.pem && test -e /etc/nginx/sites-enabled/$APP_DOMAIN && docker image inspect nichedb:latest >/dev/null" || + die "app not staged on the box: run app-deploy, app-cert, app-vhost first" + fi confirm "Cut over now? Writes fail for a few minutes; reads keep working from the old DB." log "1/6 Source read-only, sessions closed" @@ -587,15 +591,23 @@ cmd_cutover() { tgt_q "drop subscription $SUB" if [ "$SOURCE_MODE" = railway ]; then - log "5/6 Pointing the app at the new database" + log "5/6 Railway app -> the new database, web only (the worker moves here)" local old old=$(rw_vars "$RW_APP_SERVICE" true | jq -r '.DATABASE_URL') [ "${OLD_DATABASE_URL:-}" ] || save_state OLD_DATABASE_URL "$old" + rw_set_var "$RW_APP_SERVICE" ROLES web rw_set_var "$RW_APP_SERVICE" DATABASE_URL "$(target_url)" - note "DATABASE_URL updated (was $old); Railway is deploying" + note "DATABASE_URL updated (was $old), ROLES=web; Railway is deploying" wait_for_deploy "$RW_APP_SERVICE" - log "6/6 Smoke test" - curl -fsS --max-time 20 "$APP_HEALTH_URL" && echo >&2 || warn "$APP_HEALTH_URL did not answer ok" + curl -fsS --max-time 20 "$APP_HEALTH_URL" >/dev/null && note "Railway app healthy on the new database" || warn "$APP_HEALTH_URL did not answer ok" + + log "6/6 The app on this box, then DNS" + cmd_app_deploy --start + local via + via=$(curl -sk --max-time 20 --resolve "$APP_DOMAIN:443:$TARGET_IP" "https://$APP_DOMAIN/healthz" || true) + [ "$via" = ok ] || die "https://$APP_DOMAIN via $TARGET_IP answered '$via', not ok; DNS NOT moved (Railway still serves it from the new DB)" + cmd_app_dns "$TARGET_IP" + note "DNS moved; Railway keeps answering old resolvers for the TTL, from the same database" note "app connections on the target: $(tgt_q "select count(*) from pg_stat_activity where usename = 'postgres' and client_addr is not null and client_addr <> '127.0.0.1'")" else note "5/6 and 6/6 skipped (rehearsal: no app to repoint)" @@ -624,7 +636,13 @@ cmd_rollback() { confirm "Roll back to the Railway database?" if [ "$SOURCE_MODE" = railway ]; then rw_set_var "$RW_APP_SERVICE" DATABASE_URL "${OLD_DATABASE_URL:-\${{Postgres.DATABASE_URL\}}}" - note "DATABASE_URL restored" + rw_set_var "$RW_APP_SERVICE" ROLES web,worker + note "DATABASE_URL restored, ROLES=web,worker" + tgt_exec "cd $APP_DIR 2>/dev/null && docker compose -f docker-compose.app.yml --env-file deploy.env stop app" 2>/dev/null || true + note "app on the box stopped" + if [ -n "${APP_DNS_FLIPPED:-}" ]; then + warn "DNS was moved to this box at $APP_DNS_FLIPPED; restore nichedb.dev at Porkbun by hand from $STATE_DIR/$APP_DOMAIN-dns.bak-*.json (ALIAS apex -> Railway, www CNAME)" + fi fi src_q "alter database $SOURCE_DB reset default_transaction_read_only" src_q "select count(pg_terminate_backend(pid)) from pg_stat_activity where datname = '$SOURCE_DB' and backend_type = 'client backend' and pid <> pg_backend_pid()" >/dev/null @@ -648,6 +666,97 @@ cmd_cleanup() { note "The Railway Postgres service itself is left in place (read-only). Delete it by hand once you no longer want the fallback." } +# ------------------------------------------------- the app, on the same box +# nichedb.dev itself moves with its database: the app runs from Docker under +# ~/www/nichedb.dev like every other site on a fleet box, nginx proxies to it, +# and at cutover the domain's A records move at Porkbun. Nothing new is named. +APP_DOMAIN=${APP_DOMAIN:-nichedb.dev} +APP_DIR=${APP_DIR:-/home/anthony/www/$APP_DOMAIN} +APP_PORT=${APP_PORT:-3010} +APP_REPO=${APP_REPO:-https://github.com/profullstack/niche-db.git} +APP_BRANCH=${APP_BRANCH:-main} +CERT_DIR=/etc/nginx/ssl/$APP_DOMAIN + +porkbun_keys() { # exports PORKBUN_API_KEY / PORKBUN_SECRET_API_KEY from cli-tools' store + local f=$HOME/.config/cli-tools/credentials.json + : "${PORKBUN_API_KEY:=$(jq -r '.porkbun.value // .porkbun // .PORKBUN_API_KEY // empty' "$f" 2>/dev/null)}" + : "${PORKBUN_SECRET_API_KEY:=$(jq -r '.porkbun_secret.value // .porkbun_secret // .PORKBUN_SECRET_API_KEY // empty' "$f" 2>/dev/null)}" + [ -n "$PORKBUN_API_KEY" ] && [ -n "$PORKBUN_SECRET_API_KEY" ] || die "no Porkbun keys: run 'cli-tools config pull' or export PORKBUN_API_KEY/PORKBUN_SECRET_API_KEY" + export PORKBUN_API_KEY PORKBUN_SECRET_API_KEY +} + +app_env_from_railway() { # the app's real environment is Railway's, rendered + rw_vars "$RW_APP_SERVICE" | jq -r 'to_entries[] | select(.key | test("^RAILWAY_") | not) | "\(.key)=\(.value)"' | + grep -Ev '^(DATABASE_URL|REDIS_URL|PORT|NODE_ENV)=' + printf 'DATABASE_URL=%s\n' "$(target_url | sed "s|@${DB_DOMAIN}:|@${APP_DB_HOST:-$DB_DOMAIN}:|")" +} + +cmd_app_deploy() { # app-deploy [--start] clone/pull + build; starts only with --start + local start=0 + [ "${1:-}" = --start ] && start=1 + log "App: $APP_REPO@$APP_BRANCH -> $TARGET_SSH:$APP_DIR" + tgt_exec "install -d -m 2755 -o anthony -g anthony $APP_DIR" + tgt_exec "if [ -d $APP_DIR/app/.git ]; then cd $APP_DIR/app && git fetch -q origin && git checkout -q $APP_BRANCH && git reset -q --hard origin/$APP_BRANCH; else git clone -q -b $APP_BRANCH $APP_REPO $APP_DIR/app; fi; cd $APP_DIR/app && git log --oneline -1" + local work + work=$(mktemp -d) + app_env_from_railway > "$work/app.env" + note "$(wc -l < "$work/app.env") variables from Railway, DATABASE_URL -> this box" + tgt_copy "$work/app.env" "$APP_DIR/app.env" + tgt_copy "$HERE/app/docker-compose.app.yml" "$APP_DIR/docker-compose.app.yml" + printf 'APP_PORT=%s\n' "$APP_PORT" > "$work/deploy.env" + tgt_copy "$work/deploy.env" "$APP_DIR/deploy.env" + rm -rf "$work" + tgt_exec "chmod 600 $APP_DIR/app.env; cd $APP_DIR && docker compose -f docker-compose.app.yml --env-file deploy.env build -q app" + note "image built" + if [ "$start" = 1 ]; then + tgt_exec "cd $APP_DIR && docker compose -f docker-compose.app.yml --env-file deploy.env up -d --wait" + local ok=0 + for _ in $(seq 1 30); do + [ "$(tgt_exec "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$APP_PORT/healthz")" = 200 ] && { ok=1; break; } + sleep 4 + done + [ "$ok" = 1 ] || { tgt_exec "cd $APP_DIR && docker compose -f docker-compose.app.yml logs --tail 40 app" >&2; die "app did not answer on 127.0.0.1:$APP_PORT/healthz"; } + note "app healthy on 127.0.0.1:$APP_PORT" + else + note "not started (it would write to the target before the cutover); 'app-deploy --start' or 'cutover' starts it" + fi +} + +cmd_app_cert() { # DNS-01 through Porkbun, so the cert exists BEFORE the domain moves + porkbun_keys + log "Certificate for $APP_DOMAIN + www via acme.sh (Porkbun DNS-01)" + tgt_exec "test -x /root/.acme.sh/acme.sh || curl -fsSL https://get.acme.sh | env HOME=/root sh -s email=anthony@profullstack.com >/dev/null" + tgt_exec "install -d -m 0750 $CERT_DIR && env HOME=/root PORKBUN_API_KEY='$PORKBUN_API_KEY' PORKBUN_SECRET_API_KEY='$PORKBUN_SECRET_API_KEY' /root/.acme.sh/acme.sh --issue --server letsencrypt --dns dns_porkbun -d $APP_DOMAIN -d www.$APP_DOMAIN --force >/tmp/acme-$APP_DOMAIN.log 2>&1 || { tail -n 20 /tmp/acme-$APP_DOMAIN.log; exit 1; }" + tgt_exec "env HOME=/root /root/.acme.sh/acme.sh --install-cert -d $APP_DOMAIN --fullchain-file $CERT_DIR/fullchain.pem --key-file $CERT_DIR/privkey.pem --reloadcmd 'systemctl reload nginx' >/dev/null" + note "$(tgt_exec "openssl x509 -in $CERT_DIR/fullchain.pem -noout -subject -enddate | tr '\n' ' '")" +} + +cmd_app_vhost() { + log "nginx vhost for $APP_DOMAIN -> 127.0.0.1:$APP_PORT" + tgt_exec "test -s $CERT_DIR/fullchain.pem" || die "no cert at $CERT_DIR: run 'nichedb-db app-cert' first" + local work + work=$(mktemp -d) + sed -e "s/__DOMAIN__/$APP_DOMAIN/g" -e "s/__APP_PORT__/$APP_PORT/g" "$HERE/app/nginx-vhost.conf" > "$work/vhost" + tgt_copy "$work/vhost" "/tmp/nichedb-vhost.conf" + rm -rf "$work" + tgt_exec "if [ -e /etc/nginx/sites-enabled/$APP_DOMAIN ]; then n=1; while [ -e /etc/nginx/sites-enabled/$APP_DOMAIN.bak-\$(printf %03d \$n) ]; do n=\$((n+1)); done; cp -a /etc/nginx/sites-enabled/$APP_DOMAIN /etc/nginx/sites-enabled/$APP_DOMAIN.bak-\$(printf %03d \$n); fi; install -m 0644 /tmp/nichedb-vhost.conf /etc/nginx/sites-available/$APP_DOMAIN && ln -sf /etc/nginx/sites-available/$APP_DOMAIN /etc/nginx/sites-enabled/$APP_DOMAIN && nginx -t 2>&1 | tail -n 1 && systemctl reload nginx" + note "vhost live; test: curl -sk --resolve $APP_DOMAIN:443:$TARGET_IP https://$APP_DOMAIN/healthz" +} + +cmd_app_dns() { # the flip: nichedb.dev + www -> this box, at Porkbun + local ip=${1:-${TARGET_IP:-}} + [ -n "$ip" ] || die "usage: nichedb-db app-dns " + log "Porkbun: $APP_DOMAIN and www -> $ip" + # The apex is an ALIAS to Railway today and www a CNAME; an A record cannot + # coexist with either, so they go first. Backed up as JSON beside the state. + porkbun ls "$APP_DOMAIN" --json > "$STATE_DIR/$APP_DOMAIN-dns.bak-$(date -u +%Y%m%dT%H%M%SZ).json" + porkbun rm "$APP_DOMAIN" @ --type ALIAS --yes 2>/dev/null || true + porkbun rm "$APP_DOMAIN" www --type CNAME --yes 2>/dev/null || true + porkbun set "$APP_DOMAIN" @ A "$ip" --ttl 600 + porkbun set "$APP_DOMAIN" www A "$ip" --ttl 600 + save_state APP_DNS_FLIPPED "$(date -u +%FT%TZ)" +} + cmd_apply_migration() { # apply-migration packages/db/migrations/00xx_name.sql local f=${1:?usage: nichedb-db apply-migration } log "Applying $(basename "$f") to the target (schema only; its schema_migrations row arrives through replication)" @@ -673,7 +782,11 @@ In order: status copy progress, slot lag, errors indexes build indexes/unique/foreign keys once the copy is done verify schema diff + row counts (FULL=1 counts items too) - cutover source read-only, drain, sequences, switch DATABASE_URL + app-deploy clone + build the app under ~/www/nichedb.dev (not started) + app-cert cert for nichedb.dev + www over Porkbun DNS-01 (before DNS moves) + app-vhost nginx vhost -> 127.0.0.1:3010 + cutover source read-only, drain, sequences; then Railway app -> + new DB (web only), start the app here, move DNS at Porkbun rollback point the app back at Railway and make it writable cleanup drop publication/slot and the TCP proxy (after you're sure) @@ -702,6 +815,10 @@ main() { case "$cmd" in provision) cmd_provision "$@" ;; prepare-db) cmd_prepare_db "$@" ;; + app-deploy) cmd_app_deploy "$@" ;; + app-cert) cmd_app_cert ;; + app-vhost) cmd_app_vhost ;; + app-dns) cmd_app_dns "$@" ;; fetch-connection) cmd_fetch_connection ;; vault) cmd_vault ;; dns) cmd_dns "$@" ;; From 5961f1f830319268ce4e4d8137744e2687dbc0b9 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 19:49:14 +0000 Subject: [PATCH 06/12] ops: indexes re-reads the source's post-data at build time Merges keep landing on main while the copy runs, and the app now builds some indexes after boot; the index step must carry whatever exists then. Co-Authored-By: Claude Fable 5.1 --- ops/selfhost-supabase/nichedb-db | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index 9dd6940..997c847 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -480,10 +480,17 @@ cmd_status() { cmd_indexes() { local d=$STATE_DIR/schema - [ -s "$d/rest.sql" ] || die "no $d/rest.sql: run 'nichedb-db schema' first" + [ -s "$d/pk.sql" ] || die "no $d/pk.sql: run 'nichedb-db schema' first" if ! all_ready && [ "${FORCE:-0}" != 1 ]; then die "the initial copy is not finished ($(sync_states | tr '\n' ' ')); FORCE=1 to build now" fi + # Re-read post-data NOW: deploys since `schema` may have added indexes on the + # source (the app builds some after boot), and those must exist here too. + src_exec pg_dump -U postgres -d "$SOURCE_DB" --schema-only --no-owner --no-privileges --section=post-data > "$d/post.sql" + python3 "$HERE/lib/split_post_data.py" "$d/post.sql" "$d/pk.now.sql" "$d/rest.sql" >&2 + if ! diff -q "$d/pk.sql" "$d/pk.now.sql" >/dev/null; then + warn "primary keys changed on the source since 'schema'; a new table needs the subscription refreshed" + fi log "Building indexes, unique and foreign keys on the target (detached; replication keeps streaming)" local mwm work mwm=$(tgt_q "show maintenance_work_mem") From 04445d325f85fa266dfdf2230bf32a22e73fda5b Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 20:11:25 +0000 Subject: [PATCH 07/12] ops: README: why subscribe hangs (snapshot waits on stuck ClientWrite backends) Co-Authored-By: Claude Fable 5.1 --- ops/selfhost-supabase/README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/ops/selfhost-supabase/README.md b/ops/selfhost-supabase/README.md index 9de36d3..c50baec 100644 --- a/ops/selfhost-supabase/README.md +++ b/ops/selfhost-supabase/README.md @@ -123,6 +123,18 @@ written to the new server after the cutover is not copied back. State and secrets live in `~/.local/state/nichedb-selfhost/` (mode 0700). +### `subscribe` hangs + +`create subscription` builds the slot on Railway, and the slot needs a +consistent snapshot: every transaction open on the source at that moment has +to finish first. `pg_stat_replication` shows the sender in `startup` until +then. On 2026-09-24 the blockers were read queries 10 to 34 minutes old, +all waiting in `ClientWrite` on clients that had stopped reading; a backend +blocked in a socket write cannot even honour `statement_timeout`. Terminate +those (`pg_terminate_backend` where `wait_event = 'ClientWrite'` and +`xact_start` is older than a few minutes) and the sender moves to +`streaming` within seconds. Long index builds hold it back the same way. + ### A migration landed mid-copy The subscription disables itself on the first apply error, and `status` shows From f43e538149813c1b689607027442deda98c4e6d6 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 20:33:41 +0000 Subject: [PATCH 08/12] ops: cutover never points Railway's app at the box dev2's 5432 is allowlisted and Railway egress is not fixed. Railway keeps serving its read-only copy for the DNS TTL instead. Co-Authored-By: Claude Fable 5.1 --- ops/selfhost-supabase/nichedb-db | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index 997c847..9a95231 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -598,15 +598,14 @@ cmd_cutover() { tgt_q "drop subscription $SUB" if [ "$SOURCE_MODE" = railway ]; then - log "5/6 Railway app -> the new database, web only (the worker moves here)" - local old - old=$(rw_vars "$RW_APP_SERVICE" true | jq -r '.DATABASE_URL') - [ "${OLD_DATABASE_URL:-}" ] || save_state OLD_DATABASE_URL "$old" - rw_set_var "$RW_APP_SERVICE" ROLES web - rw_set_var "$RW_APP_SERVICE" DATABASE_URL "$(target_url)" - note "DATABASE_URL updated (was $old), ROLES=web; Railway is deploying" - wait_for_deploy "$RW_APP_SERVICE" - curl -fsS --max-time 20 "$APP_HEALTH_URL" >/dev/null && note "Railway app healthy on the new database" || warn "$APP_HEALTH_URL did not answer ok" + log "5/6 Railway app stays on its read-only copy for the DNS TTL, web only" + # Not repointed at the new database: dev2's 5432 is allowlisted and + # Railway's egress addresses are not fixed. Stale resolvers get reads + # from Railway for up to ten minutes; writes there fail until they move. + if [ "$(rw_vars "$RW_APP_SERVICE" | jq -r '.ROLES')" != web ]; then + rw_set_var "$RW_APP_SERVICE" ROLES web + wait_for_deploy "$RW_APP_SERVICE" + fi log "6/6 The app on this box, then DNS" cmd_app_deploy --start @@ -642,9 +641,8 @@ cmd_rollback() { warn "Rollback points the app back at Railway. Writes made on the new server since the cutover are NOT copied back." confirm "Roll back to the Railway database?" if [ "$SOURCE_MODE" = railway ]; then - rw_set_var "$RW_APP_SERVICE" DATABASE_URL "${OLD_DATABASE_URL:-\${{Postgres.DATABASE_URL\}}}" rw_set_var "$RW_APP_SERVICE" ROLES web,worker - note "DATABASE_URL restored, ROLES=web,worker" + note "ROLES=web,worker on Railway again" tgt_exec "cd $APP_DIR 2>/dev/null && docker compose -f docker-compose.app.yml --env-file deploy.env stop app" 2>/dev/null || true note "app on the box stopped" if [ -n "${APP_DNS_FLIPPED:-}" ]; then @@ -792,8 +790,8 @@ In order: app-deploy clone + build the app under ~/www/nichedb.dev (not started) app-cert cert for nichedb.dev + www over Porkbun DNS-01 (before DNS moves) app-vhost nginx vhost -> 127.0.0.1:3010 - cutover source read-only, drain, sequences; then Railway app -> - new DB (web only), start the app here, move DNS at Porkbun + cutover source read-only, drain, sequences; start the app on the + box, move DNS at Porkbun (Railway serves reads for the TTL) rollback point the app back at Railway and make it writable cleanup drop publication/slot and the TCP proxy (after you're sure) From 48d8ffaabb227484f7880cc7fa3d05ff9793dd62 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 20:58:35 +0000 Subject: [PATCH 09/12] ops: primary-key drift check ignores pg_dump's restrict token Co-Authored-By: Claude Fable 5.1 --- ops/selfhost-supabase/nichedb-db | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index 9a95231..c0043a6 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -488,7 +488,8 @@ cmd_indexes() { # source (the app builds some after boot), and those must exist here too. src_exec pg_dump -U postgres -d "$SOURCE_DB" --schema-only --no-owner --no-privileges --section=post-data > "$d/post.sql" python3 "$HERE/lib/split_post_data.py" "$d/post.sql" "$d/pk.now.sql" "$d/rest.sql" >&2 - if ! diff -q "$d/pk.sql" "$d/pk.now.sql" >/dev/null; then + # pg_dump stamps each dump with a fresh \restrict token: compare without it. + if ! diff -q <(grep -v '^\\' "$d/pk.sql") <(grep -v '^\\' "$d/pk.now.sql") >/dev/null; then warn "primary keys changed on the source since 'schema'; a new table needs the subscription refreshed" fi log "Building indexes, unique and foreign keys on the target (detached; replication keeps streaming)" From 302621f7f4b8fda178b2c7515b8ef4c191f88222 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 21:00:04 +0000 Subject: [PATCH 10/12] ops: cutover never leaves a replication slot on the source Co-Authored-By: Claude Fable 5.1 --- ops/selfhost-supabase/nichedb-db | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index c0043a6..870401e 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -596,7 +596,19 @@ cmd_cutover() { cmd_verify || warn "verify reported differences; check before trusting the target" log "4/6 Dropping the subscription (also drops the slot on the source)" - tgt_q "drop subscription $SUB" + if ! tgt_q "drop subscription $SUB"; then + warn "drop subscription failed; detaching the slot and dropping locally" + tgt_q "alter subscription $SUB disable; alter subscription $SUB set (slot_name = none); drop subscription $SUB" + fi + # An orphaned slot would hold WAL on the source (capped at SLOT_KEEP_GB, but + # still): make sure nothing is left behind. + local left + left=$(src_q "select string_agg(slot_name, ',') from pg_replication_slots where not active") + if [ -n "$left" ]; then + src_q "select pg_drop_replication_slot(slot_name) from pg_replication_slots where not active" >/dev/null + note "dropped leftover slot(s) on the source: $left" + fi + note "slots left on the source: $(src_q "select coalesce(string_agg(slot_name||' active='||active, ', '), 'none') from pg_replication_slots")" if [ "$SOURCE_MODE" = railway ]; then log "5/6 Railway app stays on its read-only copy for the DNS TTL, web only" From 15a0a22de58f6359adfb5ba5242bd7cb695463d0 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 21:25:51 +0000 Subject: [PATCH 11/12] ops: cutover preflight counts valid indexes only A failed CREATE INDEX CONCURRENTLY leaves an invalid empty index on the source that pg_dump omits; it blocked the cutover with 163 vs 162. Co-Authored-By: Claude Fable 5.1 --- ops/selfhost-supabase/nichedb-db | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index 870401e..d599008 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -561,10 +561,13 @@ cmd_cutover() { log "Preflight" all_ready || die "not every table is in state r: $(sync_states | tr '\n' ' ')" [ "$(tgt_q "select subenabled from pg_subscription where subname = '$SUB'")" = t ] || die "subscription is disabled: see 'nichedb-db status'" - local si ti - si=$(src_q "select count(*) from pg_indexes where schemaname = 'public'") - ti=$(tgt_q "select count(*) from pg_indexes where schemaname = 'public'") - [ "$si" = "$ti" ] || die "indexes: source $si, target $ti. Run/finish 'nichedb-db indexes' first" + # Valid indexes only: a failed CREATE INDEX CONCURRENTLY leaves an invalid, + # empty index on the source that pg_dump omits and nothing should copy. + local si ti q + q="select count(*) from pg_index i join pg_class c on c.oid = i.indrelid join pg_namespace n on n.oid = c.relnamespace where n.nspname = 'public' and i.indisvalid" + si=$(src_q "$q") + ti=$(tgt_q "$q") + [ "$si" = "$ti" ] || die "valid indexes: source $si, target $ti. Run/finish 'nichedb-db indexes' first" cmd_verify >/dev/null 2>&1 || die "schemas differ: run 'nichedb-db verify' and reconcile first" local lag lag=$(slot_lag_bytes) From c11693c05ffc4139639024d8fccdd3ca29aac920 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Thu, 24 Sep 2026 21:26:36 +0000 Subject: [PATCH 12/12] ops: the app on the box always runs web,worker app.env is taken from Railway's variables, which are web-only while the copy runs; the box takes both roles at cutover. Co-Authored-By: Claude Fable 5.1 --- ops/selfhost-supabase/nichedb-db | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/ops/selfhost-supabase/nichedb-db b/ops/selfhost-supabase/nichedb-db index d599008..0e1d659 100755 --- a/ops/selfhost-supabase/nichedb-db +++ b/ops/selfhost-supabase/nichedb-db @@ -629,7 +629,7 @@ cmd_cutover() { via=$(curl -sk --max-time 20 --resolve "$APP_DOMAIN:443:$TARGET_IP" "https://$APP_DOMAIN/healthz" || true) [ "$via" = ok ] || die "https://$APP_DOMAIN via $TARGET_IP answered '$via', not ok; DNS NOT moved (Railway still serves it from the new DB)" cmd_app_dns "$TARGET_IP" - note "DNS moved; Railway keeps answering old resolvers for the TTL, from the same database" + note "DNS moved; Railway answers stale resolvers for the TTL from its read-only copy" note "app connections on the target: $(tgt_q "select count(*) from pg_stat_activity where usename = 'postgres' and client_addr is not null and client_addr <> '127.0.0.1'")" else note "5/6 and 6/6 skipped (rehearsal: no app to repoint)" @@ -708,8 +708,10 @@ porkbun_keys() { # exports PORKBUN_API_KEY / PORKBUN_SECRET_API_KEY from cli-too app_env_from_railway() { # the app's real environment is Railway's, rendered rw_vars "$RW_APP_SERVICE" | jq -r 'to_entries[] | select(.key | test("^RAILWAY_") | not) | "\(.key)=\(.value)"' | - grep -Ev '^(DATABASE_URL|REDIS_URL|PORT|NODE_ENV)=' + grep -Ev '^(DATABASE_URL|REDIS_URL|PORT|NODE_ENV|ROLES)=' printf 'DATABASE_URL=%s\n' "$(target_url | sed "s|@${DB_DOMAIN}:|@${APP_DB_HOST:-$DB_DOMAIN}:|")" + # Railway may be running web-only while the copy runs; the box takes both. + printf 'ROLES=web,worker\n' } cmd_app_deploy() { # app-deploy [--start] clone/pull + build; starts only with --start