Skip to content

deployment.md

Latest commit

 

History

History
746 lines (559 loc) · 56.9 KB

deployment.md

File metadata and controls

746 lines (559 loc) · 56.9 KB
Raw

Phaze Deployment Guide

Production Phaze runs as two compose files on two (or more) hosts:

  • Application server (docker-compose.yml): API/UI, controller worker, Postgres, Redis. No music/model/output file mounts. HTTPS via an internal CA. Redis requirepass + LAN binding.
  • File servers (docker-compose.agent.yml, one per host): three per-lane agent workers (worker-analyze, worker-meta, worker-io) and a watcher. Holds music/video files locally; reaches the app-server over HTTPS for every state change.

This guide walks through bringing up a fresh two-host deployment from a clean checkout, then covers the build pipeline, rollback, and monitoring.

Optionally, the control plane can offload analysis to cloud backends declared in a backends.toml — a cloud-compute agent and/or one-or-more Kueue clusters. Cloud offload is entirely additive: with no backends.toml mounted, the deployment is the local-only two-host topology below.

flowchart TB
    subgraph app["Application server (docker-compose.yml)"]
        api["api<br/>FastAPI + admin UI (TLS :8000)"]
        ctl["worker<br/>PHAZE_ROLE=control"]
        pg[("postgres :5432<br/>metadata + SAQ broker")]
        rd[("redis :6379<br/>cache / counters")]
        cfg{{"backends.toml<br/>(optional, mounted)"}}
        api --- pg
        api --- rd
        ctl --- pg
        ctl --- rd
        cfg -. loaded by .-> api
        cfg -. loaded by .-> ctl
    end

    subgraph fs["File server(s) (docker-compose.agent.yml, one per host)"]
        aw["worker-analyze/-meta/-io<br/>PHAZE_ROLE=agent (3 lanes)"]
        wt["watcher"]
        media[("/data/music (ro)")]
        aw --- media
        wt --- media
    end

    subgraph cloud["Cloud-compute agent (docker-compose.cloud-agent.yml)"]
        ca["worker (arm64, kind=compute)<br/>network_mode: host over Tailscale"]
    end

    subgraph kueue["N Kueue clusters (kind=kueue)"]
        k1["Cluster A<br/>LocalQueue"]
        s1[("S3 staging A")]
        k2["Cluster B<br/>LocalQueue"]
        s2[("S3 staging B")]
        k1 --- s1
        k2 --- s2
    end

    fs -->|"HTTPS :8000 API (DIST-04)"| api
    fs -->|"Postgres :5432 SAQ broker"| pg
    fs -->|"Redis :6379 cache"| rd
    cloud -->|"HTTPS :8000 API + Postgres :5432 broker"| api
    api -->|"submit/watch Jobs (kube API)"| k1
    api -->|"submit/watch Jobs (kube API)"| k2
    k1 -->|"callback HTTPS :8000 API"| api
    k2 -->|"callback HTTPS :8000 API"| api
Loading

Deployment Targets

The repo ships three deployment compose files plus a dev overlay:

File Host Services Notes
docker-compose.yml Application server api, worker (control role), postgres, redis Built locally from Dockerfile. No file mounts on api/worker except ./certs/ on api (DIST-01).
docker-compose.agent.yml File server (one per host) worker-analyze, worker-meta, worker-io (three per-lane agent-role workers, plus an off-by-default worker-drain profile service), watcher All services pull from GHCR via PHAZE_IMAGE_TAG (ghcr.io/simplicityguy/phaze). See agent-queue-lanes.md for the lane split.
docker-compose.cloud-agent.yml OCI A1 (cloud) worker (agent role, kind=compute) arm64 image, no media, named scratch. Cloud-burst compute agent over Tailscale. See cloud-burst.md.
docker-compose.dev.yml Application server (dev only) overlays api + worker Explicit opt-in only — included via just up-dev (-f docker-compose.yml -f docker-compose.dev.yml), NEVER auto-merged (phaze-476w: it was formerly docker-compose.override.yml, which docker compose auto-merged and silently hijacked the production just up). Mounts ./src for live reload, runs uvicorn --reload, sets PHAZE_DEBUG=true, and deliberately skips the cert-bootstrap entrypoint. just up (base compose only) is unaffected.

Application-server services (docker-compose.yml)

Service Image / build Command Ports Role
api build Dockerfile uv run python -m phaze.entrypoint ${API_PORT:-8000}:8000 FastAPI + admin UI behind TLS. Mounts ${CA_PATH:-./certs}:/certs:rw for the cert bootstrap.
worker build Dockerfile uv run saq phaze.tasks.controller.settings — Control-role SAQ worker (PHAZE_ROLE=control). Fileless; no volume mounts.
postgres postgres:18-alpine — ${POSTGRES_BIND_IP:-127.0.0.1}:5432:5432 Primary database. Loopback-only by default; set POSTGRES_BIND_IP to the app-server's private LAN IP in production so agents reach the SAQ broker (mirrors REDIS_BIND_IP). POSTGRES_PASSWORD is ${POSTGRES_PASSWORD:?} — compose parse fails if unset (phaze-rnh7). shm_size: "256m" (phaze-knwk) — Docker's 64 MB /dev/shm default starves Postgres's parallel dynamic-shared-memory allocations, which surfaces as failed parallel queries and index builds rather than as an obvious out-of-memory; justfile:21 mirrors the same value for the test container. Data on the pgdata named volume mounted at /var/lib/postgresql.
redis redis:8-alpine redis-server --requirepass ${REDIS_PASSWORD:?...} ${REDIS_BIND_IP:-127.0.0.1}:6379:6379 Cache / rate-limit / counters (no longer the SAQ broker — Postgres is, via PHAZE_QUEUE_URL). --requirepass fails fast at compose-parse time if REDIS_PASSWORD is unset.

api and worker are built from the same Dockerfile and differ only by their command: api runs the cert-bootstrap entrypoint then uvicorn; worker runs the controller SAQ worker with PHAZE_ROLE=control.

Bumping the Postgres tag

The pinned Postgres image lives in exactly three places, and all three must move together:

# Site Form
1 justfile:14 postgres_image := "postgres:18-alpine" (test container)
2 docker-compose.yml:84 image: postgres:18-alpine (production)
3 .github/workflows/tests.yml:48 image: postgres:18-alpine (CI service)

tests/agents/deployment/test_postgres_image_pin.py fails the build if they disagree, so a partial bump is caught in CI rather than in production. This is the phaze-tcqq outcome: the tag was previously hardcoded in 10 places — four of them echo strings, so a partial bump would happily print the old tag while running the new one. That was consolidated to these three sites plus the guard test, not to a single source of truth, because Compose and GitHub Actions cannot read a justfile variable.

File-server services (docker-compose.agent.yml)

Service Image / build Command Role
worker-analyze ghcr.io/simplicityguy/phaze:${PHAZE_IMAGE_TAG:-latest} uv run saq phaze.tasks.agent_worker.settings Agent-role SAQ worker (PHAZE_ROLE=agent) consuming the analyze lane (process_file; default concurrency 4). Heartbeats with lane=analyze (PHAZE_AGENT_HEARTBEAT=true — as do all three lane workers, phaze-30fo).
worker-meta ghcr.io/simplicityguy/phaze:${PHAZE_IMAGE_TAG:-latest} uv run saq phaze.tasks.agent_worker.settings Agent-role SAQ worker consuming the meta lane (extract_file_metadata/scan_directory/execute_approved_batch; default concurrency 2). Heartbeats with lane=meta.
worker-io ghcr.io/simplicityguy/phaze:${PHAZE_IMAGE_TAG:-latest} uv run saq phaze.tasks.agent_worker.settings Agent-role SAQ worker consuming the io lane (s3_upload/push_file; default concurrency 4). Heartbeats with lane=io.
worker-drain (profile drain, off by default) ghcr.io/simplicityguy/phaze:${PHAZE_IMAGE_TAG:-latest} uv run saq phaze.tasks.agent_worker.settings Transitional all-mode consumer of the legacy un-suffixed phaze-agent-<agent_id> queue during the lane-split migration. Start with docker compose -f docker-compose.agent.yml --profile drain up -d worker-drain. The only worker with PHAZE_AGENT_HEARTBEAT=false: it is unlaned, so an untagged beat would wipe the per-lane breakdown the three lane workers maintain.
watcher ghcr.io/simplicityguy/phaze:${PHAZE_IMAGE_TAG:-latest} uv run python -m phaze.agent_watcher Always-on directory watcher (PHAZE_ROLE=agent).

The three lane workers and watcher all mount the music library read-only via ${SCAN_PATH:?SCAN_PATH required}:/data/music:ro. There is no postgres or redis service here — agents reach the app-server's Redis (cache) and, as of the Phase-36 queue-backend migration, the app-server's Postgres broker directly over the LAN via PHAZE_QUEUE_URL (Postgres:5432). Application/file metadata is still reached only via the HTTP API — DIST-04 — and there is no DATABASE_URL on any agent service; the agent's only Postgres connection is the SAQ broker pool. See agent-queue-lanes.md for the full lane topology, concurrency knobs, and the legacy-queue drain runbook.

Cloud backends (backends.toml)

Cloud offload is optional and additive. The control plane discovers its execution backends from a backends.toml file mounted into the api + worker (control) services and loaded via PHAZE_BACKENDS_CONFIG_FILE (default /etc/phaze/backends.toml). If no file is mounted, the app resolves an implicit local-only registry and cloud offload is off — there is no env master toggle to set. (As of Phase 67, the old PHAZE_CLOUD_TARGET=local|a1|k8s selector and the flat kube_* / s3_* / compute_* fields are removed with no back-compat; the app-server compose comments document the mount that replaces them.)

backends.toml is a typed list of [[backends]] entries, each with a kind, a cost-tier rank (lower runs sooner), and a concurrency cap:

kind Purpose Requires
local On-prem file-server queues (the default). nothing beyond rank/cap
compute Cloud-compute agent (rsync/push) — e.g. an OCI Ampere A1 (arm64) over Tailscale for long sets that would time out on a file server. agent_ref + scratch_dir
kueue A Kubernetes Kueue cluster that runs one-shot Jobs. N clusters can be declared simultaneously, each with its own LocalQueue and S3 staging bucket. a nested [backends.kube] table (api_url / namespace / local_queue + _file-mounted kubeconfig/SA-token)

Per-backend S3 staging is declared in [[buckets]] entries bound to backends by id. Each bucket carries a scope — shared (referenceable by any number of kueue backends) or cluster-specific (the sharing-cardinality invariant D-09 fails fast if a cluster-specific bucket is referenced by more than one kueue backend). Bucket endpoint_url values carry a per-bucket http(s) SSRF guard.

Cloud on/off is therefore derived from the registry (cloud_enabled is true iff at least one non-local backend resolves), never set by an env flag.

Deep-dive runbooks

The current configuration surface for every cloud path is this one backends.toml. The two sibling runbooks below are the operational deep-dives (provisioning, transport, credentials, deploy ordering); both are being updated to the registry model, so treat backends.toml as authoritative where they differ:

  • cloud-burst.md — the kind=compute agent: the docker-compose.cloud-agent.yml walkthrough, the OCI A1 provisioning runbook (OpenTofu spec, Tailscale grants ACL, least-privilege phaze_broker Postgres role), and the smoke test. On the file-server/agent host, bring the compute agent up/down with just cloud-agent-up / just cloud-agent-down (standalone docker-compose.cloud-agent.yml).
  • k8s-burst.md — the kind=kueue clusters: the cluster-admin Kueue objects (ResourceFlavor / ClusterQueue / LocalQueue), the namespaced RBAC Role phaze's ServiceAccount needs to submit/watch Jobs, the _FILE-mounted kubeconfig/SA-token + S3-credential Secret, the S3 staging bucket + lifecycle rule, and the transport (Tailscale or WireGuard) to the kube API and S3 endpoint.

Disabling cloud offload

Because on/off is derived from the registry, disabling cloud offload means removing the non-local entries from backends.toml (or unmounting the file entirely, which resolves the implicit local-only registry), then restarting the control services:

docker compose up -d --force-recreate worker api

The registry is read from the import-time settings singleton, so the change takes effect only after a restart — like every other knob. After the restart, every file routes to the local file-server queues exactly as it did before any cloud backend existed; in-flight work drains rather than aborting (files already dispatched finish on their current backend). You do not need to tear down the kube API, LocalQueues, S3 buckets, mounted Secrets, or the OCI A1 host to disable — they can all be left in place, inert. Re-enabling later is just re-adding the [[backends]]/[[buckets]] entries plus a restart, with no re-provisioning.

Controller vs Agent roles

Phaze selects its settings class at process boot from the PHAZE_ROLE env var (default control), via phaze.config.get_settings():

  • PHAZE_ROLE=control → ControlSettings (LLM proposal generation, Discogs matching, fileless tasks). Used by the app-server api + worker.
  • PHAZE_ROLE=agent → AgentSettings (HTTP client to the app-server, file-bound tasks). Used by the file-server's three lane workers (worker-analyze/worker-meta/worker-io) + watcher. The validators in AgentSettings raise at construction time if PHAZE_AGENT_API_URL, PHAZE_AGENT_TOKEN, or PHAZE_AGENT_SCAN_ROOTS is missing — agents fail fast with a clear error rather than emitting runtime 401s.

The api container does not start uvicorn directly. It runs uv run python -m phaze.entrypoint, which:

  1. Runs phaze.cert_bootstrap.ensure_certs_present(/certs, ...) to generate (or no-op past) the internal CA + leaf cert before uvicorn binds.
  2. os.execvp-replaces the process with uvicorn phaze.main:app --ssl-keyfile /certs/phaze-server.key --ssl-certfile /certs/phaze-server.crt, so signals and PID 1 propagate cleanly.

The entrypoint reads three env vars, all with safe defaults so a plain docker compose up works in dev: PHAZE_CERTS_DIR (default /certs), PHAZE_API_HOST (default localhost, baked into the leaf CN), and PHAZE_API_TLS_SANS (default localhost,127.0.0.1,api). The bootstrap is idempotent — restarts against a populated /certs/ skip regeneration.

Internal CA / mTLS bootstrap

phaze.cert_bootstrap generates a self-signed ECDSA P-256 CA (10-year validity) and a CA-signed leaf cert (2-year validity) into /certs/ on the app-server's first start:

File Mode Distribution
phaze-ca.crt 0644 Public. Copied to every file server; agents point PHAZE_AGENT_CA_FILE at it.
phaze-ca.key 0600 Private CA signing key. Never leaves the app-server host.
phaze-server.crt 0644 Leaf cert presented by uvicorn over TLS.
phaze-server.key 0600 Leaf private key.

On actual generation (not the idempotent no-op path) a loud banner is emitted via both print() (interactive docker compose up) and logger.warning() (docker compose logs api). The banner references only the public CA path. Agents trust the app-server by validating its TLS chain against the operator-distributed phaze-ca.crt; the agent's outbound bearer token (PHAZE_AGENT_TOKEN) is what authenticates the agent to the app-server.

The env vars that gate this bootstrap on the agent side are PHAZE_AGENT_CA_FILE (default /certs/phaze-ca.crt), PHAZE_AGENT_API_URL (must be https:// when PHAZE_AGENT_ENV=production), and PHAZE_AGENT_TOKEN. See docs/configuration.md for the full env-var reference.

Prerequisites

  • Docker Engine 20.10+ and docker compose v2.x on both hosts
  • just installed on both hosts (or run docker compose directly)
  • Both hosts on the same private LAN; no firewall blocking ports 6379 (Redis cache), 5432 (Postgres SAQ broker — agents reach it directly as of Phase 36), or 8000 (API) between them
  • Postgres + Redis are NOT directly exposed to the public internet (LAN-only; the agent→Postgres:5432 edge is private-LAN scoped)
  • On the app-server host: ./certs/ (materialized by git clone via a committed .gitkeep, then cert-populated on first start; must be owned by uid 1000 — phaze-he8m), .env
  • On each file-server host: ./certs/ (CA only, scp'd from app-server), ./models/ (materialized by git clone, weights auto-download on first agent start; both owned by uid 1000 — phaze-he8m), .env

Step 1 — Bring up the application server

On the app-server host:

git clone https://github.com/simplicityguy/phaze.git
cd phaze
cp .env.example .env
# Edit .env: set POSTGRES_PASSWORD to a strong unique value — compose parse FAILS if unset
# Edit .env: set POSTGRES_BIND_IP to the app-server's private LAN IP (e.g., 192.168.1.10) so
#            agents can reach the SAQ broker; at the 127.0.0.1 default they cannot
# Edit .env: set REDIS_PASSWORD to a strong unique value (>= 32 chars)
# Edit .env: set REDIS_BIND_IP to the app-server's private LAN IP (e.g., 192.168.1.10)
# Edit .env: set PHAZE_QUEUE_URL (raw libpq postgresql:// DSN) — the SAQ Postgres broker.
#            Defaults to the in-compose `postgres` service; treat as a secret (PHAZE_QUEUE_URL_FILE).
mkdir -p certs   # phaze-he8m: own ./certs as the operator (uid 1000) BEFORE `up`
just up

just up runs docker compose -f docker-compose.yml up -d (base compose only, so the dev overlay is never auto-merged — phaze-476w; it also runs mkdir -p certs for you). The api container bind-mounts ./certs:/certs:rw and runs as the non-root phaze user (uid 1000). The ./certs dir must exist and be owned by uid 1000 before up (phaze-he8m): on a rootful Linux docker engine a missing bind-mount source is auto-created by the daemon as root:root, and the uid-1000 cert bootstrap then dies with PermissionError writing /certs/phaze-ca.crt before uvicorn binds — an opaque crash-loop. A fresh git clone already materializes ./certs/ (it ships a committed .gitkeep) owned by the cloning operator, and just up re-creates it defensively; the explicit mkdir -p certs above is belt-and-suspenders. If you hit the crash anyway, the entrypoint now prints the exact fix: sudo chown -R 1000:1000 certs.

On first start, the api container's entrypoint generates the internal CA + leaf cert into ./certs/. Watch the logs:

docker compose logs -f api

You will see a multi-line banner:

==============================================================
GENERATED NEW PHAZE INTERNAL CA at /certs/phaze-ca.crt
COPY THIS FILE TO EVERY FILE SERVER and point each agent's
PHAZE_AGENT_CA_FILE env var at it. EXISTING AGENTS WILL FAIL
TO CONNECT UNTIL THEY HAVE THIS NEW CA.
==============================================================

After the banner, uvicorn binds port 8000 with TLS.

Verify: curl --cacert ./certs/phaze-ca.crt https://localhost:8000/docs returns the OpenAPI UI, and curl --cacert ./certs/phaze-ca.crt https://localhost:8000/health returns {"status":"ok"} once Postgres is reachable.

Step 2 — Copy the CA cert to each file server

The CA private key (./certs/phaze-ca.key) stays on the app-server host (mode 0600). Only the public CA cert (./certs/phaze-ca.crt, mode 0644) is distributed to file-server hosts.

From the app-server host, for each file server:

scp ./certs/phaze-ca.crt operator@fileserver-east:/home/operator/phaze/certs/phaze-ca.crt

Or use rsync, ansible, or any one-time file transfer mechanism. The operator-distributed CA is a public cert; non-secret.

Step 3 — Register an agent and mint a token

On the app-server host, run the bundled phaze agents add management CLI (it ships with the application image; run it inside the API container, or any environment that has the phaze package installed and DATABASE_URL configured):

docker compose exec api uv run phaze agents add \
    --id fileserver-east \
    --name "File Server East" \
    --scan-roots /data/music,/data/concerts

uv run is required, not optional (phaze-u5k0d). The api/worker containers' CMD is uv run ... — the project resolves through uv's environment rather than an installed system Python — so a bare docker compose exec api phaze agents add ... fails with executable file not found in $PATH (verified against a deployed container, image 2026.8.4, 2026-08-17; same defect as docs/runbook.md's "Stranded active SAQ jobs" section). The image also now puts the venv's bin/ on PATH, so the bare form works too if you type it from memory — uv run is kept here anyway because it's correct regardless of that PATH env change. Outside the container (a host or CI environment with phaze pip/uv-installed directly and DATABASE_URL reachable), drop the docker compose exec api uv run prefix and just run phaze agents add ....

The CLI mints a strong bearer token, stores only its sha256 hash in the agents table, and prints two things you need:

  • The cleartext token — printed exactly once. Save it now — it is NOT recoverable from the database (only the hash is stored). This is the value you put in PHAZE_AGENT_TOKEN on the file-server side (Step 4).
  • The derived queue name phaze-agent-fileserver-east — put this in PHAZE_AGENT_QUEUE on the file-server side (Step 4). The queue name is always phaze-agent-<agent_id>; the agent worker asserts this at startup and exits if it disagrees, so copy the value the CLI printed.

The --id must match ^[a-z0-9]+(-[a-z0-9]+)*$ (the agents.id charset constraint); the CLI rejects an invalid id (or a non-absolute scan root) with a non-zero exit before touching the database, and reports a friendly error if the id already exists. --name defaults to the titleized id when omitted.

A sentinel LIVE ScanBatch row is auto-created the first time the agent posts a file.

Under the hood / SQL fallback (no CLI access)

phaze agents add is equivalent to inserting the row by hand. If you only have a psql session, generate a token and insert its sha256 hash (the full wire string, prefix included) yourself:

python -c "import secrets; print('phaze_agent_' + secrets.token_urlsafe(32))"
INSERT INTO agents (id, name, token_hash, scan_roots, created_at)
VALUES (
    'fileserver-east',
    'File Server East',
    -- token_hash is sha256() of the chosen plaintext token (prefix included)
    encode(sha256('phaze_agent_REPLACE_WITH_RANDOM_32_URLSAFE'::bytea), 'hex'),
    '["/data/music", "/data/concerts"]'::jsonb,
    now()
);

Same rules apply: save the plaintext token (only the hash is stored), and the agent's queue name is phaze-agent-fileserver-east.

Step 4 — Populate the file-server .env

On the file-server host, get the compose file and the .env template. All agent images are pulled from GHCR, so the checkout is only needed for docker-compose.agent.yml + .env.example.agent — not to build anything locally for a normal deployment:

git clone https://github.com/simplicityguy/phaze.git
cd phaze
cp .env.example.agent .env
mkdir -p models certs   # phaze-he8m: own ./models (weights auto-download) and ./certs as uid 1000

The mkdir -p models certs (also run for you by just up-agent) is required for the same reason as Step 1 (phaze-he8m): the uid-1000 worker auto-downloads essentia weights into ./models:/models:rw and reads the CA from ./certs, so both bind-mount sources must exist owned by uid 1000 before up — otherwise rootful dockerd creates them root:root and the download fails with EACCES. A git clone already materializes both via committed .gitkeep files.

Edit .env to set the required variables. The agent stack uses ${VAR:?msg} interpolation on SCAN_PATH, so docker compose fails fast at parse time if it is unset:

  • PHAZE_AGENT_API_URL=https://<app-server-lan-ip>:8000
  • PHAZE_REDIS_URL=redis://<app-server-lan-ip>:6379/0 + REDIS_PASSWORD=<the app-server's REDIS_PASSWORD> (cache only — no longer the SAQ broker). Prefer this split form over hand-typing redis://default:<REDIS_PASSWORD>@<app-server-lan-ip>:6379/0: config.py's _apply_redis_password percent-encodes REDIS_PASSWORD into the URL for you (phaze-1g89i), so a strong password containing /, #, ?, or % can't corrupt the DSN the way pasting it directly into the URL would.
  • PHAZE_QUEUE_URL=postgresql://<user>:<password>@<app-server-lan-ip>:5432/phaze — the SAQ Postgres broker DSN (raw libpq form, NOT postgresql+asyncpg://). The agent now opens a psycopg3 pool to the app-server Postgres, so its host must be reachable on 5432 (new firewall edge relaxing D-25). Treat it as a secret — prefer PHAZE_QUEUE_URL_FILE=/run/secrets/phaze_queue_url.
  • PHAZE_AGENT_ID=fileserver-east
  • PHAZE_AGENT_TOKEN=<the plaintext token from Step 3>
  • PHAZE_AGENT_QUEUE=phaze-agent-fileserver-east — by convention this MUST equal phaze-agent-<agent_id> (the value phaze agents add printed in Step 3). There is no queue column on the agents table; the worker derives the expected name from the token's agent_id and exits non-zero on mismatch.
  • PHAZE_AGENT_CA_FILE=/certs/phaze-ca.crt
  • PHAZE_AGENT_ENV=production
  • SCAN_PATH=/path/to/your/music/library
  • MODELS_PATH=./models — a HOST path only; it picks the bind-mount source. Every worker + watcher service pins its own container-side MODELS_PATH=/models (phaze-bvkah), so this value never changes where the container looks for models — only which host directory backs /models.
  • CA_PATH=./certs
  • PHAZE_AGENT_SCAN_ROOTS=/data/music,/data/concerts
  • PHAZE_IMAGE_TAG=2026.8.4 (or latest for first-time setup)

See docs/configuration.md for the complete env-var reference and defaults.

Step 5 — Bring up the agent stack

On the file-server host:

just up-agent

just up-agent runs docker compose -f docker-compose.agent.yml up -d, which brings up all three lane workers (worker-analyze, worker-meta, worker-io) plus watcher (the off-by-default worker-drain profile service is not included). On first start, the lane workers call /api/internal/agent/whoami to verify their token and validate ~3.1 GB of essentia weights (68 model/config files) in the shared ./models/ volume. A file lock lets one lane perform the download while its siblings wait and then revalidate; a fresh multi-GB transfer can take many minutes. The watcher comes up in parallel.

Watch any lane's logs. Every active lane worker publishes its own heartbeat; worker-analyze is usually the most useful stream when checking audio-analysis startup:

docker compose -f docker-compose.agent.yml logs -f worker-analyze

You should see (per lane worker; lane= varies by service):

  • phaze.tasks.agent_worker startup role=agent api=https://... auth_id_prefix=phaze_agent_a1b2... queue=phaze-agent-fileserver-east-analyze lane=analyze
  • validating model weights -- essentia weights at /models (~3.1 GB across 68 files); ...
  • another worker holds the model download lock -- waiting for it to finish before re-validating (the sibling lanes on a fresh start)
  • models validated
  • phaze.tasks.agent_worker startup complete agent_id=fileserver-east queue=phaze-agent-fileserver-east-analyze lane=analyze

After each lane completes startup, its heartbeat — an asyncio background task in each of the three lane workers (every 30s, each beat tagged with its own lane; see agent-queue-lanes.md) — starts firing against POST /api/internal/agent/heartbeat.

Run both stacks on one host (dev convenience): just up-all runs docker compose -f docker-compose.yml -f docker-compose.agent.yml up -d. This is for development only — production keeps the app-server and file-server stacks on separate hosts to preserve filesystem isolation (DIST-01).

Step 6 — Verify on the admin page

From any browser on the LAN (or via SSH tunnel from your laptop):

# Trust the CA in your local browser, or use curl:
curl --cacert ./certs/phaze-ca.crt https://<app-server-lan-ip>:8000/admin/agents

The /admin/agents page renders an agent table and self-refreshes via an HTMX poll every 5 seconds. Each agent row shows a liveness status derived from agents.last_seen_at (phaze.services.agent_liveness.classify):

Status Condition
alive now - last_seen_at < 90s (3x the 30s heartbeat cadence)
stale 90s <= now - last_seen_at < 300s (one or more missed beats)
dead now - last_seen_at >= 300s (~10 missed beats)
never agent registered but has never sent a heartbeat (last_seen_at IS NULL)
revoked agent has a revoked_at timestamp

You should see the agent reach alive within ~60s of just up-agent.

If the row shows never: the agent worker has not completed startup yet. Check the worker logs.

If the row shows stale then dead: the worker is up but heartbeats are not reaching the app-server. Check the agent worker logs for heartbeat failed: ... WARNING lines, and verify the agent can reach https://<app-server>:8000/api/internal/agent/heartbeat with the correct CA cert and token. If a busy worker (high CPU, actively analyzing) still appears stale/dead, confirm the agent image carries the Phase 46 build — before it, the heartbeat ran as a SAQ cron job that competed for the worker_max_jobs dispatch slots and was starved by multi-hour process_file jobs; the fix runs the heartbeat as an in-process asyncio background task that cannot be starved.

The watcher service

The watcher service (src/phaze/agent_watcher/, runnable via python -m phaze.agent_watcher) is an always-on asyncio process — not a SAQ worker. On startup it:

  1. Loads AgentSettings via get_settings() (raises if PHAZE_ROLE != agent).
  2. Calls /api/internal/agent/whoami with bounded retry to resolve the calling agent's identity and scan roots. A bad token short-circuits immediately (fail fast, no restart loop).
  3. Schedules one watchdog Observer per scan root and posts each settled file to the app-server.

Tunables (see AgentSettings in docs/configuration.md): PHAZE_WATCHER_SETTLE_SECONDS (default 10), PHAZE_WATCHER_SWEEP_INTERVAL_SECONDS (default 2), PHAZE_WATCHER_MAX_PENDING_SECONDS (default 3600), and PHAZE_WATCHER_POLLING_MODE (default false — set true for macOS Docker bind mounts where inotify does not propagate).

Build Pipeline

Images are built and published to the GitHub Container Registry (GHCR) by two reusable GitHub Actions workflows, both invoked from .github/workflows/ci.yml via workflow_call.

docker-validate.yml (validation, runs on every PR/push)

Called from the CI docker job (after quality, only when non-markdown files change). It:

  • Builds each Dockerfile via a two-entry matrix and lints them with hadolint (failure-threshold: error): Dockerfile (api) and Dockerfile.agent-arm64 (the cloud-compute arm64 agent image — lint-only here, its build:/push: steps are guarded off because a full QEMU C++ essentia compile on the x86 runner is forbidden; the real native build runs in docker-publish.yml).
  • Validates both compose files parse cleanly: docker compose -f docker-compose.yml config --quiet (with placeholder REDIS_PASSWORD/REDIS_BIND_IP) and docker compose -f docker-compose.agent.yml --env-file .env.agent config --quiet (with placeholder agent vars).

No images are pushed by this workflow — it is a gate.

docker-publish.yml (build + push to GHCR)

Called from the CI docker-publish job, which runs only after aggregate-results passes and only when code changed. It:

This workflow produces three GHCR artifacts across three build stages (push is true for non-PR events):

  • A single-image matrix (build-and-push) builds Dockerfile (api) on linux/amd64.
  • A native-arm64 stage builds Dockerfile.agent-arm64 on an ubuntu-24.04-arm runner. It loads (does not push) the image, runs an import smoke test, and hands the resolved -arm64 tags + OCI labels to a parity-gated pusher (parity-guard) — the arm64 image reaches GHCR only after its analysis output matches the x86 golden byte-for-byte, so a parity-divergent image can never publish ahead of the guard.
  • A build-job-runner stage builds Dockerfile.job FROM the freshly-pushed x86 api image (a needs: build-and-push dependent, not a matrix row) — the Kueue one-shot Job image.
Artifact Dockerfile Platform Consumed by
ghcr.io/simplicityguy/phaze Dockerfile amd64 app-server api/worker (built locally) and docker-compose.agent.yml's three lane workers (worker-analyze/worker-meta/worker-io) + watcher (pulled)
ghcr.io/simplicityguy/phaze:*-arm64 Dockerfile.agent-arm64 arm64 docker-compose.cloud-agent.yml cloud-compute agent (kind=compute); parity-gated push
ghcr.io/simplicityguy/phaze/job Dockerfile.job amd64 Kueue backend (kind=kueue) one-shot Jobs submitted by the control plane
  • The api image publishes to the bare repo URL ghcr.io/simplicityguy/phaze (no sub-path) so docker-compose.agent.yml's three lane workers + watcher can pull it directly; the arm64 variant is the same bare URL with a -arm64 tag suffix. ghcr.io/simplicityguy/phaze/api is a deprecated/orphaned path from a pre-D-15 convention — it is no longer published and must NOT be pulled or referenced.
  • Tag strategy (via docker/metadata-action): latest on the default branch, plus {{version}} and {{major}}.{{minor}} semver tags, ref-based tags (tag/branch/PR), and a dated schedule tag. Tagged releases therefore produce both :latest and :<version>.
  • Release tags MUST be 3-part CalVer (YYYY.M.REVISION, e.g. 2026.7.0) — ci.yml triggers the publish pipeline on push of a [0-9]+.[0-9]+.[0-9]+ tag, and the {{version}} / {{major}}.{{minor}} image tags are only produced for a 3-part ref. A 2-part tag (2026.7) will not match the trigger and will not publish version-pinnable images, so the 3-part shape is still required.
  • Builds with provenance: true and sbom: true for supply-chain attestation, on linux/amd64.

Release version scheme (CalVer). Releases use CalVer YYYY.M.REVISION with a bare tag (no v prefix — the first CalVer tag is 2026.7.0) and a no-leading-zero month (2026.7.0, not 2026.07.0). REVISION is a per-month zero-based counter: the Nth release within a given YYYY.M, starting at 0 and resetting each calendar month, so same-month patch releases just increment REVISION (2026.7.0 → 2026.7.1). Milestones are named; versions are dated — the two are decoupled.

Publish trigger (invariant). GHCR publish fires on the push of an annotated tag (git tag -a 2026.7.0 -m "…" then git push origin 2026.7.0) — creating the tag locally publishes nothing; the push of the tag ref is the sole trigger. If a tag was pushed wrong (bad SHA, premature), delete-and-recreate it: git push --delete origin 2026.7.0, fix, then re-tag and push again to re-fire the pipeline. docker-publish.yml's metadata-action is scheme-agnostic (type=semver parses 2026.7.0 → {{version}}=2026.7.0, {{major}}.{{minor}}=2026.7), so no workflow edit is needed to adopt CalVer.

flowchart LR
    df1["Dockerfile"] --> api["ghcr.io/…/phaze<br/>(amd64)"]

    api -->|"docker-compose.yml + agent.yml"| use1["api / worker / watcher"]

    api -.->|"FROM (needs build-and-push)"| dfjob["Dockerfile.job"]
    dfjob --> job["…/phaze/job (amd64)"]
    job -->|"kind=kueue Jobs"| kueue["Kueue backend"]

    df4["Dockerfile.agent-arm64"] --> loadarm["build + load (arm64)<br/>import smoke"]
    loadarm --> gate{"parity guard<br/>arm64 == x86 golden?"}
    gate -->|"pass"| arm["…/phaze:*-arm64"]
    gate -->|"fail"| block["push blocked"]
    arm -->|"docker-compose.cloud-agent.yml"| compute["kind=compute agent"]
Loading

The Dockerfile is multi-stage. A css-builder stage (FROM python:3.14-slim AS css-builder) compiles assets/src/app.css → src/phaze/static/css/app.css with the pinned standalone Tailwind v4 binary (TAILWIND_VERSION=v4.3.2, kept in sync with the justfile tailwind recipe — no Node, no CDN); the final base stage (FROM python:3.14-slim AS base) copies only the generated CSS. base installs deps with uv sync --frozen --no-dev in cached layers, copies src/, alembic/, and alembic.ini, runs as the non-root phaze user, and exposes port 8000. The api and worker containers share this image and diverge only by command.

You can also build/push manually with just: just docker-build, just docker-validate (hadolint), just docker-compose-validate, and just image-push (requires a gh token with packages:write).

Static asset caching (phaze-315t)

Every /static/... URL the app's templates emit (app.css, favicons, site.webmanifest, og_image.png) carries a ?v=<content-hash> query parameter — phaze.web.static.STATIC_VERSION, a SHA-256 fingerprint over the content of every file under src/phaze/static/, computed once at process import time. base.html / shell.html build these URLs via the static_url(...) Jinja global (registered per-router in shell.py, pipeline.py, execution.py, admin_agents.py — the four routers that render a full page extending base.html or rendering shell/shell.html).

RevalidatingStaticFiles (src/phaze/web/static.py, mounted at /static in main.py) answers a request whose ?v= matches the CURRENT STATIC_VERSION with Cache-Control: max-age=31536000, immutable — safe forever, because that exact URL can never later serve different bytes: if the content changes, the fingerprint changes too, and the browser fetches the new URL instead of reusing anything. Any other request (a stale fingerprint from a previous deploy, or a direct request with no ?v= at all — e.g. a browser's automatic /favicon.ico probe, or the icon paths inside site.webmanifest itself, which is static JSON and not template-rendered) falls back to Cache-Control: no-cache, forcing a cheap conditional-request/304 revalidation instead of a stale heuristic hit.

This replaces phaze-mw9l's no-cache-only fix (every hit, even an unchanged file, paid a round trip) and closes the defect it was filed against: a browser that had visited the site before a CSS-changing deploy previously kept rendering the OLD stylesheet against the NEW server HTML on an ordinary reload, because stock StaticFiles sent no Cache-Control at all and browsers applied heuristic freshness across deploys. Because the compiled src/phaze/static/css/app.css is produced by the Dockerfile's css-builder stage and COPY --from=css-builder'd into the final image (never a stale local copy — see .dockerignore), STATIC_VERSION computed at container startup always reflects exactly what that specific deployed image serves.

Environment Setup

The full environment-variable reference, including required-vs-optional status and defaults, lives in docs/configuration.md. The two templates in the repo are .env.example (app-server) and .env.example.agent (file-server agent).

Production-critical variables:

Variable Host Why it matters
POSTGRES_PASSWORD app-server ${POSTGRES_PASSWORD:?} — compose parse fails if unset (phaze-rnh7); there is no weak silent default. Use a strong unique value and keep DATABASE_URL / PHAZE_QUEUE_URL in sync with it.
POSTGRES_BIND_IP app-server Interface the published :5432 binds to. Must be the app-server's private LAN IP so agents can open their PHAZE_QUEUE_URL psycopg3 pool to the broker — at the 127.0.0.1 default they cannot reach it. Never 0.0.0.0, never a public IP. Mirrors REDIS_BIND_IP.
REDIS_PASSWORD app-server redis-server --requirepass; compose parse fails if unset. Use a unique high-entropy value (>= 32 chars).
REDIS_BIND_IP app-server Must be the app-server's private LAN IP so agents on other hosts can reach Redis. Never 0.0.0.0, never a public IP.
PHAZE_QUEUE_URL app-server + file-server The SAQ Postgres broker DSN (raw libpq postgresql://…, NOT +asyncpg). On agents it points at the app-server Postgres LAN IP:5432 — open that firewall edge (relaxes D-25). Carries DB credentials; use the _FILE secret form. Keep the per-queue pool budget under Postgres max_connections.
PHAZE_AGENT_ENV=production file-server Activates the AgentSettings guards: refuses non-https:// agent_api_url (CR-01) and passwordless redis_url (D-06). Note: there is no production credential guard on PHAZE_QUEUE_URL yet — protect it via the LAN-scoped firewall + a strong DB password.
PHAZE_AGENT_TOKEN file-server The plaintext bearer token; must match the token_hash row in agents. Generate via secrets.token_urlsafe(32).
PHAZE_AGENT_CA_FILE file-server Path to the operator-distributed phaze-ca.crt; the agent's HTTP client verifies the app-server TLS chain against it.
PHAZE_IMAGE_TAG file-server Pin to a specific version (for example, 2026.8.4) in production rather than latest.
SCAN_PATH file-server The music-library root, bind-mounted read-only into all agent services. Compose parse fails if unset.

Secrets via files (Docker secrets)

Every secret-bearing variable accepts a <VAR>_FILE sibling that points at a file holding the value, so you can mount a Docker/Swarm secret (or a Kubernetes secret / SOPS-decrypted file) instead of inlining cleartext into the environment. Supported: DATABASE_URL, REDIS_URL, PHAZE_QUEUE_URL (the SAQ Postgres broker DSN — queue_url), OPENAI_API_KEY, ANTHROPIC_API_KEY (each also via its PHAZE_* alias where one exists), and PHAZE_AGENT_TOKEN / AGENT_TOKEN. See docs/configuration.md → Secrets via files for the precedence and newline-stripping rules.

Example — mount the Anthropic key as a Docker secret on the app server and reference it by path (no ANTHROPIC_API_KEY in the environment):

# docker-compose.yml (app server)
secrets:
  anthropic_api_key:
    file: ./secrets/anthropic_api_key   # contents: the raw sk-ant-... key

services:
  api:
    secrets: [anthropic_api_key]
    environment:
      ANTHROPIC_API_KEY_FILE: /run/secrets/anthropic_api_key

The same pattern works for the agent's bearer token on a file server — mount the secret and set PHAZE_AGENT_TOKEN_FILE=/run/secrets/phaze_agent_token. The file's trailing newline is stripped, so the hashed wire string still matches the token_hash row (a stray \n would otherwise cause a permanent 401). A _FILE path that is missing or unreadable fails fast at startup with a clear error.

Rollback Procedure

There is no automated rollback in CI — rollback is a manual re-deploy of a previously published image tag.

File servers (agent stack) pull from GHCR, so rolling back is a tag swap:

# On the file-server host:
# 1. Edit .env: set PHAZE_IMAGE_TAG back to the last-known-good version, e.g.
#    PHAZE_IMAGE_TAG=2026.7.0
# 2. Re-pull and recreate the agent containers:
docker compose -f docker-compose.agent.yml pull
docker compose -f docker-compose.agent.yml up -d

Because docker-publish.yml tags both :latest and :<version>, every release remains pullable by its version tag — keep PHAZE_IMAGE_TAG pinned in production so a rollback is just editing one line.

Application server is built locally from the checkout, so rolling back means checking out the previous git tag and rebuilding:

# On the app-server host:
git checkout 2026.7.0        # the last-known-good CalVer release tag (a pre-CalVer rollback legitimately still uses its old v4.0.0 tag — mechanism unchanged)
just rebuild                 # docker compose -f docker-compose.yml up -d --build

To stop and restart cleanly without rebuilding: just down (docker compose down) then just up. The pgdata named volume and ./certs/ persist across down/up, so no data or cert state is lost.

Do not rm -rf ./certs/ as part of a rollback — that triggers a full CA regeneration and breaks every agent until the new phaze-ca.crt is re-distributed (see CA Rotation below).

One-time cleanup after the Phase 46 heartbeat fix

Builds before Phase 46 registered the heartbeat as a SAQ CronJob with unique=True, which parks a deterministic row keyed cron:heartbeat_tick in the Postgres broker (saq_jobs). After you redeploy the new agent image, nothing re-schedules that row (the heartbeat now runs as an in-process asyncio background task), so it lingers as a stale parked entry. Run this one-time cleanup after the new agent image is deployed, against the queue database (PHAZE_QUEUE_URL):

DELETE FROM saq_jobs WHERE key = 'cron:heartbeat_tick';

It is harmless if the row is already absent (e.g. on a fresh broker or after a saq_jobs truncate). This mirrors the prior Redis orphaned-cron-purge runbook, adapted to the Postgres broker.

Historical: sidecar volume chown runbook (removed, phaze-0jpe)

Prior revisions of this guide carried a one-time volume-ownership repair for the audfprint/ panako fingerprint sidecars (a uid-999-to-1000 mismatch on pre-7b9adec hosts). Both engines and their sidecars were removed in full — 2026-07-28, epic phaze-0jpe; see docs/design/0002-fingerprint-removal.md for why — so the runbook no longer applies to a current checkout and has been deleted from this guide. The live audfprint_data/panako_data Docker volumes on any already-deployed file server are not touched by this removal; their disposal is a separate, explicitly-approved operator step (see docs/runbook.md, phaze-0jpe.6) because they are the only surviving evidence for the two production outages the ADR cites.

Monitoring & Health

  • API health endpoint: GET /health returns {"status":"ok"} and checks database connectivity (SELECT 1). It requires Postgres to be reachable. Use it as the app-server liveness probe: curl --cacert ./certs/phaze-ca.crt https://<app-server>:8000/health.
  • Agent heartbeat / liveness: every lane worker (worker-analyze/worker-meta/worker-io, all with PHAZE_AGENT_HEARTBEAT=true) runs an asyncio background task (every 30s — phaze.tasks.heartbeat._heartbeat_loop, gated by PHAZE_AGENT_HEARTBEAT) that POSTs to /api/internal/agent/heartbeat with {agent_version, worker_pid, queue_depth, lane}. It is launched in the worker startup hook and cancelled on shutdown, so it runs outside the SAQ job-dispatch pool and is never starved by long-running analysis jobs (Phase 46). Each beat carries its own lane tag and that lane's depth; the control plane keeps the per-lane breakdown and sums an honest all-lane queue_depth, while last_seen_at is inherently max(last_seen) across lanes. This replaced the former single-heartbeat convention (phaze-30fo): pinning liveness to worker-analyze alone meant one stalled process marked the whole agent DEAD and cost it work-routing rank (select_active_agent orders by last_seen_at DESC) while its other lanes were busy. Only the transitional worker-drain sets PHAZE_AGENT_HEARTBEAT=false — it is unlaned, and an untagged beat would wipe the per-lane breakdown (see agent-queue-lanes.md). The endpoint stamps agents.last_seen_at and persists the payload to the agents.last_status JSONB column. The /admin/agents page classifies each agent as alive/stale/dead/never/revoked from last_seen_at (thresholds: alive < 90s, dead >= 300s) and self-refreshes every 5s via HTMX.
  • Worker health: just worker-health runs the SAQ --check against the controller worker; just worker-logs follows its logs.
  • Logging: services log to stdout/stderr (docker compose logs -f <service>). The cert-bootstrap banner additionally lands in docker compose logs api via logger.warning(). No external metrics/tracing exporter (Sentry, Datadog, OpenTelemetry) is configured in this repo.

Filesystem-Isolation Smoke (D-20)

To verify DIST-01 (the app-server has no way to read or write music files), exec into the api container and try to read a file:

docker compose exec api ls -la /data/music
# Expected: ls: cannot access '/data/music': No such file or directory

Or trust the structural test that runs in CI:

uv run pytest tests/agents/deployment/ -v

The compose-parse tests assert that docker-compose.yml declares no SCAN_PATH, MODELS_PATH, or OUTPUT_PATH bind mounts on api or worker services — only ./certs/ is mounted on api (and that one is required for the cert bootstrap).

CA Rotation (caution)

The CA + leaf cert generated in Step 1 is valid for 10 years (CA) / 2 years (leaf). If you ever need to rotate:

# On the app-server host:
rm -rf ./certs/                       # destructive — all current cert state is lost
docker compose restart api            # cert_bootstrap regenerates + prints the loud banner again
# Then repeat Step 2 (copy ./certs/phaze-ca.crt to every file server) and restart each agent.

Every file-server agent will fail to connect until you re-distribute the new phaze-ca.crt. The loud banner is the only safeguard — do not delete the certs directory casually.

Kubernetes burst path (v6.0). The K8s one-shot Job does not bake the CA into its image — it mounts the public phaze-ca.crt from an operator-created core/v1 Secret read-only at /certs (KDEPLOY-06; PHAZE_KUBE_CA_SECRET_NAME, default phaze-internal-ca). After regenerating the CA above, re-create that Secret with the new phaze-ca.crt (kubectl create secret generic phaze-internal-ca --from-file=phaze-ca.crt=./certs/phaze-ca.crt) and let in-flight Jobs re-submit — no Job-image rebuild. See k8s-burst.md §6 for the full runbook.

Pinning the agent image for production

For first-time setup, PHAZE_IMAGE_TAG=latest pulls the most recent tagged release from GHCR. For production, pin to a specific version:

# On the file-server host's .env:
PHAZE_IMAGE_TAG=2026.8.4

Then just up-agent pulls exactly that version. The docker-publish.yml workflow tags both :latest and :<version> on tagged releases. The pin MUST be a 3-part CalVer YYYY.M.REVISION value (for example, 2026.8.4) matching a published release tag (ci.yml only publishes on push of a [0-9]+.[0-9]+.[0-9]+ tag).

Pre-warming models (skip the first-start wait)

To avoid the multi-GB model download on first agent boot:

# On the file-server host BEFORE just up-agent:
just download-models

This runs bash scripts/download-models.sh models, populating ./models/ directly; the agent's auto-download check then no-ops.

1001Tracklists render worker: residential-IP constraint (phaze-fq9h.5)

The 1001Tracklists lookup pipeline (epic phaze-fq9h) renders detail pages with a real, HEADFUL Google Chrome (services/tracklist_render.py, TracklistRenderer/PatchrightLauncher) — the only client spike phaze-dmvs found that clears the Cloudflare Turnstile widget without evasion (fingerprint MATCHING: no proxies, no UA randomization, no CAPTCHA solvers — the epic's ethics ceiling). That client-fidelity work is undone if the network path still looks like a bot.

This is a deployment constraint, not something the render engine's code can work around. 1001Tracklists' own FAQ states its anti-scraping measures block requests from datacenter, VPN, and Tor IP ranges. Whatever host actually executes the render worker — a worker-* lane, a future tracklist-drain service, or an ad hoc scripts/capture_tracklist_render.py run — must egress through a residential (consumer ISP) IP address. A cloud VM, VPS, colo box, the OCI Always Free tier used for cloud-burst compute (docker-compose.cloud-agent.yml), and hosted CI runners (GitHub Actions ubuntu-latest included) are all datacenter ranges and are all in scope of this block — none of them are a valid home for this worker, regardless of how the browser itself is configured.

What happens if you get this wrong. The render engine cannot tell "IP-blocked" apart from an ordinary flaky Turnstile challenge from its own vantage point — both look like the page never delivering the .tlpItem track container. Every render will exhaust tracklist_render_turnstile_attempts and report RenderOutcome.INTERSTITIAL_PERSISTED (retryable, so nothing gets negative-cached — see RenderResult.is_retryable), forever, silently spending the shared crawl-delay-8 host budget (phaze-hu8v) on requests that can never succeed no matter how many attempts or how long the drain runs. There is no distinct "blocked" outcome to alert on; the operator-visible signal is an unusually high, persistent INTERSTITIAL_PERSISTED rate that does not improve across the corpus. Before trusting a bulk run's results, confirm the render host clears the anchor URL at all (scripts/capture_tracklist_render.py --trials 6, run manually, one process at a time, never wired into CI — see that script's own docstring).

For a future operator considering a VPS or cloud migration for this worker specifically: it will stop working, silently in the sense above (no crash, no explicit "blocked" error — just a 100% INTERSTITIAL_PERSISTED rate), the moment the egress IP moves off a residential connection. The epic's ethics bound rules out the usual fix (routing through a residential proxy service is exactly the kind of evasion phaze-dmvs deliberately avoided), so the render worker's host placement is a real, permanent constraint on where this feature can run — keep it on hardware with a residential ISP connection (e.g. the same home-server host the file-server agent already runs on).

Production Checklist

Before shipping a file-server host to production:

  • POSTGRES_PASSWORD set to a strong unique value — docker compose fails to parse without it (phaze-rnh7); DATABASE_URL / PHAZE_QUEUE_URL updated to match
  • POSTGRES_BIND_IP set to the app-server's private LAN IP — at the 127.0.0.1 default agents cannot reach the PHAZE_QUEUE_URL broker on :5432 (never 0.0.0.0, never the public IP)
  • REDIS_PASSWORD set to a unique high-entropy value (>= 32 chars) — never the default
  • REDIS_BIND_IP set to the app-server's private LAN IP (never 0.0.0.0, never the public IP)
  • PHAZE_AGENT_ENV=production — enables the redis-password-required and https-required guards in AgentSettings
  • PHAZE_AGENT_TOKEN generated via secrets.token_urlsafe(32), not a placeholder
  • phaze-ca.crt distributed via secure channel (scp over SSH, not email/chat)
  • phaze-ca.key NEVER copied off the app-server host
  • PHAZE_IMAGE_TAG pinned to a specific release (for example, 2026.8.4), not latest
  • SCAN_PATH points at the actual music library root (compose parse fails if unset)
  • Production bring-up uses just up (base docker-compose.yml only) — the dev overlay docker-compose.dev.yml is opt-in via just up-dev and is no longer auto-merged, so it cannot bypass the cert-bootstrap entrypoint (phaze-476w)
  • Filesystem-isolation smoke confirmed (see above) — docker compose exec api ls /data/music returns "No such file or directory"
  • /admin/agents page shows alive status within ~60s of just up-agent
  • If this host runs the 1001Tracklists render worker: confirmed to egress through a residential IP (never a cloud VM/VPS/colo/CI runner) — see "1001Tracklists render worker: residential-IP constraint" above

See also

  • .env.example — app-server environment template
  • .env.example.agent — file-server agent environment template
  • docker-compose.yml — app-server compose
  • docker-compose.agent.yml — file-server agent compose
  • docker-compose.cloud-agent.yml — OCI A1 cloud compute-agent compose (just cloud-agent-up / cloud-agent-down)
  • docker-compose.dev.yml — dev-only overlay (live reload; just up-dev, explicit -f, never auto-merged)
  • backends.toml — the cloud-backend registry (mounted at PHAZE_BACKENDS_CONFIG_FILE; absent ⇒ implicit local-only)
  • docs/cloud-burst.md — kind=compute agent deploy + runbook (deep-dive; config surface is backends.toml)
  • docs/k8s-burst.md — kind=kueue cluster deploy + runbook (deep-dive; config surface is backends.toml)
  • docs/configuration.md — full environment-variable reference
  • docs/architecture.md — system architecture overview