Contributing to stl

Welcome!

TL;DR — the non-negotiables (read this even if you skip the rest)

• Pick a category first (see "What goes where" below): worker, cronjob, backfiller, model, or API endpoint — each has a home.
• Go/Python for workers / cronjobs / backfillers. Python for APIs and quantitative risk models. Anything else needs prior discussion.
• On-chain data should come from chain RPC or our cached block payload, not from third-party indexers or other derived/intermediate feeds. Off-chain feeds (CoinGecko, DefiLlama, etc.) need a good reason and case-by-case approval from the maintainers.
• Keep data pipelines separate from model pipelines — ingest ≠ compute-meaning. The correct pattern is for data pipelines to write to the data store, and model pipelines to ingest the data needed from that store.
• Create tables plain. Compression and tiering policies come with the later migration that partitions the table, once measurement says to (see §11 rule 4).
• Never modify an applied migration — write a new one.
• PR title: TICKET-1234: <description>. GitHub squash-merges; don't squash locally.

This document is aimed at anyone who has never seen the repo before — in particular:

• contributors of quantitative risk models (the app/risk_engine/ side)
• contributors of data pipelines that ingest on-chain and off-chain data (the workers, cronjobs, backfillers)

It explains the layout, how to run things locally, how data acquisition works, and the conventions you're expected to follow when opening a pull request.

If anything here is wrong or unclear, fix it in the same PR as the work that surfaced the problem. Docs rot fast — keep them honest.

Stuck on the actual work? Jump to §16 Getting help.

---

1. Prerequisites

Tool	Why	Install
Go 1.26+	Every shipped service is Go (experiments/ is scratch and exempt)	https://go.dev/dl/
Docker	Local infra (Postgres, Redis, Temporal, LocalStack)	Docker Desktop / Colima
kind	Runs a Kubernetes cluster inside Docker — mirrors prod	brew install kind
kubectl	Talks to the kind cluster	brew install kubectl
kustomize (optional)	Only needed for manually previewing a deploy (kustomize build … | kubectl diff -f -). make dev-up uses kubectl's built-in kustomize.	brew install kustomize
AWS CLI (optional)	Only needed if you want to fetch real Alchemy keys via make dev-env	brew install awscli
An Alchemy API key	Mainnet access	ask a team member or sign up at alchemy.com

You do not need AWS credentials to develop locally — the kind cluster runs LocalStack to emulate SNS/SQS/S3.

AI coding assistants (optional)

AGENTS.md is the canonical source of repository instructions. The sibling CLAUDE.md files import it for Claude Code; Codex and other AGENTS.md-aware tools read it directly. Before an assistant modifies a subtree, it must read every AGENTS.md from the repository root through that subtree. In particular, Codex builds its automatic instruction chain from the repository root through its startup directory and does not add nested or sibling guidance later in the session.

Repository skills have canonical sources in skills/ and are deployed to each supported tool through the root Skillfile. After a fresh clone, run this once from the repository root to make the same skills available to Codex, Claude Code, and Copilot:

make skills-install
make skills-list

The generated deploy directories (.claude/skills/, .codex/skills/, and .github/skills/) are all git-ignored; rerun make skills-install after cloning or pulling a skill change, then start a new assistant session so its skill catalogue is refreshed. When adding or editing a skill, change its source under skills/ (local skills) or the Skillfile (remote skills), redeploy with make skills-install, and commit the canonical source plus the Skillfile/Skillfile.lock — never the generated deploy copies.

---

2. Five-minute quick start

git clone git@github.com:archon-research/stl.git
cd stl/stl-verify

# One-time: install lint/test tools.
make tools

# Spin up a full local pipeline (kind cluster, Postgres, Redis,
# LocalStack, Temporal, watcher, workers, cronjobs, python-api).
make dev-up

The first run builds every Docker image and takes several minutes. Warm re-runs skip already-built images; run make dev-up-rebuild (alias for COLD=1 make dev-up) to force a rebuild.

When it finishes:

kubectl --context=kind-vector get pods -n vector

Everything should be Running. For local-only pause/resume use make dev-suspend and make dev-resume (do not use these in CI/prod). Use make dev-down to delete the cluster; nuke persistent volumes too with make dev-wipe.

Need a second cluster next to someone else's (two agents, one machine)?

make dev-up-new                                  # free offset, name derived from it
KIND_CLUSTER=mine KIND_PORT_OFFSET=100 make dev-up   # or name and offset yourself

It gets its own cluster name, host ports (every mapped port +100), image tags (stl-*:local-mine) and data dir (~/.mine), so it cannot disturb the default vector cluster. dev-up-new prints the host endpoints when it is done; afterwards only export KIND_CLUSTER=<name> is needed, since every run-*, dev-* and kind-* target derives the offset from the cluster's own control plane.

⚠️ You need an Alchemy key for anything to actually work. By default make dev-up points the watcher at a mock blockchain server that ships with the repo. The mock is enough to boot the cluster and exercise plumbing, but it is not a fully implemented chain — most RPC methods are stubs, responses are synthetic, and any worker that expects realistic block / receipt / trace data will produce garbage or fall over.

To run against the real chain, put your key in .env.secrets at the repo root (the file is created by make dev-preflight on first run) and switch the cluster over:

make kind-secrets        # propagate .env.secrets into the cluster
make kind-use-alchemy    # point the watcher at Alchemy instead of the mock

To go back to the mock (e.g. offline dev): make kind-use-mock.

To iterate on a single service without rebuilding the whole cluster, use the run-* targets, which run the binary on your host against the kind cluster's Postgres/Redis:

make run-watcher                      # Ethereum live watcher
make run-oracle-price-worker          # Oracle-price per-block worker
make run-sparklend-position-tracker   # SparkLend per-block worker
# ... grep `^run-` in stl-verify/Makefile for the full list

If a run-* target needs secrets (e.g. ALCHEMY_API_KEY), run make dev-env first — it pulls secrets from AWS Secrets Manager and writes per-service .env files.

---

3. Repo layout

stl/
├── stl-verify/           # All Go code lives here (single Go module)
│   ├── cmd/              # One subdirectory per binary (see §7)
│   ├── internal/
│   │   ├── domain/       # Business entities — zero dependencies
│   │   ├── ports/        # Interfaces (inbound = use cases, outbound = infra)
│   │   ├── services/     # Use-case implementations
│   │   ├── adapters/
│   │   │   ├── inbound/  # HTTP / gRPC / CLI handlers
│   │   │   └── outbound/ # alchemy, postgres, redis, sns, sqs, s3, temporal…
│   │   └── pkg/          # Cross-cutting helpers (env, telemetry, lifecycle…)
│   ├── db/migrations/    # SQL migrations, applied automatically
│   ├── python/           # Python API (served via k8s)
│   ├── ts/               # TypeScript helpers
│   └── Makefile          # The canonical entry point for every workflow
├── k8s/
│   ├── base/<svc>/       # env-agnostic Kustomize manifests per service
│   └── overlays/{staging,prod}/  # image tags + namespace per environment
├── docs/                 # Protocol specs, ADRs, entity diagrams
├── experiments/          # Scratch — not shipped
└── CLAUDE.md / AGENTS.md # Agent conventions (also at stl-verify/AGENTS.md — worth reading as a human too)

Infrastructure (Terraform/OpenTofu) lives in a separate private repo for security reasons. If your change needs new AWS resources (a new SQS queue, an SNS subscription, an IAM policy, a secret, etc.), the change to that repo must land before the code here can deploy cleanly.

What goes where

If you're adding…	It goes in…
A quantitative risk model (Python)	stl-verify/python/app/risk_engine/<model>/, exposed via a service in app/services/
An on-chain per-block data pipeline	cmd/workers/<name>/ (Go) or cli/workers/<name>/ (Python)
An off-chain API data pipeline (periodic)	cmd/cronjobs/<name>/ (Go) or cli/cronjobs/<name>/ (Python)
A historical on-chain backfill	cmd/backfillers/<name>/
A new HTTP endpoint for serving data out	stl-verify/python/app/api/v1/ (extend the python-api service)
A new TimescaleDB schema (table, index, backfill)	A new migration file under stl-verify/db/migrations/
A protocol spec or design doc	docs/

Don't touch experiments/ (scratch, not shipped), the image tags in k8s/overlays/{staging,prod}/kustomization.yaml (CI owns those).

---

4. Language policy — what to write new code in

Pick the language before you start, and when in doubt, ask first.

• APIs are Python. Anything user- or client-facing — endpoints, anything that serves data out to another system — belongs in stl-verify/python/. The existing python-api service is the reference; extend it rather than starting a new HTTP server.
• Workers, cronjobs, and backfillers can be any language you prefer, but we strongly prefer Go or Python. Most of the pipeline is Go today (the watcher, all SQS workers, all Temporal cronjobs), which means Go is the path of least resistance: shared helpers, the lifecycle.Run/temporal.RunCronjob harnesses, the hexagonal scaffolding, the Makefile wiring, and CI are all built around it. Python is a fully supported second option.
• Anything outside Go or Python needs prior discussion. Open an issue or start a thread with @archon-research/stl-engineers before writing code. PRs introducing a new runtime (Rust, TypeScript, Java, …) without a prior design conversation may be rejected regardless of code quality — every new language adds build infrastructure, observability integration, deployment, dependency-management, and on-call load that the team has to absorb forever. That cost is only worth paying when we've agreed it is.
• The existing stl-verify/ts/ is for the frontend/UI (currently being built out). That's the established use case; anything else in TypeScript still needs prior discussion.

---

5. Patterns and anti-patterns

Two rules that apply everywhere in this repo — they override local convenience:

• Prefer on-chain data whenever we can. If the data lives on the chain, it should be read from the chain (via Alchemy / Erigon and the cached block payload) — it's auditable (we can replay a block), trust-minimized (no third party), and stays consistent with the rest of the pipeline. Off-chain sources (CoinGecko, Anchorage, Etherscan, etc.) are allowed only with a good reason and with explicit approval from the repo maintainers — e.g., the data only exists off-chain. Write the justification in the PR description so that reviewers can see it.
• Data pipelines and model pipelines stay separate. A data pipeline writes "what happened on the chain or at an external API" into Postgres as append-only time-series. A model pipeline reads that data from Postgres and writes "what it means" (risk scores, derived metrics, liquidation forecasts) into its own tables. Don't merge the two into a single worker or cronjob — coupling ingest to model output means every model change forces a re-ingest, and every ingest bug corrupts the model surface. Split them across separate entry points (and usually separate PRs).

---

6. Architecture in one picture

The repo follows a hexagonal (ports and adapters) architecture. Dependencies point inward: domain ← ports ← services ← adapters, and cmd/* wires concrete adapters into services.

(Exported from the team Excalidraw board. The ASCII diagram below is the text-searchable canonical — if it drifts from the image, one of them is wrong.)

                                 ┌─► Redis (block cache, 2d TTL)
                                 │
   Alchemy WS ──► watcher ───────┼─► Postgres (block_state)
                                 │
                                 └─► SNS FIFO (one topic per chain)
                                             │
                                             ▼
                                       SQS FIFO queues
                                             │
                            ┌────────────────┼────────────────┐
                            ▼                ▼                ▼
                     oracle-price-    sparklend-       morpho-indexer
                       worker          tracker          (and others)
                            │                │                │
                            └────────────────┴────────────────┘
                                             ▼
                                   Postgres / TimescaleDB
                                   (time-series, append-only)

   Temporal schedules ──► cronjob pods ──► external APIs ──► Postgres
                          (anchorage,       (CoinGecko,
                           offchain-prices,   Anchorage,
                           data-validator)    …)

The watcher fans out three independent writes per block: Redis (hot cache), Postgres (block_state bookkeeping), and SNS (the message that triggers workers). The cache key convention is:

stl:{chainId}:{blockNumber}:{version}:{dataType}

where version is bumped when a reorg invalidates a block and dataType is one of block, receipts, traces, blobs. The SNS/SQS message only carries a block pointer — workers fetch the actual payload from Redis using this key.

---

7. The cmd/ tree — where binaries live

Each subdirectory under cmd/ is one main.go. They are grouped by lifecycle:

Group	Purpose	Example
cmd/base/	The chain watcher — the source of block events	watcher
cmd/workers/	Long-running SQS consumers — one message per block	oracle-price-indexer, morpho-indexer, sparklend-indexer, raw-data-backup
cmd/cronjobs/	Long-running Temporal workers run on a schedule (plus morpho-v2-bootstrap, which has none and is started by hand)	offchain-price-indexer, anchorage-indexer, watcher-data-validator
cmd/backfillers/	One-shot jobs that fill historical gaps	oracle-pricing-backfill, sparklend-backfill, raw-block-bulk-downloader
cmd/util/	Dev tooling (migrate, generate-er, stress-test helpers)	—

If you're adding data acquisition, pick the category first — it determines the plumbing, deployment shape, and tests you'll write.

Examples: "Track liquidations on a new lending protocol" → per-block worker in cmd/workers/. "Snapshot an off-chain API every 15 min" → cronjob in cmd/cronjobs/. "Re-ingest last month's oracle prices" → backfiller in cmd/backfillers/.

---

8. Data acquisition — per block

This is the hot path: extract something from every new Ethereum (or Avalanche / Arbitrum / Base / Optimism / Unichain) block.

How it works

1. cmd/base/watcher subscribes to newHeads over an Alchemy WebSocket, fetches the full block (header, receipts, traces, optional blobs) via HTTP, writes the raw block into the Redis cache, and publishes a BlockEvent to an SNS FIFO topic. It also handles reorgs by bumping the cache version and re-emitting affected blocks.
2. SNS → SQS FIFO fan-out (defined in the Infrastructure repo) gives each worker its own queue, partitioned by block number for ordered processing.
3. A worker in cmd/workers/<name> pulls a message, reads the cached block data from Redis, does its protocol-specific work (e.g. call oracle contracts at that block, diff SparkLend positions, etc.), and writes rows to Postgres/TimescaleDB.

Every worker follows the same skeleton (cmd/workers/oracle-price-indexer/main.go is the reference):

func run(ctx context.Context, args []string) error {
    cfg, err := parseConfig(args)          // flags + env, fail fast on missing config
    ...
    consumer, err := sqsadapter.NewConsumer(awsCfg, sqsadapter.Config{...}, logger)
    pool, err   := postgres.OpenPool(ctx, postgres.DefaultDBConfig(cfg.dbURL))
    buildReg, err := buildregistry.New(ctx, pool)                       // artefact identity; hard error if incomplete
    referenceEffectiveAt, err := env.ReferenceEffectiveAt(time.Now().UTC())
    runID, err  := buildReg.OpenRun(ctx, referenceEffectiveAt, nil)     // one writer_run per process start
    repo, err   := postgres.NewOnchainPriceRepository(pool, logger, buildReg.BuildID(), runID, 0)
    service, err := oracle_price_worker.NewService(shared.SQSConsumerConfig{...}, consumer, repo, ...)

    return lifecycle.Run(ctx, logger, service) // runs the consume loop; handles SIGINT/SIGTERM graceful stop
}

Adding a new per-block worker

1. Create cmd/workers/<my-worker>/main.go. Copy an existing worker as a template. Keep main() small — it parses flags, wires adapters, and calls lifecycle.Run. Every binary that connects to Postgres registers its artefact and opens a writer run at startup (buildregistry.New + OpenRun, ADR-0006 §2) and passes the RunID into its repositories next to the BuildID; startup reads of an append-on-change reference table go inside OpenRun's load callback (internal/pkg/oraclewire is the reference).

2. Create a service in internal/services/<my_worker>/. The service owns the business logic, depends only on ports, and exposes a public API tested in isolation (mock the repo + consumer + any contract caller). Services are expected to have ~100% unit-test coverage.

3. Add outbound ports/adapters if needed. A new protocol reader goes in internal/adapters/outbound/<vendor>/; expose it behind a small interface in internal/ports/outbound/.

4. Add migrations (see §11) for any new tables.

5. Add k8s manifests under k8s/base/<my-worker>/: deployment.yaml, serviceaccount.yaml, kustomization.yaml. Copy k8s/base/oracle-price-worker/ as the template. Wire the new service into k8s/overlays/{staging,prod}/kustomization.yaml by adding the base dir under resources: only — the images: block there is generated from k8s/image-roster.txt, so add one roster line instead (kind, image name, the image: alias your manifests use; ORB-362). For local kind, k8s/overlays/dev/workers/kustomization.yaml still takes both the base dir under resources: and a localhost/stl-<name>:local images: entry, plus a patch setting AWS_SQS_QUEUE_URL to the LocalStack queue. The dev runtime Component replaces the base's envFrom with the shared stl-config + stl-secrets, so anything your base reads from a per-service ConfigMap (e.g. the DEX indexers' DEX) must be set explicitly in that patch.

6. Create the local SQS queue in the LocalStack init script, stl-verify/localstack-init/init-aws.sh — the single source of truth — via create_consumer_queue <chain> <name>, which creates the FIFO queue + DLQ and subscribes it to the chain's blocks topic with raw delivery. make kind-infra generates the localstack-init ConfigMap from that file and stamps a hash of it into the LocalStack pod template, so the pod restarts and your queue exists after the next make dev-up. Nothing is inlined in k8s/dev-infra/localstack.yaml.

7. Add build/deploy targets to the Makefile (docker-build-<name>, docker-release-<name>, and register the worker in the run-* / kind-load-workers / kind-deploy-workers groupings, plus the rollout lists in _dev-up-alchemy-workers and dev-up). Grep for an existing worker name in the Makefile to see every site you need to touch.

   Include docker-release-all. It is an explicit chain of _docker-release-<name>-internal calls, not a wildcard — only cronjobs are picked up automatically, via its $(CRONJOBS) loop. Forgetting it is caught loudly rather than silently: that target is what the deploy builds with, so the image is never pushed and the overlay's pinned tag reaches the cluster as ImagePullBackOff (ORB-313).

8. Coordinate with infra. Open a PR in the Infrastructure repo for the SQS queue, SNS subscription, IAM policy, and any secrets — your code PR depends on those resources existing.

Once steps 5–7 are in place, make dev-up runs the worker in kind (when ALCHEMY_API_KEY is set in .env.secrets), consuming the in-cluster watcher's blocks through LocalStack SNS→SQS — no host-run binary, no ad-hoc queue script. The DEX indexers (curve-indexer, uniswap-v3-indexer, uniswap-v4-indexer, all one stl-dex-indexer image) are wired this way.

Tip: It's welcome (often preferred) to split the k8s-manifest and Infrastructure-repo changes into a follow-up PR. The code PR stays narrowly scoped for engineering review, and the deploy/infra PR stays narrowly scoped for the infra owner — each reviewer only sees their domain.

If you're writing the worker in Python

The flow is identical to the Go version — same SNS FIFO fan-out, same SQS FIFO queue, same Redis cache lookup by stl:{chainId}:{blockNumber}:{version}:{dataType}, same Postgres writes. Only the language changes.

No Python SQS worker exists in the repo yet. If you're the first, you'll be introducing the shared plumbing (SQS/Redis adapters, Dockerfile, Makefile targets) on top of the business logic — budget for it. Put every reusable piece under app/adapters/… so the second Python worker is copy-paste.

Tooling: we use uv for Python packaging and execution — not pip, pip-tools, or poetry. Add deps with uv add <pkg>; run anything under the project env with uv run <cmd>; refresh the lockfile with uv sync.

Code skeleton — entry point at stl-verify/python/cli/workers/<my_worker>/main.py:

import asyncio
import signal

from app.adapters.postgres import open_pool
from app.adapters.redis import BlockCache           # thin wrapper around redis-py
from app.adapters.sqs import SQSConsumer            # thin wrapper around aioboto3
from app.config import load_config
from app.services.my_worker import MyWorkerService

async def run() -> None:
    cfg = load_config()                             # env-driven, fail fast on missing vars
    consumer = SQSConsumer(cfg.queue_url, cfg.aws_region)
    cache    = BlockCache(cfg.redis_url)
    pool     = await open_pool(cfg.database_url)

    service = MyWorkerService(consumer, cache, pool, chain_id=cfg.chain_id)

    stop = asyncio.Event()
    loop = asyncio.get_running_loop()
    for sig in (signal.SIGINT, signal.SIGTERM):
        loop.add_signal_handler(sig, stop.set)

    try:
        await service.run_until(stop)               # consume loop + per-message handling
    finally:
        await pool.close()

if __name__ == "__main__":
    asyncio.run(run())

Run it locally with uv run python -m cli.workers.<my_worker>.main (from stl-verify/python/).

Directory layout — mirrors the Go hexagonal structure (Python uses cli/ for entry points where Go uses cmd/):

• stl-verify/python/cli/workers/<my_worker>/main.py — the entry point shown above. No business logic here.
• stl-verify/python/app/services/<my_worker>/ — business logic for one block event. Full unit tests; mock the SQS consumer, the Redis reader, the repo, and any contract caller.
• stl-verify/python/app/domain/entities/ — pure entities.
• stl-verify/python/app/ports/ — interfaces.
• stl-verify/python/app/adapters/{sqs,redis,postgres,onchain,…}/ — concrete implementations, one subpackage per vendor.

Operational contract (must match Go workers exactly):

• Long-poll SQS receive; process one message at a time in FIFO order; delete on success; let it redrive on failure.
• Handle SIGINT / SIGTERM — finish the in-flight message, close the DB pool, exit within Go's lifecycle.ShutdownTimeout (40s) plus lifecycle.ShutdownTailBudget (45s), the Python equivalent of lifecycle.Run.
• Read block data from Redis using the exact cache-key convention above; do not refetch from Alchemy unless cache-miss rate indicates a real bug.

Adding a new Python per-block worker:

1. Create stl-verify/python/cli/workers/<my_worker>/main.py from the skeleton above. Keep it small.
2. Create the service in stl-verify/python/app/services/<my_worker>/ with full unit tests (mock every adapter).
3. Add ports/adapters if needed. A new protocol reader goes in app/adapters/<vendor>/, exposed via a small interface in app/ports/. If this is the first Python worker, you'll also create app/adapters/sqs/ and app/adapters/redis/.
4. Add dependencies with uv — e.g. uv add aioboto3 redis from stl-verify/python/. This updates both pyproject.toml and uv.lock atomically.
5. Add migrations (see §11) for any new tables.
6. Add a Dockerfile. The existing stl-verify/python/Dockerfile is FastAPI-specific. First Python worker introduces either a multi-target Dockerfile (build arg picks cli/workers/<name>) or a parallel Dockerfile.worker. Subsequent workers reuse it. Use the uv base image pattern already in the existing Dockerfile (COPY --from=ghcr.io/astral-sh/uv …, uv sync --frozen --no-dev).
7. Add Makefile targets — docker-build-<name>, docker-release-<name>, kind-load-<name>, and a run-<name> target for local dev (invokes uv run under the hood). Local deploy goes through the dev overlay (kind-deploy-apps, or kind-deploy-workers for SQS workers) — there is no per-service kind-deploy-<name> target. Follow the pattern of docker-build-python-api / docker-release-python-api in stl-verify/Makefile.
8. Add k8s manifests under k8s/base/<my-worker>/ (deployment.yaml, serviceaccount.yaml, kustomization.yaml). Copy k8s/base/oracle-price-worker/ as the template; update image name and any probes. Register in k8s/overlays/{staging,prod}/kustomization.yaml and, for local kind, k8s/overlays/dev/workers/kustomization.yaml.
9. Coordinate with infra. Same SQS queue / SNS subscription / IAM PR as for a Go worker — the plumbing outside the worker is language-agnostic.

Tip: Same as Go workers — split the k8s-manifest and Infrastructure-repo changes into a follow-up PR.

---

9. Data acquisition — on a cron schedule

Use this when the data source is not keyed on block height — typically an external REST API (CoinGecko, Anchorage) or a periodic consistency check of our own data.

How it works

We use Temporal for scheduling, not Kubernetes CronJobs. Each cronjob is a plain Deployment running a Temporal worker that registers a schedule and an activity on startup:

• The shared runner is internal/adapters/outbound/temporal.RunCronjob.
• Schedules are stored in Temporal, not in k8s. On startup the worker calls ScheduleClient().Create — if it already exists, Temporal returns AlreadyExists and we carry on.
• Changing the interval env var does not take effect automatically: you must delete the schedule in Temporal (UI or CLI) and restart the worker. This is intentional — schedules are Temporal state, not k8s state.

A minimal cronjob (cmd/cronjobs/offchain-price-indexer/main.go):

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer cancel()

    err := temporal.RunCronjob(ctx, temporal.BuildMeta{...}, temporal.CronjobConfig{
        Name:            "offchain-price-indexer",  // → task queue, schedule ID
        IntervalEnv:     "PRICE_FETCH_INTERVAL",    // overrideable at runtime
        IntervalDefault: "5m",
        OpenDatabase:    postgres.PoolOpener(postgres.DefaultDBConfig(env.Get("DATABASE_URL", ...))),
        Setup:           setupRunner,               // returns temporal.Runner
    })
    if err != nil { slog.Error("fatal", "error", err); os.Exit(1) }
}

func setupRunner(ctx context.Context, deps temporal.Dependencies) (temporal.Runner, error) {
    // Build your service here; return a Runner whose Run(ctx) is the body
    // of one tick. Temporal wraps it in an activity + workflow for you.
}

Adding a new cronjob

1. Create cmd/cronjobs/<my-cronjob>/main.go using the skeleton above. Pick a sensible default interval — err towards longer, and always make it overrideable via an env var.
2. Put the business logic in internal/services/<my_cronjob>/ with full unit tests (mock external HTTP clients).
3. Add k8s manifests under k8s/base/<my-cronjob>/: deployment.yaml, serviceaccount.yaml, kustomization.yaml. Copy k8s/base/offchain-price-indexer/ as the template — cronjob Deployments are small (50m/64Mi requests) because the work happens inside Temporal activities. Register the service in k8s/overlays/{staging,prod}/kustomization.yaml (base dir under resources: only — the images: block is generated from k8s/image-roster.txt, so add a cronjob <name> <alias> roster line instead; ORB-362) and, for local kind runs, in k8s/overlays/dev/kustomization.yaml (add the base dir to resources: and a localhost/stl-<name>:local entry under images:).
4. Wire the Makefile. Image builds auto-discover via the CRONJOBS := ... glob, so a docker-build-cronjob-<name> target is already covered. Add the k8s Deployment name to CRONJOB_DEPLOYMENTS in stl-verify/Makefile so dev-up rolls it out and waits on it (this list is hand-maintained now that the manifest generator is retired; make check-dev-overlay-sync enforces it matches the dev overlay).
5. Infra PR for any new secrets/IAM + a Temporal namespace entry if needed.

Tip: As with workers, splitting the k8s-manifest and Infrastructure-repo changes into a follow-up PR keeps each review narrow and domain-scoped.

Cronjobs are idempotent by design — a tick may be retried by Temporal. Your service must tolerate running twice on the same window without producing duplicates. Model-output cronjobs are the deliberate exception: core-model-runner disables retries (maximum_attempts=1) because its rows are keyed by wall-clock computed_at, so a mid-run retry would append duplicates instead of colliding on them.

If you're writing the cronjob in Python

The flow is identical to the Go version — Temporal is still the scheduler, the worker registers a schedule on startup, and each tick runs one activity. Only the language changes. Same uv tooling rules as the worker section above.

The shared harness is app/adapters/temporal/ (client connect, schedule ensure, worker run), built with core-model-runner, the first Python cronjob. Reuse it rather than repeating the skeleton below — the skeleton is kept as an explanation of what the harness does for you.

Two things that only fail once deployed, both handled by the harness:

• The workflow module is re-imported in a sandbox, and numpy cannot load twice in one process. Keep the workflow free of the model stack: reference the activity by name and keep it in its own module, and re-export nothing from the package __init__.
• A CPU-bound tick must be a sync activity run on the worker's activity_executor. An async activity doing the work blocks the event loop; a sync one without an executor is rejected at worker startup.

Code skeleton — entry point at stl-verify/python/cli/cronjobs/<my_cronjob>/main.py. Uses the temporalio SDK:

import asyncio
import os
from concurrent.futures import ThreadPoolExecutor
from datetime import timedelta

from temporalio.client import (
    Client, Schedule, ScheduleActionStartWorkflow,
    ScheduleAlreadyRunningError, ScheduleIntervalSpec, ScheduleSpec,
)
from temporalio.worker import Worker

from app.config import load_config
from app.services.my_cronjob.service import MyCronjobService
# The workflow comes from its own module, never the package __init__ — the
# sandbox re-imports the workflow module, and anything the __init__ drags in
# (the service, numpy) comes with it.
from app.services.my_cronjob.workflow import TickWorkflow

NAME     = "my-cronjob"
INTERVAL = timedelta(minutes=int(os.getenv("MY_CRONJOB_INTERVAL_MIN", "15")))

async def run() -> None:
    cfg     = load_config()
    client  = await Client.connect(cfg.temporal_host, namespace=cfg.temporal_namespace)
    service = MyCronjobService(cfg)                 # wires DB + HTTP clients

    # Idempotent schedule creation — AlreadyExists is the normal path on every
    # restart after the first; the harness reconciles the interval into the
    # existing schedule there, so an interval change only needs a redeploy.
    try:
        await client.create_schedule(
            NAME,
            Schedule(
                action=ScheduleActionStartWorkflow(
                    TickWorkflow.run,
                    id=f"scheduled-{NAME}",
                    task_queue=NAME,
                ),
                spec=ScheduleSpec(intervals=[ScheduleIntervalSpec(every=INTERVAL)]),
            ),
        )
    except ScheduleAlreadyRunningError:
        pass  # see the harness's ensure_schedule for the reconcile

    # A sync (CPU-bound) activity needs an executor — Worker rejects it at
    # startup otherwise. One slot: one tick at a time.
    with ThreadPoolExecutor(max_workers=1) as executor:
        worker = Worker(
            client,
            task_queue=NAME,
            workflows=[TickWorkflow],
            activities=[service.tick],
            activity_executor=executor,
            max_concurrent_activities=1,
        )
        await worker.run()

if __name__ == "__main__":
    asyncio.run(run())

Run it locally with uv run python -m cli.cronjobs.<my_cronjob>.main (from stl-verify/python/).

Directory layout (mirrors Go; cli/ ≈ cmd/):

• stl-verify/python/cli/cronjobs/<my_cronjob>/main.py — entry point (above). No business logic.
• stl-verify/python/app/services/<my_cronjob>/ — business logic, the body of one tick, plus the TickWorkflow wrapper in its own workflow.py. Full unit tests; mock external HTTP clients.
• stl-verify/python/app/domain/entities/ — pure entities.
• stl-verify/python/app/ports/ — interfaces.
• stl-verify/python/app/adapters/{postgres,onchain,temporal,…}/ — concrete implementations, one subpackage per vendor.

Temporal contract (must match Go cronjobs exactly):

• Task queue, schedule ID, and workflow ID all derive from the cronjob name (lowercase, hyphenated).
• Interval read from <NAME>_INTERVAL env var with a sensible default — prefer longer, and always overrideable.
• Create the schedule on startup; on AlreadyExists, reconcile the interval into the existing schedule (both harnesses do this), so an interval change only needs a redeploy.
• The activity must be idempotent — Temporal retries. Guard every write against duplicates.

Adding a new Python cronjob:

1. Create stl-verify/python/cli/cronjobs/<my_cronjob>/main.py from the skeleton above.
2. Create the service in stl-verify/python/app/services/<my_cronjob>/ with unit tests (mock HTTP / repo adapters). Put the tick_workflow wrapper here or in app/adapters/temporal/ if it's generic.
3. Add ports/adapters for any new external client.
4. Add dependencies with uv — e.g. uv add temporalio (and httpx if you need a new HTTP client, though it's already in). Run from stl-verify/python/; updates pyproject.toml and uv.lock together.
5. Add migrations (see §11) for any new tables.
6. Dockerfile & Makefile targets — same situation as Python workers; the first Python cronjob introduces the shared plumbing (use the uv base image pattern already in stl-verify/python/Dockerfile).
7. Add k8s manifests under k8s/base/<my-cronjob>/. Copy k8s/base/offchain-price-indexer/ as the template (small: 50m/64Mi, because the work happens inside the activity). Register in k8s/overlays/{staging,prod}/kustomization.yaml and, for local kind, k8s/overlays/dev/kustomization.yaml (and add the deployment name to CRONJOB_DEPLOYMENTS in stl-verify/Makefile).
8. Infra PR for any new secrets / IAM + a Temporal namespace entry if needed.

Tip: Same as with workers — split the k8s-manifest and Infrastructure-repo changes into a follow-up PR.

Idempotency and migration rules above still apply.

---

10. Adding a new risk model

Risk models are Python and live under stl-verify/python/app/risk_engine/<model>/. Every model exposes the same contract: Required Risk Capital (RRC) in USD, computed at the model's documented default stress and overrideable via a scenario map. Many models are expected, and more than one may apply to the same (asset_id, prime_id) — the API returns all applicable results in one response.

Two families exist today and illustrate the range:

• Asset-level rating (risk_engine/suraf/) — pre-computed CRR per asset; RRC = usd_exposure × CRR. Default usd_exposure derived from prime_id.
• Position-level (risk_engine/crypto_lending/) — reads collateral state via ports; RRC computed under a default x% gap between liquidation trigger and execution.

Where code goes

app/risk_engine/<model>/          # Pure math. No I/O.
app/ports/                        # One `-er` interface per data dependency
app/adapters/{postgres,onchain}/  # One adapter per protocol variant
app/services/<model>_service.py   # Wraps math + ports; implements compute(...)
app/api/v1/risk.py                # The two routes below. Never import risk_engine/ here.

API contract — two routes cover every model

Both return { asset_id, prime_id?, results: [{ version, risk_model, rrc_usd, details }, ...] }. Each details is a discriminated union keyed on risk_model (the discriminator field is named risk_model to avoid collision with Pydantic's model_* protected namespace). Version is a reference to block_version and processing_version. These allow auditability of the results.

• GET /v1/risk/rrc?asset_id=…&prime_id=… — every applicable model at its defaults.
• POST /v1/risk/rrc/scenario with { asset_id, prime_id?, overrides: { <model>: {...knobs} } } — same, with per-model scenario overrides. Unknown models/keys → 422.

Model-side interface

Every service implements compute(asset_id, prime_id, overrides) -> RrcResult, with defaults and override schema documented in its docstring. A registry in app/services/ maps (asset_id, prime_id) to the set of applicable services — handlers iterate the registry, never branch on model.

Startup and tests

Anything loaded once (rating packages, mappings) goes in app/main.py::create_app before resources are acquired, so a bad config fails startup, not a request. Stash on app.state.<name>, expose via app/api/deps.py. Unit-test services with mocked ports; integration-test the endpoint against real Postgres (testcontainers) — mock only third-party APIs. Migrations for any new tables: see §11.

---

11. Database migrations

A migration is a single SQL file that makes one versioned change to the database schema (create or alter a table, add an index, backfill a column, etc.). The migrator tracks which files have already run, by checksum, and applies any new ones in order — so "adding a migration" is how you roll a schema change out to every environment reproducibly.

Migrations live in stl-verify/db/migrations/ and are applied automatically — by the migrator Job on make dev-up, and by the ArgoCD PreSync hook in staging/prod.

Rules (non-negotiable):

1. Filename: YYYYMMDD_HHMMSS_description.sql (use date +"%Y%m%d_%H%M%S").

2. Plain SQL only, ending with a self-tracking insert:

   INSERT INTO migrations (filename) VALUES ('20260420_120000_my_change.sql')
   ON CONFLICT (filename) DO NOTHING;

3. Never modify an applied migration. The migrator checksums every file; a changed checksum fails the deploy. To fix a mistake, write a new migration.

4. Create tables plain; partition later, if at all. A chunk interval, segmentby/orderby and a tiering horizon are bets on ingest rate and query shape — let measurement settle them. Convert when the numbers say to, in a new migration: that is where the compression and tiering policies land, and it re-checks that the hot reads prune chunks. A few tables stay plain permanently — see stl-verify/db/migrations/AGENTS.md.

   A plain table owes a row-growth alert plus a runbook section carrying its conversion path in the same PR (copy VectorUniswapV4AppendOnChangeGrowthHigh) — that alert is what notices it growing. Say in the COMMENT that it is plain and what would change that.

5. Append-only, and enforced by the database. No UPDATE, no DELETE, no INSERT … ON CONFLICT … DO UPDATE (a no-op SET still needs UPDATE privilege and still fails) on a converted table. Identity rows are written once; everything time-varying, lifecycle included, is a new row with the version tuple, and "the current value" is a query. A converted table's creating migration ends with REVOKE UPDATE, DELETE ON <table> FROM stl_readwrite;. The converted set is listed in stl-verify/db/migrations/AGENTS.md; the rest of the schema is being converted table by table, so the absence of a REVOKE on an old table is debt, not permission.

6. Use CREATE INDEX CONCURRENTLY on big tables. Test on staging first.

---

12. Testing

Command	What it runs	When you need it
make test	Unit tests	Everyday dev
make test-race	Unit tests with -race	What CI runs — run before pushing
make test-integration	Tests tagged integration (requires Docker — uses testcontainers)	Anything touching Postgres, Redis, SQS, S3
make e2e	End-to-end watcher tests	Changes to the live/backfill pipeline
make cover / make cover-all	Coverage report	When working on a service with a coverage goal

Philosophy:

• Services are the unit under test. Test only the public API; mock outbound ports with small handwritten fakes. Use table-driven tests. Services and main.go files should strive for 100% coverage on meaningful code paths. The only exceptions are branches that genuinely can't be exercised (generated code, OS-specific shims, truly unreachable error branches); each exception should be justified in the PR.
• Integration tests may only mock things we do not control. Alchemy and other third-party APIs: mock. Our Postgres, our Redis, our SQS: use the real thing (via testcontainers) — mocking them has bitten us in production.
• For main.go, extract a run(ctx, args) error function and call it from main() — then the integration test calls run directly.

Python code uses its own Makefile and uv. We use uv as the Python package manager and runner — not pip, pip-tools, or poetry. uv sync installs deps from uv.lock; uv add <pkg> adds a dep; uv run <cmd> runs anything under the project env.

Commands for stl-verify/python/ live in stl-verify/python/Makefile (each one wraps uv run), run from that directory:

Command	What it runs
make test-unit	Python unit tests (uv run pytest tests/unit)
make test-integration	Python integration tests (Docker needed)
make lint / make lint-fix	Ruff check / autofix
make run	Start the FastAPI server locally

The root stl-verify/Makefile targets in the table above only cover Go code.

---

13. Code conventions

Most of these are also spelled out in CLAUDE.md and stl-verify/AGENTS.md, which apply to humans too.

• Hexagonal boundaries. Domain has zero imports outside the standard library. services/ depends on ports/, never on adapters/. Adapters are dumb — no business logic.
• Interfaces use the -er suffix (Reader, Publisher, Multicaller).
• Constructors are NewFoo(...).
• Files are snake_case.go.
• Errors: wrap with context — fmt.Errorf("fetching block %d: %w", n, err). Never ignore an error. Prefer returning the error over continuing.
• Prefer the standard library. Pull in a dependency only when the stdlib equivalent is materially worse. De-duplicate by extracting a helper, not by copy-paste.
• Keep main() at the top of the file and small — it should read like prose. Extract helpers aggressively.
• No global state, no singletons. Inject everything through constructors.
• Binaries built with go build go into stl/dist/ (gitignored).

---

14. Pull request workflow

1. Branch off main. Name the branch after the Linear ticket if there is one (VEC-123-short-slug).

2. Open a PR early — drafts are fine. The CODEOWNERS file auto-requests review from @archon-research/stl-engineers.

3. Before you push, run:

   cd stl-verify
   make ci                  # lint + static + vuln + unit + race
   make test-integration    # if you touched anything data-adjacent

   CI runs the same commands. A CI run that modifies files (e.g. a stray go mod tidy diff) is a hard fail — commit the result locally.

4. PR title format: TICKET-1234: <good description> — e.g. VEC-123: Add SparkLend position tracker. The ticket prefix is how we link PRs to Linear and find things later; the description should say what the PR does, not what it is (prefer VEC-123: Backfill oracle prices for new Aave markets over VEC-123: Oracle changes). Commit messages can be whatever you like — do not bother squashing WIP commits locally, GitHub is configured for squash-and-merge, so the PR title becomes the commit message on main and your intermediate commits are discarded automatically.

5. Merge to main — CI then triggers .github/workflows/deploy.yaml, which regenerates the images: block of k8s/overlays/staging/kustomization.yaml from k8s/image-roster.txt and ArgoCD rolls the change into the vector namespace on the staging EKS cluster. Once staging is healthy, the same run promotes the images to the prod ECR, auto-commits the prod tag bump to main, and posts an "awaiting approval" message to Slack. Prod does not roll out until someone approves the production GitHub Environment review on that run; on approval the run syncs stl-prod in ArgoCD to the approved commit. Rejecting leaves the bump on main (it batches into the next approved deploy). The deploy runs whenever main holds code that has not been promoted yet — CI diffs each main push against the prod overlay's image tag, not just against the push itself — so a docs-only merge can carry an earlier merge's deploy, or retry a promotion that failed (ORB-361).

6. Adding a brand-new service image? Split it across two PRs. If one PR both introduces a new image (a new make docker-* target plus its k8s/image-roster.txt line) and the Deployment/CronJob that runs it, ArgoCD syncs the new manifest on merge before the image exists in ECR and before the deploy bot has rendered its images: entry: the pods sit in ImagePullBackOff until that run's stamp lands (see ORB-313). Instead:

   • PR 1 adds the build (Makefile target) and the roster line. The next deploy builds and pushes the image and writes its overlay entries.
   • PR 2 adds the k8s/base/... Deployment plus the resources: entry that references it — it renders against the entry PR 1's deploy wrote.

   This split is a recommendation, not an enforced rule. A combined PR still works: build-push-staging builds the new image in the same run before update-staging renders the block, and the staging health gate tolerates the brief first-rollout ImagePullBackOff. Splitting simply avoids that race. Never hand-add an images: entry (the old "copy a sibling's tag" placeholder): the Manifests check rejects anything the roster does not render, and the bot rewrites the block wholesale on every deploy (ORB-362). The one sanctioned hand edit is removal: dropping or re-homing an image means deleting its stale entry in the same PR as the roster change. (A re-home leaves the alias unpinned in the merge commit until the post-merge rewrite — the same brief ImagePullBackOff window as a first rollout.) scripts/deploy/verify-ecr-images.sh (run before each stamp) remains the backstop for an image that was never built — e.g. a roster line without a matching Makefile release target. It fails the deploy with an explicit missing-image list instead of letting a silent prod ImagePullBackOff through.

---

15. Things that will save you time

• Read the Makefile directly — stl-verify/Makefile is the source of truth for every workflow and has many more targets than are documented here (Erigon management, bulk downloads, stress tests, bastion tunnels, …). Grep the top of the file or search for the target name you're after; there is no make help.
• Look at existing examples before inventing a pattern. Every category (workers/, cronjobs/, backfillers/) has a reference implementation that wires the adapters in the approved way. Copy it.
• Secrets never go in git. Local secrets live in .env.secrets at the repo root (gitignored, created by make dev-preflight). Cloud secrets live in AWS Secrets Manager and are pulled by make dev-env.
• If you're stuck on infra, contact the vector team.

---

16. Getting help

• Code questions / design review: @archon-research/stl-engineers on GitHub, or #proj-verify-beacon on Laniakea Slack (review is required anyway — ask early).
• Protocol specs: see docs/ — aave_v3_spec.md, morpho_spec.md, sparklend_spec.md, etc.
• Architecture decisions: docs/adr/ holds the short record of why we chose kind over Minikube, why every row is versioned, and so on.
• Anything else / general questions: #proj-verify-beacon on Laniakea Slack.

Thanks for contributing — ship data-correct code and we'll all sleep better.