validation-engine.md

Validation engine

ChaptarrNG keeps dotnet test and its disposable PostgreSQL migration smoke as the native test runners. The engine records each runner's source snapshot, discovered test inventory, native case totals, platform identity, workflow attempt, and private task logs. A final GitHub job reconciles the Linux, Windows, macOS, and PostgreSQL smoke receipts before the workflow is complete.

GitHub runner routing

Every platform matrix leg runs on GitHub-hosted runners. The Linux leg uses ubuntu-latest; Windows and macOS keep their corresponding hosted runners.

Local use

Use Node.js 26.12.1 and .NET SDK 10.0.402, matching the CI toolchain. The root global.json pins that exact SDK with roll-forward disabled, so a newer SDK already present on a hosted runner cannot silently change the compiler or validation runtime. Build the existing CI outputs first:

ImageSharp 4.1.3 requires the complete Six Labors license at build time. Store it as a single shell-quoted SIXLABORS_LICENSE_KEY assignment in the ignored .env.sixlabors file; use the full contents of sixlabors.lic, not only its Key field. Do not commit either file. Source it before restore/build commands and unset the variable before starting the validation engine, whose child processes intentionally do not inherit credential-shaped environment values. GitHub Actions reads the same value from the private SIXLABORS_LICENSE_KEY repository secret. Fork pull request workflows cannot read repository secrets; those builds need a locally supplied license or maintainer validation from a reviewed in-repository branch. See the ImageSharp build-license guide for the licensing rationale, application steps, and full local, CI, and container setup instructions.

yarn install --frozen-lockfile
yarn typecheck
yarn build
set -a
. ./.env.sixlabors
set +a
dotnet restore src/Chaptarr.NoTests.sln
dotnet restore src/Chaptarr.Core.Test/Chaptarr.Core.Test.csproj
dotnet build src/Chaptarr.NoTests.sln --configuration Release --no-restore
dotnet build src/Chaptarr.Core.Test/Chaptarr.Core.Test.csproj --configuration Release --no-restore -p:UseSharedCompilation=false
unset SIXLABORS_LICENSE_KEY
if [ "$(uname -s)" = Linux ]; then
  export PLAYWRIGHT_BROWSERS_PATH="$(mktemp -d "${TMPDIR:-/tmp}/chaptarr-playwright.XXXXXX")"
  pwsh ./_tests/net10.0/playwright.ps1 install --with-deps chromium
else
  export PLAYWRIGHT_BROWSERS_PATH="$HOME/.cache/ms-playwright"
  pwsh ./_tests/net10.0/playwright.ps1 install chromium
fi
TRUSTED_LAUNCHER=/path/to/trusted-test-launcher.mjs
node "$TRUSTED_LAUNCHER" "$PWD" --playwright-cache "$PLAYWRIGHT_BROWSERS_PATH" -- node scripts/validation-engine/engine.mjs --plan --tests-only --json
node "$TRUSTED_LAUNCHER" "$PWD" --playwright-cache "$PLAYWRIGHT_BROWSERS_PATH" -- node scripts/validation-engine/engine.mjs --tests-only

Local execution of test tasks requires the trusted Linux bubblewrap launcher. For a branch you have not reviewed, copy trusted-test-launcher.mjs and trusted-test-sandbox-policy.mjs from the same reviewed revision to a location outside the checkout, keeping them together. The launcher snapshots only paths in the Git index, hides the original checkout and common host configuration and data roots, and runs candidate code offline. Dotted and extensionless credential names, private/API keys, access/refresh-token names, client-secret names, and credential material with source-like extensions are excluded. Source modules under credential folders remain available unless their own names identify credential material. Stage new candidate files first. Suffixed dotenv credentials such as tokens.env.local are excluded, while example templates such as tokens.env.example remain available. It opens each workspace, run-data, dependency, cache, and marker source before the sandbox masks host paths, then binds those pinned sources by inherited file descriptor. Each persisted task log is capped at 16 MiB and ends with a marker when output was truncated.

node /path/to/trusted-test-launcher.mjs "$PWD" --playwright-cache "$PLAYWRIGHT_BROWSERS_PATH" -- node scripts/validation-engine/engine.mjs --tests-only

The optional Playwright cache is mounted read-only. Run planning for an unreviewed candidate through the same launcher because planning imports its validation configuration. GitHub Actions remains responsible for hosted validation.

The engine-contract command caps Node's test-file workers at two, independent of the available CPU count. Local engine-contract runs can sit beside the Core NUnit suite because their logs, NUnit results, and task environments are separate. PostgreSQL smoke tests run on GitHub-hosted Linux and use only the workflow's disposable databases. Tasks that share writable output must use the same concurrency group so the scheduler runs them in separate batches.

During a run, the engine writes a compact task summary to stderr when each task starts and finishes, then once per minute while a task is still running. The counts cover completed tasks; every active task and its elapsed time are shown separately. GitHub renders GITHUB_STEP_SUMMARY only after its workflow step finishes, so these updates appear in the live step log instead.

For BuildKit container builds, source the same file and pass the value as a BuildKit secret, for example:

docker buildx build --secret id=SIXLABORS_LICENSE_KEY,env=SIXLABORS_LICENSE_KEY -f Dockerfile.cross .

The Dockerfiles consume the secret only in the restore/publish instruction; it does not enter the build context, image layers, or image metadata.

The Core task runs the existing NUnit project with --no-build and --no-restore, so its receipt refers to the test assembly built from that same checkout. Linux direct-download tests need the Playwright Chromium cache. Hosted CI installs it with system dependencies in a run-specific cache and passes only that cache path to the NUnit task. Local Linux runs should set PLAYWRIGHT_BROWSERS_PATH for both browser installation and the trusted launcher; the launcher mounts the cache read-only. The engine passes the cache path only to the Core task and keeps its per-task home, temporary, XDG, and environment settings private. The PostgreSQL task is GitHub-only and assigned to the dedicated PostgreSQL node: CI supplies three throwaway databases from its PostgreSQL service, while ordinary platform nodes select only the NUnit task.

Receipt contract

Each task requires a nonzero native case count. NUnit XML provides total, passed, failed, skipped, and inconclusive counts; the Node test contracts use the standard TAP summary with the same zero-case and accounting checks. The migration smoke emits one case only after the app starts against the workflow's ephemeral PostgreSQL service and /ping responds. If startup fails, the receipt instead records a bounded diagnostic with the stage, reason, readiness timeout, process exit state, and up to eight exception type names and stack-frame identifiers. It omits exception messages, method arguments, and filesystem paths. Frame order and repeats are retained so a recursive startup path remains visible. NUnit receipts also include at most 100 failed test-case names, exception types, allowlisted method names, and (when available) the positive integer source line number. They omit test stdout, assertion messages, test values, and source file paths, so platform failures can be narrowed down from the short-lived CI artifact without exposing failure payloads.

The source fingerprint binds the commit, tree, working diff, configured build inputs, and test-source inventory. Text inputs and inventory files use Git's CRLF-to-LF normalization so Windows checkouts produce the same fingerprint as Linux and macOS. Each platform receipt separately hashes its compiled test assembly byte-for-byte; platform binaries are allowed to differ. Before and after each concurrent task batch the engine checks that the source fingerprint has not moved and blocks successful receipts if it changes. Logs stay private under the system temporary directory; the exported receipt carries counts, bounded failing test names, exception types, allowlisted method names, and safe source line numbers. For SQLite failures it also records only the numeric primary error code. Exception messages, paths, source values, and assertion contents stay out of receipts. Overlapping runs in one checkout are rejected. Runtime probes and task child processes receive an explicit environment allowlist and each task's declared settings. Every task gets run-owned home, temporary, XDG, and .NET CLI directories. NuGet packages use a private cache shared only within the validation run. Test tasks run with CI=true and NODE_ENV=test regardless of the invoking shell's values. For Linux tests that use Chromium or start a nested trusted sandbox, the engine creates a short private symlink under /dev/shm that points to the isolated task temp directory. This keeps UNIX-domain socket paths within the kernel limit without depending on shared /tmp; temporary data still lands in the task directory.

The artifact downloader preserves each runner receipt in its own subdirectory, and the reconciler finds receipt.json recursively so same-named artifacts do not overwrite one another. It requires one passing receipt for each expected node, from the same workflow run, attempt, commit, source fingerprint, and inventory. It rejects duplicate or missing platform nodes, test tasks with zero cases, runner identity mismatches, and stale artifacts. The CI build, frontend build, artifact publication, and release-note checks remain separate workflow gates.