Skip to content

testing.md

Latest commit

 

History

History
163 lines (123 loc) · 10.3 KB

testing.md

File metadata and controls

163 lines (123 loc) · 10.3 KB
Raw

Continual Testing

The installer should not be validated against a developer laptop as the primary signal. A developer machine usually already has Node, npm, shells, PATH entries, app configs, and cached packages.

Local Tests

Fast local check:

npm test

The initial migration keeps explicit noImplicitAny and strictNullChecks exceptions for legacy CommonJS module boundaries. Other strict checks remain enabled; remove these two exceptions incrementally as those contracts are typed.

This runs:

  • TypeScript 7 type-checking and JavaScript emission.
  • Source, emitted application, and packaging scaffold validation.
  • npm package-age verification against the supply-chain cutoff.
  • Managed secure skills packaging, ownership, migration, collision, and failure-safety tests.
  • A clean-home dry run that writes all configs into a temp directory and injects fake command execution.

Runtime bootstrap check:

npm run test:clean-runtime

This creates a temp home directory, strips PATH down to OS basics, downloads the pinned Node.js runtime, verifies the checksum, extracts it, and proves the private node and npm run.

What This Catches

  • Accidental writes to the real user home.
  • Missing clean-machine folders.
  • Tool installs that are not pinned.
  • Missing npm --before cutoff.
  • Config files that stop being generated.
  • Private Node/npm bootstrap regressions.
  • Missing, malformed, wrong-target, truncated, or checksum-mismatched bundled Node.js runtime payloads.
  • Insufficient disk capacity, including the payload-expansion, staging, and rollback reserve, before any managed installation directory is created.
  • Interrupted development-runtime downloads that would otherwise leave a reusable partial cache file.
  • macOS launchers that capture the installing account's literal home or fail the whole install when a standard account cannot write /Applications.
  • Regressions where TritonAI Harness keeps a stale legacy provider-status cache.
  • Regressions where the installer uses a random system codex instead of the managed UCSD Codex path.
  • Regressions where the installer exports its managed PATH or CODEX_HOME through shell profiles, Windows user variables, or its own process environment, including upgrades from the legacy global environment behavior.
  • Regressions where the managed Codex backend is missing, stale, copied from the wrong packaged payload, or falls back to npm without the package-age cutoff.
  • Regressions where Installer upgrades overwrite public/user skills, retain retired managed secure skills, accept malformed manifests, or lose the prior secure bundle after a staged copy failure.
  • Regressions where stable plugin packaging drifts from the reviewed catalog's source, package set, package digests, or manifest digests; accepts policy-looking fields that are not enforced; or accepts a noncanonical origin, moving or mismatched ref/commit, implicit dirty checkout, ambiguous package selection, source/tests, symlinks, special files, unsafe paths, or manifest/package/file drift.
  • Regressions where a Harness artifact omits or changes the exact managed plugin composition proof, or a packaged Installer omits that proof on macOS, Windows Setup, or portable paths.
  • Missing or publisher-mismatched macOS Harness signatures at both build-time vendoring and runtime activation, while allowing Developer ID certificate renewal under the pinned Team ID and bundle identifier.
  • Missing, malformed, caller-selected, or artifact-mismatched Windows Harness trust policies; unsigned policy must skip only publisher checks while retaining release hash, version, composition, upgrade, and packaged-boot gates.
  • Missing, invalid, publisher-mismatched, or stale Authenticode verification in the future signed Windows lane, including independent verification during the final signed release contract.
  • Missing, invalid, publisher-mismatched, or untimestamped nested Harness Authenticode signatures when the bound policy is signed, at Windows vendoring time, immediately before NSIS execution, and on the installed Harness executable.
  • Final macOS DMGs, Windows Setup executables, or Windows portable executables that are signed but cannot actually render and remain healthy as packaged applications, plus stale boot evidence copied from different candidate bytes.
  • Accidental window close or ordinary quit while an install is mutating state, duplicate concurrent install requests, renderer progress-channel failures that would otherwise abort setup, and support-report failures that would otherwise mask the real install outcome.
  • A second Installer process bypassing the per-user Electron lock, truncated environment/profile/migration/launcher writes, non-atomic Windows shortcut replacement, or interrupted Node, Codex, managed Harness, launcher, and secure-skill swaps that cannot deterministically commit or restore on retry.
  • Support reports that lose stable component attribution for startup, prerequisite, TritonAI service, managed Codex, Harness desktop, or finalization failures.
  • Managed command/version probes, Windows environment discovery/cleanup, desktop native helpers, and direct or PowerShell Windows installer paths that hang indefinitely, omit their timeout, retry an ambiguously timed-out native installer, or fall through to an unowned cmd.exe start process.
  • Command trees or Node extractors that report timeout before confirmed termination, Windows smoke candidates that write a marker but hang or exit nonzero, and signed-policy Harness executables probed before Authenticode publisher verification.
  • Recovery downloads that trickle forever, exceed the manifest size ceiling, loop through redirects, poison a verified cache target, or leave partial files after failure.

Legacy Provider Stale-State Reproduction

Do not use the developer laptop home directory as the primary reproduction. The clean dry-run test creates a temp home, seeds the failure shape, and verifies the installer behavior without touching ~/.tritonai-harness or ~/.agents.

The dry run now covers:

  • A stale provider cache at .tritonai-harness/caches/legacy-provider.json reporting version: 1.4.3 and status: error, which should be cleared during the Codex migration.
  • A managed Codex binary that reports 0.140.0 and should be upgraded.
  • A current managed Codex binary that reports 0.151.0 and should not reinstall.
  • A packaged Codex payload that should be copied instead of running npm during a normal packaged install.
  • A missing packaged Codex payload that should use the pinned npm fallback only for development/unpackaged runs.
  • A separate system codex outside the versioned managed prefix, which must not become TritonAI Harness's selected backend.
  • macOS and Windows settings, including the managed codex or codex.cmd path and ~/.tritonai-harness/codex home.

Run the reproduction with:

npm run test:clean-dry-run

CI Shape

Run the same checks on hosted clean machines:

  • macos-latest: npm ci, npm test, npm run test:clean-runtime
  • windows-latest: npm ci, npm test, npm run test:clean-runtime

The unit suite proves packaged runtime identity and transactional activation without live downloads. The clean-runtime test intentionally exercises the network fallback in a temporary home, so it is safe for CI and does not depend on whatever the runner image already has installed.

Release Gate

Release validation includes a mandatory non-destructive native boot gate. macOS packaging verifies the final mounted DMG app directly. Windows packaging may run on macOS/Wine, so it records hashes and validates PE structure without claiming native execution; npm run verify:win-installer:native must then run against those exact, unchanged bytes on a clean Windows host. The final macOS app and the exact Windows portable and Setup-installed apps must render a visible window, complete renderer initialization, remain responsive for five seconds, and exit cleanly. The Windows gate also refuses to replace a pre-existing Installer and proves its temporary Setup installation was removed. npm run release:contract rejects missing, stale, cross-version, or hash-mismatched boot proofs. This gate intentionally does not accept source, unpacked-app, signature-only, or copied marker evidence.

The native boot gate does not replace a full end-to-end install test because it never accepts an API key or mutates managed TritonAI state.

Before distributing a signed installer, run one manual VM test per platform:

  1. Start from a fresh macOS or Windows VM snapshot.
  2. Install nothing manually.
  3. Run the packaged installer.
  4. Confirm ~/.agents/ucsd and ~/.tritonai-harness are created.
  5. Confirm TritonAI Harness launches from the desktop/application launcher.
  6. Confirm configs point at UCSD/TritonAI and do not use default vendor endpoints.
  7. Revert the VM snapshot.

UTM macOS VM Handoff

Keep UCSD Installer Clean macOS as the stopped baseline. For each installer test, clone it and run the clone:

utmctl clone "UCSD Installer Clean macOS" --name "UCSD Installer Test Run"
utmctl start "UCSD Installer Test Run"

Build the Developer ID signed and notarized macOS release DMG, then serve only that release artifact from the host Mac:

npm run package:mac-release
npm run serve:mac-release

The serve command validates the stapled DMG before opening the HTTP server and prints the URL to use from the VM. In the VM, download the installer from Safari using that printed URL, for example:

http://192.168.64.1:8790/TritonAI-Installer-0.1.0-arm64.dmg

If Safari searches instead of downloading, use Terminal in the VM with the printed URL:

cd ~/Desktop
curl -fL -o TritonAI-Installer.dmg \
  http://192.168.64.1:8790/TritonAI-Installer-0.1.0-arm64.dmg
open TritonAI-Installer.dmg

Do not hand off dist/mac-arm64, ad-hoc builds, copied .app bundles, or artifacts/macos-share to the VM. Clean macOS downloads should use the signed and notarized DMG under artifacts/macos-release/ only.

UTM's Shared Folder button opens the host-side folder picker. It is useful if the guest mounts the shared folder, but this macOS Apple Virtualization VM did not expose the selected folder under /Volumes during testing. The HTTP handoff above is the reliable path.

After the run, stop and delete only the disposable clone:

utmctl stop "UCSD Installer Test Run"
utmctl delete "UCSD Installer Test Run"