Skip to content

Latest commit

 

History

History
59 lines (31 loc) · 12.8 KB

operations.md

File metadata and controls

59 lines (31 loc) · 12.8 KB

Operations

Scanner publication

The Scan IPSW and Scan XIP workflows are manual, self-hosted operations. A run validates dispatch metadata, downloads or reuses the archive, verifies its format, scans it, validates the complete corpus, and packages exactly one detail file plus the updated index. A separate hosted job accepts only that artifact layout, validates it again, then mints a short-lived GitHub App token and opens a signed data pull request.

Repository helpers use the version in .ruby-version. Hosted Ruby jobs use the pinned ruby/setup-ruby action on Ubuntu 24.04; the self-hosted scanner uses Homebrew Ruby at /opt/homebrew/opt/ruby/bin and requires that version or a newer patch in the same major.minor series. Each scanner job checks the version and JSON/Fiddle capabilities before archive work. For a patch upgrade, upgrade and verify the scanner host first, then merge the .ruby-version bump. For a minor or major upgrade, pause scanner dispatches and wait for active scanner runs to finish, update and verify the host and repository pin together, then resume dispatches once both use the new series. The version gate deliberately rejects either side of an incomplete series upgrade.

Shared JSON validation rejects comments, duplicate keys, malformed Unicode, and nonfinite numbers. Bundled JSON and Fiddle libraries provide parsing and descriptor-relative file access without installing gems in the publisher. Publication artifacts must contain one gzip stream; concatenated streams, invalid checksums, and trailing non-padding tar data are rejected. bundle install is needed only for the local and CI Minitest suite. Node-based site checks and deployment helpers use the Astro/Wrangler runtime, and the fleet-managed Codecov uploader uses Python.

The IPSW and XIP archive caches are persistent. An existing SHA-256 sidecar is verified before scanning. An IPSW without a sidecar, including a new download before it enters the cache, must record the dispatched version and build in its own BuildManifest.plist or Restore.plist; its filename is not evidence. A mismatched cached IPSW is preserved for investigation, and a mismatched new download is removed. A new sidecar is created and the archive files are made immutable only after the scan, corpus validation, and dispatch identity checks succeed. Hashing a new download alone does not establish that it can be expanded or scanned. A cached file whose date, format, checksum, or identity disagrees with the dispatch is an error, not an automatic replacement. AEA WKMS 404 responses intentionally require a fresh dispatch.

Dispatch only from main, after verifying the Apple source URL, product version, build number, release date, and prerelease flags. Auto-merge relies on the independent hosted artifact validation and required CI checks. Inspect the resulting data diff and completed run when investigating unexpected catalog changes.

Pending catalog publications

The three scanner workflows share the scan queue. Before allocating the scanner, each hosted prepare job also checks every page of open PRs for same-repository Starhaven Bot catalog publications. An open scan or rescan PR stops preparation, including after an earlier run's 30-minute merge wait has timed out. A failed or incomplete API response also stops preparation. After that check, the job refreshes main before recording the scan's base, so a PR merged during preparation is included in the next index.

Resolve and merge the earlier PR, or close it if its output should be discarded, then start a fresh dispatch. A closed, unmerged publication is not reusable. Keep the existing bounded merge wait and exact-base artifact verification; do not bypass approval or publish a later scan's stale index to clear the queue. This guard covers the normal bot publication paths serialized by these workflows; manual catalog changes still require review of their index contents.

Rescanning a published release

After a scanner fix, dispatch Rescan Release from main with the product, version, and build of each affected release. The hosted prepare job reads the release date, prerelease flags, device flag, and source URL from the published detail on main, so a rescan reproduces the published metadata and changes only scanner output. The self-hosted job never downloads: it requires the cached archive and its checksum sidecar as regular files, verifies the checksum and the release-date modification time, reuses a saved AEA key when present, and writes nothing to the archive drive. The publisher verifies the artifact in replace mode, which accepts only a change to that one release, and opens a fix(data): rescan pull request. A rescan that reproduces the published data opens no pull request. Publication branches are unique to each dispatch. A retry reuses its branch or pull request only when the signed commit has the recorded base as its sole parent and exactly matches the verified artifact overlay; unexpected state stops publication. A closed, unmerged pull request requires a fresh dispatch. If overlaying the index fails, replace mode restores the original detail bytes. Metadata corrections such as promoting a release candidate are ordinary reviewed data pull requests.

Accepted archive download risk

The repository maintainer accepts the two LOW Pinprick runtime-fetch findings in the Download IPSW and Download XIP steps, initially reviewed on 2026-09-10 and reviewed again on 2026-10-05 for explicit token scoping and deferring new archive checksums and locks until scanning and output validation succeed. The 2026-10-06 candidate workflows add a read-only pending-publication gate and refresh the trusted base; archive download and execution behavior is unchanged. The 2026-10-08 candidate Scan IPSW workflow rejects source URLs with a query or fragment and confirms an unchecksummed IPSW's recorded version and build before caching or scanning it. Fleet canon in fleet/repos/macOSdb.yml owns the complete generated .pinprick.toml; changes require hub review and release/sync. The required fleet guard rejects consumer edits to this policy. .pinprick.toml binds each acceptance to its exact logical curl command, finding identity, and a workflow context fingerprint. The fingerprint lets full SHA pin and SemVer comment updates through only for the six listed actions/* steps (checkout, cache restore and save, artifact upload and download, and GitHub App token creation). Updated revisions are still audited, and every other workflow byte, including permissions, runners, environments, triggers, input validation, commands, action inputs, and ordering, stays bound to the acceptance. This is an explicit risk acceptance, not a claim that IPSW or XIP archives cannot contain executable code. The scanner workflows set up Ruby through .github/actions/setup-ruby, so ruby/setup-ruby updates change that action instead of the accepted workflow bytes. Reassess after changes to runner isolation, Apple URL validation, archive paths, download options, archive validation, or scanner execution behavior; remove the acceptance if independently anchored publisher verification becomes available and can be enforced.

Routine updates to the listed actions need no policy change. For any other workflow change, review the candidate and regenerate its fingerprint with pinprick workflow-fingerprint, using the ignore-action-revisions list from fleet canon. Deliver the candidate entry alongside the current one through fleet release/sync before merging the workflow change, then retire the previous entry once the change reaches main. Changes to separately loaded source or hosted controls need their own review.

The scanner needs runtime-selected Apple releases and persistent per-version .part paths for resumable downloads. IPSW input validation restricts the source to HTTPS on updates.cdn-apple.com and a version/build filename, without a query or fragment; redirects remain HTTPS but are not host-restricted. XIP derives its CDN URL from a validated path under adcdownload.apple.com, does not follow redirects, and sends its ADC cookie only to that source. Both workflows reject archive-path symlinks, wait for curl to succeed before renaming the partial file, check archive size and format, and pass archives to the scanner as untrusted input. The scanner does not execute extracted binaries to discover their versions; XIP expansion uses Apple's system xip tool and propagates failure.

Residual risks include trusting Apple's delivery service and TLS, archive-parser defects, and the integrity of the self-hosted scanner environment. Locally created SHA-256 sidecars detect later cache changes; they do not independently authenticate the original download. Repository code does not prove hosted runner-group or environment protections. No global host, extension, action, severity, or rule exemption is used, and incomplete audit coverage remains fatal.

CI, scheduled, and local link checks resolve this site's production origin against the built site, because new data PRs create pages that do not exist in production yet. The 404 route resolves to its generated HTML file; missing built pages still fail. External online links retain their existing checks.

CLI release

  1. Change MacosdbVersion.current in Sources/macosdb/Version.swift on a feature branch.
  2. Open and merge a signed chore: bump version to X.Y.Z pull request.
  3. Dispatch the Release workflow from main.

The workflow builds before loading signing credentials, imports the Developer ID certificate into an isolated keychain, signs and notarizes the binary, verifies its signature and notarization ticket online, creates a tarball and SHA-256 sidecar, and generates build provenance. It creates an immutable version tag, creates or resumes the GitHub release, and opens a Homebrew cask bump.

The cask publisher adds the commit author's DCO sign-off before Git creates the commit, without disabling existing hooks. If hook preparation stops the bump, inspect the preexisting prepare-commit-msg hook or inherited core.hooksPath; the workflow preserves both and refuses hook locations outside the fresh tap checkout.

A retry is safe only when the existing tag points to the dispatched commit. Existing archive and checksum assets are reused only as a complete pair after checksum and build-provenance verification. A tag at any other commit or a partial asset pair requires investigation; do not delete or move published release state.

Website deployment

Changes under site/ or data/, or to the deployment workflow and its validation scripts, deploy from main to Cloudflare Workers. Manual production dispatches fail outside main. Rejected dispatches use separate concurrency groups so they cannot cancel production work. CI checks the install-script policy before a clean dependency install, then runs formatting and type checks, unit tests, production build, and deployment dry-run. The production workflow revalidates the release corpus and repeats the clean install and build in a job without deployment credentials. A separate cloudflare environment job installs dependencies without building, downloads the build output, and deploys it with site/scripts/deploy.mjs. That script derives the Wrangler configuration from the checked-in site/wrangler.json, ignores the generated build configuration, and rejects non-regular files in the build output. The workflow verifies the live security headers after publishing.

Dependency vulnerability monitoring runs through Dependabot alerts and security updates. Full-tree vulnerability audits are kept out of CI and deployment gates so new advisories cannot block publishing an unchanged lockfile. Run npm --prefix site run audit when investigating dependency findings.

Repository automation assumes that GitHub environments restrict scanner, release, and Cloudflare secrets to the main branch; self-hosted scanner runners are isolated, patched, and restricted to the scanner workflows at refs/heads/main; main requires the aggregate CI conclusion; and tag/release mutation is limited to the release workflow. These are control-plane settings and should be audited in GitHub and Cloudflare after changing administrators, apps, runner groups, or environments.

Local recovery

When a scan cannot detach a disk image, including after an interrupted or timed-out attach, it prints a warning and keeps its workspace and ownership marker so the image stays discoverable. macosdb cleanup lists recognized stale scanner mounts, including images attached without a mounted volume, and stale workspaces. Inspect the dry-run, then use macosdb cleanup --force. Unrecognized directories and live markers are deliberately preserved. Cleanup stops if mount discovery fails or a scanner mount cannot be detached; removal failures return a nonzero exit status. Never substitute a broad manual rm -rf over the system temporary directory.