combo-verify: hand the matrix slot numbers, never the roster — the dropped job output created no verify job

[rbuergi]

Refs #3848

The defect

combo-verify.yml's preflight published the derived roster as the job output instances (and the source map as sources), and the verify matrix was fromJSON(needs.preflight.outputs.instances).

The roster names installations that a client estate keeps private. The derivation registers those names as log masks, and the runner drops any job output that contains a masked value: the whole output, as a warning, inside a job that concludes success:

##[warning]Skip output 'instances' since it may contain secret.
Set output 'count'
##[warning]Skip output 'sources' since it may contain secret.

The matrix received an empty string, could not expand, and no per-instance job was ever created. Every post-CD run read preflight=success verify=failure instances=6 with no instance contacted and no reason printed. The only green runs were the "CD cancelled, no candidate" branch.

The change

Design choice: each verify job re-derives its own roster row, instead of receiving the roster as an artifact (per maintainer standing instruction to follow recommendations). The issue's 10-10 comment names both routes. An artifact was rejected because this repository is public: an artifact can be downloaded, and its name and the file names inside it are not masked.

• Preflight hands over three values that cannot contain a name: slots ([0,1,…,count-1]), count, and digest. The digest is a sha256 over roster and source map, salted with the private roster, written as single digits joined by underscores, so no identifier or numeric secret can be a substring of it.

• Each verify job resolves its slot itself. It runs the same script the preflight ran (derive-combo-instances.py --discover --slot N --expect-count … --expect-digest …), takes the row at its slot, and writes name, base URL and sources to $GITHUB_ENV. It exits 1 unless it derived the preflight's count and digest, so a fleet that changed between the two jobs is a red naming the drift.

• Nothing unmasked carries a name. The job is Verify instance slot N against its roll target; the work-root artifact and the lander's files are named slot-N. Before this, a job name built from matrix.instance.name would have published a private name the first time the matrix expanded.

• The verdict counts jobs. needs.verify.result is one word for the whole matrix. The verdict job now lists the run's jobs over REST (actions: read), counts those matching the verify job's exact name, and fails unless created == succeeded == the preflight's count. Zero created has its own message naming the unexpanded matrix. No continue-on-error, no input-shaped if:.

• A private roster row prints the verdict and counts only, and uploads nothing. Safe default, pending the maintainer's word. A failed verification used to upload the instance's combo.json (its module list) from this public repository. The slot step now writes INSTANCE_PRIVATE for a row the private roster names by instance id, host or declaring repository. For such a row the lander prints no name, host, response body or module id and withholds the verifier's output; the artifact upload runs only when INSTANCE_PRIVATE is literally false, so an unset value uploads nothing. The estate's own instances keep the upload and the full log. The verdict is still landed on the instance in full.

No skip-trapdoor was added: verify is still gated by needs: preflight alone, and the slot step and the job count are unconditional.

Tests and guards

check-combo-verify.py (runs on every platform PR, because this workflow never runs on one):

• Structure: preflight outputs are exactly slots/count/digest; the matrix is built from slots; nothing in verify reads a name from the matrix or a preflight output; the slot is resolved before the lander with count and digest passed; the verdict counts jobs unconditionally.
• Roster step, executed: every value it writes to $GITHUB_OUTPUT is put through the runner's own rule. A value containing any name, host or source of the roster fails.
• Verdict step, executed over the job counts, including the measured case (6 derived, 0 jobs created), a partial matrix, a missing count, and the no-candidate branch.
• --self-test: re-publishing the roster, building the matrix from it, naming the job or artifact after the installation, removing the slot step or its digest, removing or softening the job count, re-enabling the upload for a private row, and a verdict that never counts are each refused.

derive-combo-instances.py --self-test drives the slot resolution: the right row, and a red for an out-of-range slot, a missing count or digest, a roster that changed size, a moved host, and a changed source map.

check	result
derive-combo-instances.py --self-test	exit 0, 39 PASS
check-combo-verify.py --root .	exit 0, 36 PASS, 0 violations
check-combo-verify.py --self-test	exit 0, every regression refused
negative control: the new guard over origin/main's workflow	exit 1, 14 structural errors
live, public deployments repository only: derive, then --slot 0..3 with the derived count and digest	each slot resolved its row; a wrong digest exits 1 and writes nothing
the real lander driven end to end with stub curl/dotnet, as a private row (flag unset and true) and as a public one	private: 0 occurrences of name, host or module id in log and summary; public: full detail
check-workflow-timeouts, -shell, -yaml-keys, -permission-pairing, check-guard-step-masking, check-pr-secret-preflight, check-reusable-workflow-runners	exit 0
actionlint, shellcheck	exit 0
dotnet build src/MeshWeaver.Documentation -c Release -warnaserror	0 warnings, 0 errors

Not run: the workflow itself. It triggers only on workflow_run and workflow_dispatch, so its first real execution is the first CD after this merges. The full six-instance derivation needs the fleet-reader App token and the private roster, which I do not hold.

Doc page: Doc/Architecture/ComboGateWiring → "The roster never crosses a job boundary — only slot numbers do".

Still owed, and not something code can do

Once the matrix expands, each instance's verify job reaches that instance as the run's own OIDC identity. An instance verifies only when it holds both halves below (from the issue's 2026-10-09 comment). An instance missing either one shows a 401 in its own slot's job, naming both.

1. The audience config, Plugins__Registry__BuildPrincipalAudience = the instance's base URL. It is declared for memex, memex-cloud, build and client-b (Systemorph/Memex#698) and reaches each portal at its next Reconcile or Roll. The two client-estate instances in the roster need the same key in that estate's own deployments repository; that is the client estate's call.
2. The grant node Admin/_BuildPrincipal/systemorph--meshweaver, created by a global admin on each instance. It exists on memex.systemorph.com; build, memex-cloud and client-b still need it, and so do the client-estate instances if they stay in the roster:
   • repository: Systemorph/MeshWeaver, repositoryId: 743606761, repositoryOwnerId: 77832550
   • events: {workflow_run: [verify], workflow_dispatch: [verify]}
   • eventRefs for both events: [refs/heads/main]
   • scopes: [verify:combo]

Two more things for a maintainer to know:

• Private-row diagnostics are the maintainer's call. The default taken here withholds a client-estate row's module list and verifier output from this repository's public log, summary and artifacts. If the estate should get those somewhere private, that needs a destination this workflow does not have.
• Closing reading for the issue (unchanged): a combo-verify run after a successful CD in which the verdict prints verify-jobs=N verify-jobs-succeeded=N for the full roster and every slot reaches landed: with a Green verdict. The first run after this merge should show N per-instance jobs existing at all; until the provisioning above is done, most of them will be red with the 401.

No recycle is needed: this changes CI only, nothing a per-node hub serves.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Public artifacts can expose private module inventories, and digest masking collisions can still suppress matrix creation.

4 open findings

• Private evidence is uploaded to a public repository · New
• Privacy guard does not validate artifact name or path · New
• Decimal digest can collide with masked identifier values · New
• Documentation overlooks sensitive content in published evidence · New

What changed in this PR

Reworks combo verification so private fleet rosters no longer cross GitHub Actions job boundaries.

Changes:

• Uses opaque matrix slots plus count/digest validation.
• Re-derives each slot and counts actual verification jobs.
• Adds regression guards and documentation.

File	Description
ComboGateWiring.md	Documents slot-based handover.
combo-verify.yml	Rebuilds the matrix and verdict logic.
derive-combo-instances.py	Adds digest and slot resolution.
combo-verify-instance.sh	Uses slot-based filenames.
check-combo-verify.py	Adds structural and behavioral guards.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Private-row classification gaps can still expose client installation details and inventories.

7 open findings

• Mask derived identifiers for private repository rows · New
• Preserve repository identity for duplicate instance IDs · New
• Preserve privacy classification for short private identifiers · New
• Decimal digest can collide with masked identifier values
• Privacy guard does not validate artifact name or path
• Use the external instance URL in private 401 guidance · New
• Documentation overlooks sensitive content in published evidence

1 resolved since last review

• Private evidence is uploaded to a public repository

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

The digest can still collide with valid masked identifiers, and several private-row paths can disclose or misclassify client data.

5 open findings

• Private hostnames are not masked before external commands · New
• Short private IDs are omitted from privacy classification · New
• Mask derived identifiers for private repository rows
• Digest encoding can collide with registered private IDs · New
• Documentation overlooks sensitive content in published evidence

5 resolved since last review

• Preserve privacy classification for short private identifiers
• Preserve repository identity for duplicate instance IDs
• Decimal digest can collide with masked identifier values
• Privacy guard does not validate artifact name or path
• Use the external instance URL in private 401 guidance

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Digest collisions and fail-open privacy paths can still block verification or disclose private identifiers.

2 open findings

• Fail closed when the private roster is unavailable · New
• Sanitize diagnostics for all private IDs · New

5 resolved since last review

• Short private IDs are omitted from privacy classification
• Private hostnames are not masked before external commands
• Mask derived identifiers for private repository rows
• Digest encoding can collide with registered private IDs
• Documentation overlooks sensitive content in published evidence

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Private source identifiers and excluded rows can still leak, while an optional digest key permits offline identifier confirmation.

3 open findings

• Empty salt enables offline guessing of private identifiers · New
• Private plugin sources leak names and URLs in reports · New
• Excluded summary leaks private repository identifiers · New

2 resolved since last review

• Sanitize diagnostics for all private IDs
• Fail closed when the private roster is unavailable

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

The digest does not bind privacy provenance, allowing classification to change between preflight and verification.

2 open findings

• HMAC omits provenance, allowing privacy classification changes · New
• Digest guarantee ignores repository-based privacy classification · New

3 resolved since last review

• Private plugin sources leak names and URLs in reports
• Empty salt enables offline guessing of private identifiers
• Excluded summary leaks private repository identifiers

🧠 Review effort: Balanced

[rbuergi]

Not changing this, because the transition described is not a leak. The digest and the privacy bit answer two different questions on purpose.

• The digest answers "is this the same installation the preflight counted?", so a verdict cannot land on a different one. Name, URL and source map are what identify it.
• INSTANCE_PRIVATE is decided by the verify job's own derivation, at the moment it is about to print and upload, from where the row is declared at that moment. The preflight's view of privacy governs nothing in the verify job.

In the transition you describe, the row's overlay has moved into a repository listed in the committed, public instances.json before the verify job reads it. Its name and host are then already published in a public repository, so classifying it public at that point is correct, not a flip to an unsafe state. The opposite move (committed to uncommitted) makes the verify job classify it private, which is the safe direction.

Binding the preflight's bit into the digest would turn that legitimate mid-run change into a red and protect nothing: the bit that decides what is printed would still be the verify job's own.

[rbuergi]

The sentence claims the digest proves the verify job derived the same roster (names, URLs, source map), and that is what it checks. Privacy classification is deliberately not part of that claim: each verify job classifies its own row from where it is declared when the job runs. The reasoning is in my reply on the sibling thread: #6453 (comment). No doc change, since the page does not say the digest binds privacy.

[github-actions]

Test Results

    24 files      24 suites   53m 35s ⏱️
11 623 tests 11 430 ✅ 193 💤 0 ❌
11 637 runs  11 444 ✅ 193 💤 0 ❌

Results for commit 10e290d.