[rbuergi]

What was cut, and why

On 2026-10-23, attempt 1 of three runs turned Consolidate test results red because shards hit the 20-minute job cap:

• PR run 40011143666, shard 4.
• PR run 40011313475, shards 2 and 4.
• main run 40027189653, shard 4.

Nothing was hung. In every cut shard, all projects were still making progress when the cap hit.

• Memex.Portal.Shared.Test part 3/3 entered a new class every few seconds right up to the cancel.
• On green runs that part measures 315–343 s.
• The PluginBundleClient.ReadIndex stacks come from ModuleReloadFaultedTest, which injects that fault on purpose.

Two causes stacked:

1. The browser step spent its whole 480 s budget on a stalled Ubuntu mirror. It ran playwright-core install --with-deps for its apt half, and on ubuntu-24.04 that half adds only nine font packages. Each cut shard spent exactly 8:00 there; healthy shards spend 20–70 s. The step then gave up on the fonts and carried on anyway. Triage node: rbuergi/Feedback/20261024-browser-step-spends-480s-on-optional-apt-fonts.
2. Stale weights. Shard 4 printed ~454 s of load but ran 875–894 s.

The fix

No apt in a shard

• New job browser-fonts ("Stage the browser font set"). It runs in parallel with the build and needs nothing.
  • On a warm cache it does a lookup and nothing else.
  • On a miss it first checks .github/chromium-fonts.list against the pinned Playwright's own install-deps --dry-run. A font package that is neither listed nor already installed fails the job, so the set stays the one deploy/base-images/portal-ai/Dockerfile ships.
  • It then downloads the .debs with a private apt state (own lists and cache, locking off, as the runner user, so no system apt lock is involved). It extracts their font files with dpkg-deb -x and saves them as the chromium-fonts cache. It caches nothing if any part fails.
  • The logic is in .github/scripts/stage-chromium-fonts.sh; key mode gives both jobs one cache-key definition.
• The shards only restore and unpack.
  • "Restore the browser font set" has timeout-minutes: 2. That is deliberately longer than its 1 min download abort, so a stalled download ends as a reported cache miss and never as the step cap failing the shard (review thread).
  • "Provision the headless browser" has timeout-minutes: 5. It downloads the browser from the CDN (150 s budget, 90 s per attempt; the 115 MiB download takes about 2 s), installs the font set (40 s bound) and runs the smoke print.
  • Gone from the shards: --with-deps, the apt lock config, wait-for-apt.sh and the orphaned-apt reap.
  • A missing cache is a ::warning:: and the smoke print still decides. That is the same contract a failed --with-deps had.

Guards, re-modelled

• ShardWeightHeadroomGuard.TheIdealShardLoadFitsTheShardJobCap.
  • It takes the browser's cost as the two steps' step-level timeout-minutes, 420 s, which GitHub enforces end to end.
  • It sums the load over every project the scheduler discovers, with unlisted ones at DEFAULT_WEIGHT.
  • Eight shards pass (513 s ≤ 567 s). Seven fail (586 s).
• New: TheProvisionStepsInnerBoundsFitItsCap. The download budget, kill grace, font install and smoke print add up to 285 s, inside the step's 300 s. A stall therefore fails with its own message before the step cap fires.
• New: TheFontRestoreMissesBeforeItsCap. It requires the restore step's cap to exceed the download abort by at least 30 s.
• New: BrowserProvisioningGateGuard.TheTestShardsNeverReachAPackageMirror. It forbids --with-deps, install-deps and apt-get in the test job. A repo with no --with-deps provisioner is no longer a vacuous failure.

Weights and splits

• Each weight is the max over three green full main runs (40014951907, 40021939021, 40027189653).
• I chose part counts by replaying the trx per-class durations through the workflow's sorted every-Nth class split. That replay reproduces the measured parts exactly.

Project	Parts	Predicted part seconds
Memex.Portal.Shared.Test	3→4	240/179/243/254
MeshWeaver.Graph.Test	3→4	210/221/252/172
MeshWeaver.Messaging.Hub.Test	2→3	136/223/154 (part 2/2 was 409–416 s, 87% of the per-project cap)
MeshWeaver.Hosting.Test	3→2	223/226

MeshWeaver.Compiler.Pipeline.Test keeps #7087's split.

CI_SHARD_TOTAL 6→8

Eight is what the guard allows. The matrix has 8 shards and 8 upload steps, and both SHARD_TOTAL settings read ${{ env.CI_SHARD_TOTAL }}. The job cap is unchanged.

Evidence

• The staging and install, in ubuntu:24.04 as a non-root user with sudo.
  • stage fetched 9 packages (21 MB, 192 font files).
  • install took fontconfig from 13 to 47 faces (WenQuanYi Zen Hei, IPAGothic, FreeSerif, Loma, Unifont).
  • Playwright 1.62.1's dry run asks for exactly the nine listed packages.
  • Drift negative control: dropping fonts-wqy-zenhei from the list makes stage fail with "installs font package(s) fonts-wqy-zenhei on this release, which drifted.list does not stage".
• Guard negative controls, run locally against a mutated workflow and then restored:
  • CI_SHARD_TOTAL: 7 → red: "Expected 586.43 to be less than or equal to 567 … 4105 s … at CI_SHARD_TOTAL=7".
  • (previous round) Provision step capped at 5 min with the earlier 345 s of inner bounds → red: "bounds … add up to 345 s, more than its 300 s".
  • Restore step timeout-minutes: 1 → red: "Expected 0 to be greater than or equal to 30".
  • install --with-deps put back into the shard → red, naming the line.
• Local gates.
  • MeshWeaver.Documentation.Test builds with -c Release -warnaserror (0 warnings, 0 errors), and its full run passed 917 of 917.
  • actionlint (with shellcheck) is clean, shellcheck on the script is clean, check-workflow-shell.py reports 0 live findings, and check-workflow-timeouts.py reports 0 violations.

What this does NOT establish

• The new job has not run on a GitHub runner yet. This PR's own run is the first: a cache miss, then the stage. I tested the same script only on arm64 ubuntu:24.04 against ports.ubuntu.com, not on the amd64 runner image.
• Font fidelity is "same files", not "same install". No package postinst runs: no update-alternatives for the generic Japanese-gothic name and no mkfontdir. Chromium resolves faces through fontconfig, which indexes the files directly, but I did not compare a printed PDF against one from a real --with-deps install.
• Shard times are estimates. The heaviest of the eight shards should run about 545 s, from today's class list. Adding a class shifts the every-Nth parity.
• Cache-key edge case. The browser-fonts job checks out the event's ref. A shard tests the fresh merge with main. If main changes the font list mid-flight, the keys differ, and the shards warn and print without the staged fonts for that run.

Refs #7087.

Pairs-with: none — CI configuration, CI scripts, test-tree guards and a doc page only; no public surface changes.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The new guard omits scheduled fallback-weighted projects and understates browser overhead, so it can pass without the intended headroom.

2 open issues

• Guard omits projects using scheduler fallback weights · New
• Browser budget excludes post-deadline work · New

What changed in this PR

This PR rebalances MeshWeaver’s CI test workload to reduce shard cancellations without increasing job timeouts.

Changes:

• Refreshes project weights and split counts, and increases parallel shards from six to eight.
• Adds a shard-budget guard and parser self-test.
• Documents the timing evidence and remaining estimation limits; Compiler.Pipeline splitting remains assigned to #7087.

File	Description
test/​MeshWeaver.Documentation.Test/​ShardWeightHeadroomGuard.cs	Adds ideal-load budget validation and parser checks.
src/​MeshWeaver.Documentation/​Data/​Architecture/​ReadingCiSignals.md	Documents shard sizing and timing limitations.
.github/​workflows/​dotnet-test.yml	Expands the matrix and artifact uploads to eight shards.
.github/​scripts/​shard-assign.sh	Updates measured weights, project splits, and supporting comments.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The new shard timings remain estimates, and documented peak loads can still exceed the available test budget, requiring human acceptance of the remaining timeout risk.

1 open issue

• Use an immutable dictionary for the unmodified lookup · New

2 resolved since last review

• Browser budget excludes post-deadline work
• Guard omits projects using scheduler fallback weights

_{Review effort: Balanced}

[github-actions]

Test Results

    31 files  +2      31 suites  +2   1h 7m 36s ⏱️ - 1m 25s
14 384 tests +5  14 187 ✅ +5  197 💤 ±0  0 ❌ ±0 
14 414 runs  +5  14 217 ✅ +5  197 💤 ±0  0 ❌ ±0 

Results for commit dc3e3d0. ± Comparison against base commit b85e51b.

This pull request removes 44 and adds 23 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---)
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "521d9fa5-5198-4e57-a1ef-cbab7ca2aff6")
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.AggregateException: One or more errors occurred. (Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.)
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.InvalidOperationException: landing faulted
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Documentation.Test.BrowserProvisioningGateGuard ‑ TheTestShardsNeverReachAPackageMirror
…

♻️ This comment has been updated with latest results.

[Copilot]

🟡 Changes recommended

The font-cache deadline can skip shard tests, and the staging script violates the required shell gate.

2 open issues

• Avoid swallowed shell errors in Playwright lookup · New
• Prevent font-cache timeout from skipping shard tests · New

1 resolved since last review

• Use an immutable dictionary for the unmodified lookup

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Competing cache-download and step deadlines can fail shards before the intended warning-and-smoke fallback runs.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Leave timeout headroom for cache-miss fallback

.github/​workflows/​dotnet-test.yml:1071

The cache segment timeout and the entire restore step cap are both 60 seconds, but the segment timer starts after cache lookup and download setup. A stalled segment can therefore hit the step cap before the action returns a cache miss, failing the shard and skipping browser provisioning and tests instead of reaching the warning-and-smoke fallback. Give the internal timeout enough headroom inside the step cap for that fallback to run, and re-check the shard-budget accounting if the outer cap changes.

2 resolved since last review

• Prevent font-cache timeout from skipping shard tests
• Avoid swallowed shell errors in Playwright lookup

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The supplied evidence does not yet verify cold-cache font staging on the amd64 GitHub runner used by this workflow.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Use an immutable collection instead of mutable ToList()

test/​MeshWeaver.Documentation.Test/​BrowserProvisioningGateGuard.cs:131

This new ToList() introduces a mutable collection despite the immutable-collections requirement in AGENTS.md (“Collections Policy”). The guard only enumerates and counts these lines, so use an immutable list without changing its behavior.

_{Review effort: Balanced}