ci: every job is hard-cut at 45 minutes — a literal timeout-minutes, gated fleet-wide

.claude/skills/ci/SKILL.md

@@ -125,6 +125,37 @@ burst can no longer leave *nothing* completed, which is what silenced CD entirel
 merge queue (#2412), which tests the prospective combination before it lands; this repo is still
 missing the `merge_group:` trigger that MeshWeaver.Plugins already has.
 
+## 🚨 Every job is HARD-CUT at 45 minutes — `timeout-minutes` is mandatory, literal, ≤ 45
+
+Maintainer, 2026-09-02: *"hard cut ci runs after 45min — we pay all this."* GitHub's default job
+timeout is **360 minutes**. A job without an explicit cap that hangs therefore bills a runner for six
+hours — and when a required check `needs:` it, nothing behind it can merge for six hours either.
+
+What it cost, measured the morning the rule was written: the reusable module-pack lane's `pack` job
+had no cap; `dotnet test MeshWeaver.Mcp.Test` wedged before its first test on EVERY MeshWeaver.Plugins
+run; **19 runs sat `in_progress` at once** (11 of them on `main`), each holding a runner up to 360
+minutes; the three required checks all `needs:` that job, so no Plugins PR could merge and `main`
+never reached `publish-bake` — which starved every satellite of a sealed publication. One missing line.
+
+The rule, enforced by `.github/scripts/check-workflow-timeouts.py` (self-tested, fails RED):
+
+- every job in every workflow declares `timeout-minutes: <literal integer>` with `1 ≤ value ≤ 45`
+  — a `${{ … }}` expression is refused, because a cap that can only be evaluated at run time
+  cannot be proven by reading the file;
+- a job that `uses:` a reusable workflow is exempt *in the caller* (GitHub ignores the caller's
+  value there) — the cap lives on the jobs INSIDE the reusable workflow, gated in the repo that
+  defines it (this one, for every `node-repo-*.yml` lane);
+- core runs the guard on itself in `dotnet-test.yml` beside `check-workflow-shell.py`; every
+  satellite gets it through `node-repo-validate.yml`, which fetches the platform's script at
+  `platform-ref` and runs it against the caller's tree — pin `platform-ref` beside the `uses:` sha.
+
+Measured headroom on 2026-09-02: the longest honest jobs are the Plugins portal-host shards
+(25–30 min), `release-images` (36–41 min) and the Education install shards (~27 min). A job that
+reaches 45 is **stuck, not slow** — find what is not completing; never raise the bound (AGENTS.md →
+"No band-aids"). And a cap that fires destroys the hung test's transcript with it (a host-cap kill
+leaves no `.trx`), so when you are chasing a hang, capture the stack (`dotnet-stack report -p`)
+BEFORE the cap does.
+
 ## Satellite CI = thin callers of THIS repo's reusable workflows
 
 **Never hand-roll (or copy-paste) a node repo's CI.** The shared jobs live here as `workflow_call`
@@ -141,6 +172,7 @@ Full contract:
 
 ## Checklist
 
+- [ ] Every job I added or touched carries a literal `timeout-minutes` ≤ 45 (`python3 .github/scripts/check-workflow-timeouts.py --root .` is green).
 - [ ] No `continue-on-error` on a gate's input step; no `if:` asking whether a secret/variable is
       set. Fork-PR exemption expressed once, on the event.
 - [ ] Missing external inputs fail a `preflight` job RED, naming what to provision.

.github/scripts/check-workflow-timeouts.py

@@ -0,0 +1,172 @@
+#!/usr/bin/env python3
+"""Every CI job carries a hard wall-clock cap, and the cap is at most 45 minutes.
+
+WHY THIS EXISTS (maintainer, 2026-09-02: "hard cut ci runs after 45min — we pay all this")
+------------------------------------------------------------------------------------------
+GitHub's default `timeout-minutes` is 360. A job with no explicit cap therefore runs for SIX
+HOURS when it hangs, and it bills a runner the whole time. Measured the morning this was written:
+the reusable module-pack lane's `pack` job had no cap; `dotnet test MeshWeaver.Mcp.Test` wedged
+before its first test on EVERY run in MeshWeaver.Plugins; 19 runs sat `in_progress` at once, each
+holding a runner up to 360 minutes, and — because the repo's required checks `needs:` that job —
+no PR could merge and `main` never reached publish-bake, which starved every satellite of a sealed
+publication. One missing line, fleet-wide consequence, paid by the minute.
+
+The cap is also the doctrine: a job that runs past 45 minutes is STUCK, not slow (AGENTS.md — over
+budget means find what is not completing, never raise the bound). Measured on 2026-09-02, the
+longest honest jobs in the fleet were the Plugins portal-host shards (25–30 min), release-images
+(36–41 min) and the Education install shards (~27 min); everything else finishes well under 20.
+
+THE RULE
+--------
+Every job in every workflow under `.github/workflows/` either
+
+  * `uses:` a reusable workflow — exempt HERE, because GitHub ignores `timeout-minutes` on such a
+    job; the cap lives on the jobs INSIDE the reusable workflow, which this same check gates in the
+    repository that defines it (Systemorph/MeshWeaver's `node-repo-*.yml` lanes run this script
+    on themselves); or
+  * declares `timeout-minutes: <literal integer>` with `1 <= value <= MAX` (default 45).
+
+An EXPRESSION (`${{ … }}`) is refused: a cap that can only be evaluated at run time cannot be
+proven to hold, and the whole point is that the bound is provable by reading the file.
+
+USAGE
+-----
+  check-workflow-timeouts.py [--root DIR] [--max 45]      gate the tree at DIR (default: cwd)
+  check-workflow-timeouts.py --self-test                  prove the gate fires and stays silent
+
+Node repos do not copy this file: their `validate` lane (node-repo-validate.yml) fetches it from
+the platform at the pinned platform ref and runs it against the caller's tree — the same
+centralization as compile-check.py. Exit 1 on any violation; every violation is a `::error`
+annotation carrying the file and the job id.
+"""
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover - the CI step installs PyYAML; locally `pip install pyyaml`
+    print("::error::check-workflow-timeouts.py needs PyYAML (pip install pyyaml)")
+    sys.exit(2)
+
+DEFAULT_MAX = 45
+
+
+def workflow_files(root: Path) -> list[Path]:
+    wf = root / ".github" / "workflows"
+    if not wf.is_dir():
+        return []
+    return sorted(p for p in wf.iterdir() if p.suffix in (".yml", ".yaml") and p.is_file())
+
+
+def check_tree(root: Path, max_minutes: int) -> tuple[list[str], int, int]:
+    """Return (violations, jobs_checked, jobs_exempt)."""
+    violations: list[str] = []
+    checked = exempt = 0
+    files = workflow_files(root)
+    if not files:
+        violations.append(f"::error::{root}/.github/workflows has no workflow files — nothing to gate is a failure, not a pass")
+        return violations, 0, 0
+    for path in files:
+        rel = path.relative_to(root)
+        try:
+            doc = yaml.safe_load(path.read_text(encoding="utf-8"))
+        except yaml.YAMLError as e:  # a workflow that does not parse cannot be proven capped
+            violations.append(f"::error file={rel}::not valid YAML: {e}")
+            continue
+        jobs = (doc or {}).get("jobs") if isinstance(doc, dict) else None
+        if not isinstance(jobs, dict) or not jobs:
+            violations.append(f"::error file={rel}::declares no jobs — a workflow file with nothing to cap is not a workflow")
+            continue
+        for job_id, job in jobs.items():
+            if not isinstance(job, dict):
+                violations.append(f"::error file={rel}::job '{job_id}' is not a mapping")
+                continue
+            if "uses" in job:
+                exempt += 1
+                continue
+            checked += 1
+            if "timeout-minutes" not in job:
+                violations.append(
+                    f"::error file={rel}::job '{job_id}' has no timeout-minutes — GitHub's default is 360; "
+                    f"the fleet cap is {max_minutes}. Add `timeout-minutes: {max_minutes}` (or lower) under runs-on:"
+                )
+                continue
+            value = job["timeout-minutes"]
+            if isinstance(value, bool) or not isinstance(value, int):
+                violations.append(
+                    f"::error file={rel}::job '{job_id}' timeout-minutes is {value!r} — it must be a literal integer "
+                    f"<= {max_minutes}; an expression or a string cannot be proven to hold by reading the file"
+                )
+                continue
+            if value < 1 or value > max_minutes:
+                violations.append(
+                    f"::error file={rel}::job '{job_id}' timeout-minutes is {value} — the fleet cap is {max_minutes}. "
+                    f"A job that needs longer is stuck, not slow: find what is not completing, never raise the bound"
+                )
+    return violations, checked, exempt
+
+
+def self_test() -> int:
+    """Every check must FIRE on its defect and stay SILENT on its fix, or the gate is vacuous."""
+    cases: list[tuple[str, str, bool]] = [
+        # (name, workflow yaml, expect_violation)
+        ("ok-at-cap", "jobs:\n  a:\n    runs-on: ubuntu-latest\n    timeout-minutes: 45\n    steps: [{run: echo}]\n", False),
+        ("ok-below-cap", "jobs:\n  a:\n    runs-on: ubuntu-latest\n    timeout-minutes: 10\n    steps: [{run: echo}]\n", False),
+        ("missing", "jobs:\n  a:\n    runs-on: ubuntu-latest\n    steps: [{run: echo}]\n", True),
+        ("over-cap", "jobs:\n  a:\n    runs-on: ubuntu-latest\n    timeout-minutes: 60\n    steps: [{run: echo}]\n", True),
+        ("expression", "jobs:\n  a:\n    runs-on: ubuntu-latest\n    timeout-minutes: ${{ inputs.t }}\n    steps: [{run: echo}]\n", True),
+        ("string", "jobs:\n  a:\n    runs-on: ubuntu-latest\n    timeout-minutes: '45'\n    steps: [{run: echo}]\n", True),
+        ("zero", "jobs:\n  a:\n    runs-on: ubuntu-latest\n    timeout-minutes: 0\n    steps: [{run: echo}]\n", True),
+        ("uses-exempt", "jobs:\n  a:\n    uses: Systemorph/MeshWeaver/.github/workflows/x.yml@abc\n    with: {a: 1}\n", False),
+        ("second-job-missing", "jobs:\n  a:\n    runs-on: ubuntu-latest\n    timeout-minutes: 5\n    steps: [{run: echo}]\n  b:\n    runs-on: ubuntu-latest\n    steps: [{run: echo}]\n", True),
+        ("no-jobs", "name: x\non: push\n", True),
+        ("not-yaml", "jobs: [unclosed\n  - ::\n", True),
+    ]
+    failures = 0
+    for name, body, expect in cases:
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            (root / ".github" / "workflows").mkdir(parents=True)
+            (root / ".github" / "workflows" / "t.yml").write_text(body, encoding="utf-8")
+            violations, _, _ = check_tree(root, DEFAULT_MAX)
+            fired = bool(violations)
+            verdict = "ok" if fired == expect else "FAIL"
+            if fired != expect:
+                failures += 1
+            print(f"self-test {verdict:4} {name:20} expected={'fire' if expect else 'silent'} got={'fire' if fired else 'silent'}")
+    with tempfile.TemporaryDirectory() as tmp:  # a tree with no workflows dir must fail, never pass vacuously
+        violations, _, _ = check_tree(Path(tmp), DEFAULT_MAX)
+        fired = bool(violations)
+        print(f"self-test {'ok' if fired else 'FAIL':4} {'no-workflows-dir':20} expected=fire got={'fire' if fired else 'silent'}")
+        failures += 0 if fired else 1
+    if failures:
+        print(f"::error::check-workflow-timeouts.py self-test: {failures} case(s) did not behave — the gate is not proven")
+        return 1
+    print("self-test: every case fired on its defect and stayed silent on its fix")
+    return 0
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
+    ap.add_argument("--root", default=".", help="repository root holding .github/workflows (default: cwd)")
+    ap.add_argument("--max", type=int, default=DEFAULT_MAX, help=f"the cap in minutes (default {DEFAULT_MAX})")
+    ap.add_argument("--self-test", action="store_true", help="prove the gate is non-vacuous and exit")
+    args = ap.parse_args()
+    if args.self_test:
+        return self_test()
+    root = Path(args.root).resolve()
+    violations, checked, exempt = check_tree(root, args.max)
+    for v in violations:
+        print(v)
+    print(f"check-workflow-timeouts: {checked} job(s) checked, {exempt} reusable-call job(s) exempt, "
+          f"{len(violations)} violation(s), cap={args.max} min, root={root}")
+    return 1 if violations else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/workflows/alc-unload-probe.yml

@@ -54,7 +54,7 @@ jobs:
     name: "Loop native host under ALC-unload GC probe"
     # ubuntu-latest == the 2-vCPU hosted runner the crash reproduces on.
     runs-on: ubuntu-latest
-    timeout-minutes: 120
+    timeout-minutes: 45
     env:
       PROBE_PROJECT: ${{ inputs.project || 'MeshWeaver.FutuRe.Test' }}
       PROBE_ITERS: ${{ inputs.iterations || '10' }}

.github/workflows/auto-arm.yml

@@ -58,6 +58,7 @@ jobs:
   arm:
     name: Arm auto-merge on this PR
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     # A draft is the opt-out, and a fork must never arm itself. Both are properties of the
     # event, not configuration — there is no variable here to forget to set.
     if: >-

.github/workflows/base-image-acr.yml

@@ -34,7 +34,7 @@ jobs:
   base:
     name: "Build + push multi-arch base to ACR"
     runs-on: ubuntu-latest
-    timeout-minutes: 60
+    timeout-minutes: 45
     steps:
       - uses: actions/checkout@v7
 

.github/workflows/chart-drift.yml

@@ -104,6 +104,7 @@ jobs:
   preflight:
     name: Assert the inputs exist
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     steps:
       # No `if:` anywhere in this job. It asserts and fails; it does not decide whether to run.
       - name: Every external input, or a red build naming what to provision
@@ -215,6 +216,7 @@ jobs:
     name: Read the environment list
     needs: [preflight]
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     outputs:
       matrix: ${{ steps.envs.outputs.matrix }}
       aks_cluster: ${{ steps.envs.outputs.aks_cluster }}
@@ -293,6 +295,7 @@ jobs:
     name: ${{ matrix.env.ns }}
     needs: [preflight, discover]
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     env:
       AKS_CLUSTER: ${{ needs.discover.outputs.aks_cluster }}
       AKS_RESOURCE_GROUP: ${{ needs.discover.outputs.aks_resource_group }}

.github/workflows/chart-gate.yml

@@ -53,6 +53,7 @@ jobs:
   invariants:
     name: Chart invariants
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     steps:
       - uses: actions/checkout@v7
       - name: Install helm

.github/workflows/clients.yml

@@ -76,6 +76,7 @@ jobs:
   python-sdk:
     name: "meshweaver (Python test)"
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     defaults:
       run:
         working-directory: clients/python

.github/workflows/dotnet-test.yml

@@ -1789,6 +1789,15 @@ jobs:
     - name: "NodeType import-scope parser can fail (self-test)"
       run: python3 .github/scripts/compile-check.py --self-test
 
+    # 🚨 Every job in every workflow carries a LITERAL `timeout-minutes` <= 45 (maintainer, 2026-09-02:
+    # "hard cut CI runs after 45min — we pay all this"). GitHub's default is 360; one uncapped job in
+    # the module-pack lane held a runner for six hours per run across 19 concurrent Plugins runs and
+    # blocked every required check behind it. Self-test first — an unproven gate is no gate.
+    - name: "Every job is capped at 45 minutes — the gate can fail (self-test)"
+      run: python3 .github/scripts/check-workflow-timeouts.py --self-test
+    - name: "Every job is capped at 45 minutes"
+      run: python3 .github/scripts/check-workflow-timeouts.py --root .
+
     - name: "Gate the shell CI itself runs"
       run: python3 .github/scripts/check-workflow-shell.py
 
@@ -1992,6 +2001,7 @@ jobs:
     needs: [precheck, build, test, doc-gate, licence-gate, clients-gate, record-signatures, workflow-shell, shared-rules]
     if: always()
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     steps:
     # Needed for the marker push/prune steps below: gives us a git repo whose
     # `origin` carries the run's credentials, and puts github.sha's object in it.

.github/workflows/flake-repro.yml

@@ -58,7 +58,7 @@ jobs:
     name: "Loop suspect test until fail"
     # ubuntu-latest == the 2-vCPU GitHub-hosted runner the real flake needs.
     runs-on: ubuntu-latest
-    timeout-minutes: 60
+    timeout-minutes: 45
     # Effective params: workflow_dispatch inputs when present, else push-trigger defaults.
     env:
       REPRO_PROJECT: ${{ inputs.project || 'test/MeshWeaver.Hosting.Orleans.Test' }}

.github/workflows/homebrew.yml

@@ -175,6 +175,7 @@ jobs:
     name: Publish inputs are provisioned
     if: github.event_name == 'push'
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     steps:
       - name: Every publish input exists
         env:

.github/workflows/hosting-operator.yml

@@ -31,6 +31,7 @@ jobs:
   scripts:
     name: Operator scripts (shellcheck + behaviour)
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     steps:
       - uses: actions/checkout@v7
 
@@ -198,6 +199,7 @@ jobs:
   image:
     name: Operator image builds
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     steps:
       - uses: actions/checkout@v7
       # Build only — nothing is pushed from a pull request. The point is that the Dockerfile and its

.github/workflows/node-repo-module-pack.yml

@@ -376,6 +376,7 @@ jobs:
   select:
     name: Select the affected bundles
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     outputs:
       modules: ${{ steps.scope.outputs.modules }}
       count: ${{ steps.scope.outputs.count }}
@@ -525,6 +526,7 @@ jobs:
     needs: select
     if: needs.select.result == 'success' && needs.select.outputs.count != '0'
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     steps:
       - name: Check out Systemorph/MeshWeaver at the pinned ref
         uses: actions/checkout@v7
@@ -820,6 +822,7 @@ jobs:
     name: Module bundle (${{ matrix.entry.module }})
     needs: [select, prepare, build-workspace]
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     # 🚨 This is a MODE SWITCH on the selection's own output, not a skip-trapdoor: GitHub refuses
     # a matrix with no vectors, so an empty selection has to express itself as "this job does not
     # run". What makes it safe is that `verify` below runs UNCONDITIONALLY and asserts the pairing
@@ -1578,6 +1581,7 @@ jobs:
     needs: [select, pack]
     if: always()
     runs-on: ubuntu-latest
+    timeout-minutes: 45
     outputs:
       bundles-built: ${{ steps.verify.outputs.bundles_built }}
     steps:

.github/workflows/node-repo-publish-bake.yml

@@ -202,11 +202,6 @@ on:
         required: false
         type: string
         default: ''
-      timeout-minutes:
-        description: Job timeout.
-        required: false
-        type: number
-        default: 40
       allow-file:
         description: Known-debt ratchet file passed as --allow (repo-relative). Empty = none.
         required: false
@@ -336,7 +331,10 @@ jobs:
   publish-bake:
     name: Bake + publish NodeType assemblies to portal storage
     runs-on: ubuntu-latest
-    timeout-minutes: ${{ inputs.timeout-minutes }}
+    # 🚨 A LITERAL, never an input: the fleet cap is 45 minutes for every job (check-workflow-timeouts.py
+    # refuses an expression here because a caller-supplied bound cannot be proven by reading the file).
+    # Measured 2026-09-02 the bake runs ~20 min; past 45 it is stuck, not slow.
+    timeout-minutes: 45
     outputs:
       # Read by dispatch-dependents below: a run that baked NOTHING (already sealed) publishes
       # nothing and therefore wakes nobody.