ci: check the analyses reached code scanning

What the workflow declares and what code scanning received are different
claims, and only the second is evidence. A scan that runs and uploads
nothing, or whose category merges with another tool's so that both are
filed as one, is invisible to anything that reads the workflow file.

A third job waits for both uploads and asserts that the commit this run
scanned carries analyses under exactly the three expected categories,
retrying while indexing catches up. It holds read on security-events, not
write: it must not be able to delete what it is checking. It runs unless
the run was cancelled, because needs: alone would skip it when a scan
fails, and a skipped job reports success to whatever requires it.

verify-analyses.test.sh stands gh in front of canned analyses and pins the
verdict for each way the expected set can be wrong: a category filed on an
earlier commit or another ref, a set split over pages, indexing that lands
an attempt late, a call that fails once, a call that fails every time, and
an API that stops answering after it has answered. An error on one attempt
has to be retried; an error on every attempt still fails the job. The
interval between attempts is recorded rather than waited out, and the test
cannot speak to a body written to recognise its own launch and skip in the
job, which review decides rather than a sibling test.

An API that never answered is reported as such rather than as an empty
set of analyses, and the fixtures carry an unexpected category at each end
of the sort order and a substitution at each end, where a comparison that
tolerates a longer list, or counts rather than compares, would pass. A
commit sharing seven hex with the scanned one carries the whole expected
set in another, and a category differing only in case appears beside the
expected one as well as in its place, so an abbreviated match and a
case-folded comparison are each red.

The analyses are read with --paginate: the ref lists every analysis ever
filed on it, newest first, and this pull request's ref passed a hundred in
a little over a day, so a single page is a truncated answer that hides an
older commit's analyses.

Signed-off-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>

Author: kalyazin

.github/scripts/security-workflow.test.py

@@ -34,6 +34,8 @@
 SCAN_PERMISSIONS = {
     "codeql": {"contents": "read", "actions": "read", "security-events": "write"},
     "osv": {"contents": "read", "actions": "read", "security-events": "write"},
+    # Reads what the two above wrote; write here would let this job close what it is checking.
+    "verify": {"contents": "read", "security-events": "read"},
 }
 PACKS = {
     "rust": "codeql/rust-queries:codeql-suites/rust-code-scanning.qls",
@@ -57,12 +59,14 @@
 JOB_KEYS = {
     "codeql": {"runs-on", "timeout-minutes", "permissions", "strategy", "steps"},
     "osv": {"runs-on", "timeout-minutes", "permissions", "steps"},
+    "verify": {"runs-on", "timeout-minutes", "permissions", "needs", "if", "steps"},
 }
 RUNNER_JOB_KEYS = {"runs-on", "timeout-minutes", "steps"}
 STEP_SHAPES = ({"name", "uses", "with"}, {"name", "run"}, {"name", "run", "env"})
 JOB_STEPS = {
     "codeql": ("actions/checkout", "github/codeql-action/init", "github/codeql-action/analyze"),
     "osv": ("actions/checkout", "run", "run", "github/codeql-action/upload-sarif"),
+    "verify": ("actions/checkout", "run"),
 }
 JOB_ACTIONS = {
     "codeql": ("actions/checkout", "github/codeql-action/init", "github/codeql-action/analyze"),
@@ -84,31 +88,34 @@
 MATRIX_KEYS = {"language", "include"}
 GENERATE_BODY = 'cd "${RUNNER_TEMP:?}" && cargo generate-lockfile --manifest-path "${GITHUB_WORKSPACE:?}/Cargo.toml"'
 SCAN_SCRIPT = ".github/scripts/scan.sh"
-SCAN_BODIES = (GENERATE_BODY, f"bash {SCAN_SCRIPT}")
-SCRIPTS = pathlib.Path(__file__).absolute().parent
-TESTS = sorted(p.relative_to(ROOT).as_posix() for p in SCRIPTS.glob("*.test.*"))
-SHELL_SCRIPTS = sorted(p.relative_to(ROOT).as_posix() for p in SCRIPTS.glob("*.sh"))
+VERIFY_SCRIPT = ".github/scripts/verify-analyses.sh"
+SCAN_BODIES = (GENERATE_BODY, f"bash {SCAN_SCRIPT}", f"bash {VERIFY_SCRIPT}")
+SHELL_SCRIPTS = [
+    ".github/scripts/scan.sh",
+    ".github/scripts/scan.test.sh",
+    ".github/scripts/verify-analyses.sh",
+    ".github/scripts/verify-analyses.test.sh",
+]
 # A `# shellcheck disable=` line silences the lint from inside the file it lints, so what any of them may silence is pinned here.
 SILENCED = []
+TESTS = [s for s in SHELL_SCRIPTS if s.endswith(".test.sh")] + [".github/scripts/security-workflow.test.py"]
 UNTESTED = [s for s in SHELL_SCRIPTS if not s.endswith(".test.sh") and s[:-3] + ".test.sh" not in TESTS]
-EXPECTED_TESTS = [
-    ".github/scripts/scan.test.sh",
-    ".github/scripts/security-workflow.test.py",
-]
 SHELLCHECK_BODY = 'docker run --rm --user "$(id -u):$(id -g)" --volume "$GITHUB_WORKSPACE:$GITHUB_WORKSPACE:ro" --workdir "$GITHUB_WORKSPACE" "$SHELLCHECK" \\\n' + " \\\n".join(["    --norc"] + [f"    {script}" for script in SHELL_SCRIPTS])
 INTERPRETERS = {"sh": "bash", "py": "python3"}
 SHELLCHECK_REPO = "koalaman/shellcheck"
 SHELLCHECK_DIGEST = "sha256:61862eba1fcf09a484ebcc6feea46f1782532571a34ed51fedf90dd25f925a8d"
 # Keyed by what the step runs: the env names that step may set. The runner sets the rest.
 STEP_ENV = {
     SCAN_SCRIPT: {"SCANNER"},
+    VERIFY_SCRIPT: {"GH_TOKEN", "REF", "SHA", "EXPECTED"},
     '"$LINTER"': {"LINTER"},
     '"$SHELLCHECK"': {"SHELLCHECK"},
 }
 INIT_KEYS = {"languages", "build-mode", "config"}
 ANALYZE_KEYS = {"category"}
 # Pinned: another ${{ }} naming matrix.language reads as per-language but resolves to one.
 ANALYZE_CATEGORY = "/language:${{ matrix.language }}"
+VERIFY_IF = "${{ !cancelled() }}"
 UPLOAD_KEYS = {"sarif_file", "category"}
 # The whole file, compared as one value: only github-actions belongs here, the consuming fork's
 # lockfile owning dependency versions.
@@ -478,6 +485,42 @@ def minutes(value):
 )
 
 
+verify_steps = jobs["verify"]["steps"] if "verify" in jobs else []
+check(
+    jobs.get("verify", {}).get("needs") == ["codeql", "osv"],
+    f"verify: it needs {jobs.get('verify', {}).get('needs')}, not ['codeql', 'osv']; the list is "
+    "pinned because a verify that does not wait for both uploads reads an empty set as a missing scan",
+)
+check(
+    expr(jobs.get("verify", {}).get("if")) == VERIFY_IF,
+    f"verify: it runs if {jobs.get('verify', {}).get('if')!r}, not {VERIFY_IF}; the guard is pinned "
+    "because a verify that a failed scan skips reports that skip as success to whatever requires it",
+)
+verify_step = next((st for st in verify_steps if launches(str(st.get("run", "")), VERIFY_SCRIPT)), None)
+check(verify_step is not None, f"verify: no step runs {VERIFY_SCRIPT}, so nothing reads what was uploaded")
+if verify_step is not None:
+    env = verify_step.get("env") or {}
+    check(
+        set(str(env.get("EXPECTED", "")).split(" ")) == CATEGORIES,
+        f"verify: it expects {str(env.get('EXPECTED', '')).split(' ')}, not {sorted(CATEGORIES)}; the "
+        "categories it waits for are what makes a missing or merged upload visible",
+    )
+    check(
+        expr(env.get("SHA")) == "${{ github.sha }}",
+        f"verify: it reads the analyses of {env.get('SHA')!r}, not the commit this run scanned",
+    )
+    # Already refs/pull/<n>/merge on a pull request, which is where those analyses are filed.
+    check(
+        expr(env.get("REF")) == "${{ github.ref }}",
+        f"verify: it reads the analyses of {env.get('REF')!r}, not the ref this run scanned",
+    )
+    # The scopes above bound this job only while it runs on the token they apply to.
+    check(
+        expr(env.get("GH_TOKEN")) == "${{ github.token }}",
+        f"verify: it authenticates with {env.get('GH_TOKEN')!r}; only the job's own token carries "
+        "the scopes declared above, and only the one spelling of it is read here",
+    )
+
 osv_steps = jobs["osv"]["steps"]
 gen_i = next((i for i, s in enumerate(osv_steps) if "generate-lockfile" in str(s.get("run", ""))), None)
 scan_i = next((i for i, s in enumerate(osv_steps) if SCAN_SCRIPT in str(s.get("run", ""))), None)

.github/scripts/verify-analyses.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# Reading the workflow cannot see a scan that uploaded nothing, or one whose category merged
+# with another tool's so that both were filed as one.
+set -euo pipefail
+
+REPO=${GITHUB_REPOSITORY:?}
+REF=${REF:?}
+SHA=${SHA:?}
+EXPECTED=${EXPECTED:?the categories that must exist, space separated}
+
+oneline() { printf '%s' "$1" | tr '\n' ' '; }
+
+# Compared as JSON text, because a category is free text: joined by any character a category may
+# itself contain, one filed as "a b c" would read as three present and hide two missing.
+read -ra expected <<<"$EXPECTED"
+want=$(printf '"%s"\n' "${expected[@]}" | sort -u)
+found=""
+answered=""
+
+# Both uploads wait for processing, but that wait gives up after about two and a half minutes and
+# passes anyway, so this covers processing as well as listing lag. No run has needed a second
+# attempt, which is why the one reached is printed rather than assumed.
+for attempt in $(seq 10); do
+    # The ref lists every analysis ever filed on it, and one page is a truncated answer.
+    # A failed call says nothing about what is filed, so the last answer the API gave is kept.
+    if page=$(gh api --paginate "repos/$REPO/code-scanning/analyses?ref=$REF&per_page=100" \
+        --jq ".[] | select(.commit_sha == \"$SHA\") | .category | @json"); then
+        found=$(printf '%s\n' "$page" | sort -u)
+        answered=yes
+    fi
+    [ "$found" = "$want" ] && break
+    [ "$attempt" = 10 ] || sleep 15
+done
+
+if [ "$found" != "$want" ]; then
+    if [ -n "$answered" ]; then
+        echo "::error::$SHA on $REF carries analyses under [$(oneline "$found")], not [$(oneline "$want")]"
+    else
+        echo "::error::the code scanning API never answered for $SHA on $REF, so what it carries is unknown"
+    fi
+    exit 1
+fi
+echo "ok: $(oneline "$want"), all on $SHA, at attempt $attempt"