Main's nightly: the tarball in a TempDir, a join that keeps its answer, three judges that read the whole log

[Japabu]

Main's nightly is red. This branch fixes five of its reds where each one lives and files the older ones it does not fix. It is merged with origin/main at 5e446e5 (#524).

1. Every guest, tcg and audio lane: the toolchain tarball left in $TMPDIR

• Cause. Every test's scratch goes with it, and a killed run's with the next #529 (e48604c) made each guest lane's private $TMPDIR red on anything left in it. release::install downloaded toyos-toolchain.tar.zst to std::env::temp_dir() and never removed it.
• Fix. Both release paths stage in a toyos_tmpdir::TempDir: the install's tarball, and the publish's TOOLCHAIN, notes.md and tarball.
  • src/release.rs leaves TEMP_DIR_ALLOWED. Every test's scratch goes with it, and a killed run's with the next #529's leftover check is untouched.
  • Closes issues/build/the-release-path-stages-in-tmpdir-outside-the-scratch-guard.md and ARM64 port, stages 0 to 3: shared groundwork, aarch64-unknown-toyos with std and the userland, and the loader and kernel reach the PL011 on virt #524's issues/build/a-guest-lane-leaves-the-toolchain-tarball-in-its-tmpdir.md.
  • src/release.rs is hashed into the toolchain tag, so the tag moved. This branch's first nightly build job published the new one.
• Red before: main's nightly at fd62f56 (run 36278449733). All 15 guest, tcg and audio jobs red on left in … : toyos-toolchain.tar.zst.
• Green after: this branch's nightly at ccb6b4e (run 36280285913). Every guest, tcg and audio job says nothing left in $TMPDIR: every test took its scratch with it.

2. fpu_isolation: thread_join answered NotFound

• Cause. One way to wait: every waitable object has one Watch #513 (46056bf, "One way to wait"). sys_thread_join waited with wait_thread_zombie as its predicate, and that call collects: it takes the zombie out of the table.
  • Before One way to wait: every waitable object has one Watch #513, completion::wait_until returned on the Gone(Closed) that thread_exit posts, without running the predicate again.
  • watch::wait_until runs the predicate after every wake. It collected the zombie, and the loop's next collect found no such thread.
  • Every joiner that parked before its thread exited got NotFound.
  • std's and libc's joins drop the answer, so only fpu_isolation saw it. That is filed as issues/kernel/std-and-libc-drop-the-answer-thread-join-gives.md.
• Fix. The predicate keeps the first answer it collects, and the syscall returns that answer. Closes ARM64 port, stages 0 to 3: shared groundwork, aarch64-unknown-toyos with std and the userland, and the loader and kernel reach the PL011 on virt #524's issues/kernel/fpu-isolation-s-thread-join-answers-not-found-on-main.md.
• Red before, at fd62f56: cargo test --test toyos-build -- --nightly fpu_isolation EXIT=1, left: 18446744073709551614. Also red on run 36278449733, guest (12).
• Green after: EXIT=0 at 05839b7, and again on the merged tree. Green on run 36280285913, guest (12).
• Landing. One way to wait: every waitable object has one Watch #513 is named by reading its diff, and the red/green pair confirms it. It was not bisected by building its parent. It was green on main's nightly at e8d7c9c, and no landing between that and One way to wait: every waitable object has one Watch #513's parent touches kernel/, toyos-abi/src or rust.

3. blockd_survives_its_death: the reissue judge demanded an order the client never promised

• Cause. Storage stage 3, steps 1–4: toyos-blockring, SYS_DEVICE_DMA_MAP, blockd and partition sessions #525 (7345e3f). The trace judge required the blockd after a kill to rewrite the unflushed writes in the order the device first saw them.
  • The client reissues in first-acknowledged order, and requests in flight together are unordered (toyos-blockring/src/client.rs).
  • Run 36273557690 wrote [282600, 281592] and reissued [281592, 282600].
• Fix. The judge requires the same writes as a multiset, and the original order among the writes that overlap any one of them. Overlapping writes are never in flight together.
• Negative control. The client's reissue after a loss compiled out (the mutate-no-reissue-after-loss body), applied as a checked patch and restored. EXIT=1: the blockd after it wrote [281592, 280584] first.
• Oracle: QEMU's own NVMe trace.
• Green after: EXIT=0 locally, and green on run 36280285913, guest (2).

4. partition_claim_departure: kernel records that arrive after the end marker

• Cause. The judge predates Storage stage 3, steps 1–4: toyos-blockring, SYS_DEVICE_DMA_MAP, blockd and partition sessions #525; the finding filed at netd: one resolver of its own, a UDP socket bound to any address receives, and host names what it resolves #511 has the same shape. The runner's ===TEST_END=== reaches the console through logd and the kernel's records through klogd. A record the kernel made before the test ended can arrive after the window closed, and guest_verdict judged only the window.
• Reproduced at fd62f56 with 18 CPU spinners on 14 cores: 1 red in 6 runs, silent: 0 flushes were told of the loss, not 1. The capture holds the claimant's refused with Io, which the kernel answers only after it writes the record that is missing.
• Fix. guest_verdict judges the kernel's records from the test's start through the shutdown's drain, at all three call sites. Closes issues/boot-media/partition-claim-departure-told-no-flush-beside-other-guests.md.
• Green after:
  • under the same load, 8 of 8;
  • --nightly partition_claim: 3 passed, EXIT=0;
  • green on run 36280285913, guest (12).

5. exit_wait_storm: the parent's spawn record landed before the start marker

• Cause. The same two console paths.
  • The harness moves only the lines from the last matching spawn: record in before into the window.
  • When the parent and its first children were spawned before the marker arrived, the judge called a child the parent. It then counted the real parent's exit (code 0) as a child's, and read a child's syscall profile as the parent's.
  • A probe, applied as a checked patch and restored, showed pids 7 (the parent), 8 and 9 in before and the rest in the window.
• Fix. The judge reads before and the window together, takes the lowest pid as the parent (pids are never reused), and counts as children only the pids above it.
• Red before: red on run 36278449733 and on run 36280285913, guest (1), red alone as well. Locally, 3 of 4 runs.
• Green after: 8 of 8, EXIT=0 each. Green at 3f46a01, flaky at e8d7c9c, so it arrived with Program output belongs to logd: each program's pipe, the log served from the boot's first line, and the T14 found by its name #492's two-path console; not bisected.

Gates

• cargo run -- --ci host: 45 steps, all green, EXIT=0, on the merged tree and again at the head 67a430c.
• cargo test --lib -- release:: every_host_scratch_is_the_guard left_behind: 7 passed, EXIT=0.
• fpu_isolation, blockd_survives_its_death and partition_claim_departure each EXIT=0 on the merged tree.
• -- --nightly thread: 2 passed, EXIT=0.
• Fast tier (cargo test --test toyos-build) on the pre-merge tree: EXIT=1, 397 passed, 1 failed. The failure is lan_mdns_answer on path must be shorter than SUN_LEN, the macOS dev-host defect filed twice under issues/build/ (a-lane-s-tap-socket-path-…). It is not a Linux nightly red.

Nightly on this branch

Run 36285169430 at 67a430c, the head.

• Green: host, build, portability-linux, portability-macos, tcg, audio (1), audio (2), guest (2), (4), (6), (8), (11).
• portability-windows: the declared continue-on-error frontier.
• Every guest, tcg and audio lane says nothing left in $TMPDIR.
• fpu_isolation, blockd_survives_its_death, partition_claim_departure and exit_wait_storm are all green.

Run 36280285913 at ccb6b4e, before item 5. Same picture, with exit_wait_storm red in guest (1).

Still red on the head's run, none of them from the named suspects. Each is red on main's run 36278449733 or earlier, or flaky across these runs, and cargo run -- --known-red answers NO for every one:

lane	red	status
guest (1)	quiesce_leaves_the_volume_whole	red alone on main's run and on this head's run, green on the first branch run and locally (EXIT=0)
guest (3)	quiesce_wakes_on_the_last_exit	green on main's run, flaky on the branch runs, green locally; issues/build/quiesce-wakes-on-the-last-exit-lost-its-serial-ready-beside-other-guests.md
guest (5)	soundd_log_stall	timed out at 392 s; green on main's run and locally (254 s, EXIT=0)
guest (7)	usb_transport_break	red since #506 (265a0fc). Its usb-port-gone disk makes rootfs::hold_source panic, shown by removing that param as a checked patch. Filed as issues/boot-media/a-disk-whose-port-went-away-panics-the-boot-at-roots-hold.md
guest (7)	metal_job_reboot	red at e8d7c9c too; issues/build/metal-job-reboot-drained-no-kernel-output-beside-other-guests.md
guest (9)	screen_diag_boot	red since a landing between 3f46a01 and e8d7c9c; filed as issues/boot-media/screen-diag-boot-leaves-no-i8042-line-on-the-panel.md
guest (10)	log_flush_retry	red at 3f46a01 and e8d7c9c too; issues/filesystem/log-flush-retry-deadman-arm.md
guest (12)	home_budget_refusal_retried	red at e8d7c9c too; filed as issues/filesystem/home-budget-refusal-retried-is-red-on-every-nightly.md

On the first branch run, audio (2) reported a gate A wake-lateness regression, and on main's run audio (1) an instrument-broken capture. Both audio lanes are green on the head's run.

Unsure

• The klogd/logd race in items 4 and 5 applies to any judge that reads kernel records only between the markers. Only these two judges are changed. run_test_paced's move-from-the-last-spawn is left as it is.
• The quiesce pair, metal_job_reboot and soundd_log_stall alternate between red and green across three runs. Their rates are not measured here.

🤖 Generated with Claude Code

https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK

[Japabu]

Judged: the thread_join fix keeps the first answer its predicate collects (the predicate consumes the zombie; #513's recheck-after-wake made the second look NotFound) — read and correct; fpu_isolation EXIT=1 → 0. The release staging now goes through TempDir without weakening #529's leftover check; the three judges are fixed at their premises (blockd reissue order, kernel records past the marker, the parent pid). Nightly 36285169430 greens host, build, both portability lanes, tcg, audio and five guest lanes; the eight remaining reds predate or are filed. Landing.