@@ -8,8 +8,15 @@ opened: 2026-10-04
88
99` test_rs_counters_read ` in ` tests/virtsmpcase ` normally ends about 0.2 s of
1010guest clock after ` unmap_touch ` does (median 0.17-0.20 s over 570 guests, max
11- 1.46 s). Under host load it sometimes takes seconds, and twice it went silent
12- long enough for the harness's 15 s quiet bound to call the guest stalled.
11+ 1.46 s). Under host load it sometimes takes seconds, and it has gone silent
12+ long enough for the harness's 15 s quiet bound to call the guest stalled: each
13+ silent boot is below.
14+
15+ ` virt_mask_windows ` boots the same case on a ` mask-windows ` kernel and waits
16+ on two things, the job ` unmap_touch ` and then the boot's last word, so a
17+ silence in the counters read reds it as `STALLED: waiting for the boot's last
18+ word` : that wait spans the read, ` test_rs_trace_read` and the shutdown. Its
19+ stalls are recorded here.
1320
1421The first silent one: ` virt_el1_smp ` , 8 vCPUs under TCG, on a 14-core host at
15221-minute load 38-53 while the primary checkout was building the stage-2
@@ -54,9 +61,86 @@ line of the wait and none the guest runs.
5461- ** What is not known.** Whether the guest was running, spinning or not being
5562 scheduled by the host, and whether it was the guest or its console that
5663 stopped. No register capture of it exists either, so it adds a count and a
57- second profile and brings the exit no closer. The fork compiler miscompiles,
58- for AArch64, an inclusive range that ends at its integer type's maximum;
59- whether that reaches this job's code has not been examined.
64+ second profile and brings the exit no closer.
65+
66+ The third and fourth silent ones, two guests of one run: ` virt_el1_smp ` and
67+ ` virt_mask_windows ` , in the whole suite on ` wt/toyos-libc-sockets ` at
68+ ` 68cf1f870 ` , whose diff touches no line of either wait and nothing these
69+ guests start: ` tests/virtsmpcase ` runs no C program, opens no socket and
70+ starts no netstack.
71+
72+ - ** The lines.** `FAIL virt_el1_smp: STALLED: waiting for the job
73+ test_rs_counters_read to end — it went quiet` and ` FAIL virt_mask_windows:
74+ STALLED: waiting for the boot's last word — it went quiet`, both at the same
75+ second of the host's clock; 35 of 37 passed.
76+ - ** The load.** ` --jobs 4 --nocapture ` on the 14-core host, 1-minute load
77+ 75.99 when the run began and 63.37 when it ended, from other worktrees'
78+ builds; liveness ceilings paid at 1.52x; workers 366 s building against
79+ 808 s testing.
80+ - ** The waits.** ` virt_el1_smp ` : ` judge_virt_job ` 's ` await_marker ` on
81+ ` ===TEST_END test_rs_counters_read ` . ` virt_mask_windows ` : its second wait,
82+ ` await_marker ` on ` power::SHUTTING_DOWN ` , after ` judge_virt_job ` had judged
83+ ` unmap_touch ` . Each was ended by ` GUEST_QUIET ` .
84+ - ** The same profile as the second.** Each guest's last three lines are the
85+ end of ` unmap_touch ` , the start of the read and the kernel's record of its
86+ spawn, with no thread's exit after it. ` virt_el1_smp ` : `===TEST_END
87+ unmap_touch exit=0===` at 13.577, ` ===TEST_START test_rs_counters_read===`
88+ at 13.578, ` spawn: /system/bin/test_rs_counters_read pid=13 ` at 13.590 on
89+ cpu4. ` virt_mask_windows ` : the same three at 14.557, 14.558 and 14.585, on
90+ cpu4. The two spawn records reached the harness within one second of each
91+ other, and nothing followed from either for the 15 s.
92+ - ** What is known.** ` virt_smp ` , the same case launched one second before
93+ ` virt_el1_smp ` , did the step in the same seconds and passed: the read began
94+ at guest 13.936 and ended at 19.120, five seconds of the host's clock.
95+ ` virt_failed_ap_leaves_no_hole ` , launched one second after, ends at
96+ ` unmap_touch ` and does not wait on the read. Each red, run alone at the
97+ same commit, passed in 4 s, at loads 32.47 and 37.04.
98+ - ** What is not known.** Whether the guests ran, spun or were not scheduled:
99+ no register capture exists. Two guests going silent at the same step in the
100+ same second is the first occurrence that bears on the host stopping the
101+ guest, and it shows no more than that. The serial directory the harness
102+ named for that run was not kept; the run's log, outside the tree, is the
103+ only copy.
104+
105+ Two other stalls of ` virt_mask_windows ` have no console of the guest, so
106+ the step each stopped in is not read from it:
107+
108+ - ** ` wt/toyos-move-abi ` at ` 94fd25e71 ` .** `FAIL virt_mask_windows: STALLED:
109+ waiting for the boot's last word — it went quiet`, ten minutes after the
110+ two at ` 68cf1f870 ` by the two runs' logs, with nothing in what the
111+ guest said while it was waited on, ` STALL virt_mask_windows (38s) ` ; the
112+ whole suite at ` --jobs 4 ` , 34 of 36, workers 1794 s building against 703 s
113+ testing. Load 59.30 69.33 70.48 when the suite began and 85.73 69.02 67.23
114+ when it ended, 14 cores; fastest boot 2765 ms against the reference
115+ 1424 ms, ceilings at 1.94x. The wait is the second one, ended by
116+ ` GUEST_QUIET ` , which no host factor stretches. Known: ` unmap_touch ` had
117+ ended with exit 0, its line reaching the harness 15 s before the verdict,
118+ the guest named by elimination among the three of that case then running.
119+ In the same run ` virt_smp ` said its ` unmap_touch ` line at guest 21.786 and
120+ its ` counters_read ` line at 34.278, 13 s apart on the host's clock and 2 s
121+ inside the bound; ` virt_el1_smp ` , earlier in that run, 13.481 and 15.702.
122+ The branch changes nothing that guest's kernel or harness is built from.
123+ At ` c95d0ffc0 ` the test passed in the whole suite at load 12.82 rising to
124+ 36.76 and alone at 36.76 to 42.53, neither at the red's load. Not known:
125+ whether the guest was alive or what it did in those 15 s; the red run's
126+ kept serial directory holds eleven UART logs, each an x86-64 boot, and no
127+ PL011's. Whether the silence is ` virt_smp ` 's 13 s grown past 15 under the
128+ hooks' cost, or a guest that stopped. It was not run again at that commit,
129+ nor on ` main ` at that load.
130+ - ** ` wt/toyos-acpi1 ` at ` b9430ed61 ` .** The same line after 45 s, in the whole
131+ suite; workers 19038 s building against 1375 s testing, load 82.65 83.62
132+ 77.89 moments after it, 14 cores. Nothing in that branch runs on that
133+ boot. Alone at that load and commit it passed in 14 s. On ` main ` at
134+ ` d47b383cf ` , ` cargo test -- virt_mask_windows ` passed three times: alone at
135+ load 45 rising to 72 (10 s), and twice beside a whole suite at load 74 to
136+ 77 (38 s, with ceilings paid at 8.00x and 31 s between its image and its
137+ first guest line) and 75 to 67 (7 s).
138+
139+ Not known of any of them: whether the fork compiler's fault reaches
140+ ` test_rs_counters_read ` or the kernel under it.
141+ ` issues/the-forks-llvm-deletes-a-loops-exit-on-a-no-wrap-flag-scalar-evolution-gives-the-wrong-value.md `
142+ has LLVM's ScalarEvolution deleting a loop's exit on both architectures, and
143+ records every function but the ones it names as not measured.
60144
61145The slow ones, with registers: a probe that captured ` info registers -a ` over
62146QMP whenever the read had not ended 3 s after ` unmap_touch ` (` debug-slow.patch `
@@ -84,3 +168,6 @@ whether its CPUs went idle by a `sched:` line's absence; its diary
84168Exit: the cause of the silence is named from a capture of a silent guest
85169(registers over QMP before anything else touches it), and fixed with the
86170evidence, or shown to be the host stopping the guest.
171+
172+ Owner: the kernel's AArch64 bring-up, ` issues/toyos-runs-on-arm64.md ` , held
173+ by the orchestrator's next kernel worker.
0 commit comments