Repository navigation
reviewer.md and implementer.md carry the owner's ruling on tests: a T14 timing row is a test, and another OS's recording is an oracle only committed, hermetic and, for a timing, from the same machine - #848
Conversation
A test is hermetic, deterministic and makes one claim, in setup, act and assert under a name that says the claim; it reaches no internet and no other machine, compares against no other OS, and depends on no host speed, its only clock a hang ceiling. A guest test's outcome never depends on host scheduling or load: however loaded the host, no guest test goes red, and timing is valid only on metal (the owner's ruling of virtio_sound_counts, red under host load). A measurement - throughput, latency, a comparison against Ubuntu - is not a test: it is taken on a measurement-only branch or by the orchestrator on demand, posted as evidence, and never becomes a suite row. Guest tests run on demand, and CI's small tier holds only the tests that must break when something important breaks. reviewer.md's Tests bullet states the rule and makes a test or row a diff adds or changes against it a BLOCKER; its Guest tests bullet states the on-demand tier; and its two cut rules admit cutting a measurement, which "any other cut is a BLOCKER" otherwise forbade. implementer.md points at the rule and says where a measurement goes. Root CLAUDE.md and tests/CLAUDE.md are unchanged: nothing in either is false under the ruling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Review of #848 at 0e38ceb (merge base 819b308). Net: +23 −11, prompt files only. No production code, no tests. Host gate: EXIT=0 at 0e38ceb according to the body, and part 9 of the log ends "Host: 78 step(s), all green". No guest test is reached. Spot-checked against 819b308. Each matches its citation:
The facts in the audit are accurate enough. What it proposes is not, because items 2-6 and 10 rest on the broadened reading below. BLOCKER
NOTE
SEND BACK |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
The owner ruled that a timing row on the T14 is a test that can go red. reviewer.md's Tests now says so, and the hang-ceiling-only clock is a guest test's, no longer every test's. "No other machine" and "no other OS" go: the owner ruled neither, and whether a recorded Linux or host-libc fixture is allowed waits on his answer per use case. The CI-tier sentence goes from Guest tests, because no tier exists yet: issues/ci-runs-the-whole-guest-suite.md owns the reshape. The load rule is said once. implementer.md's bullet is a pointer, and the measurement-only-branch procedure no role prompt carried is dropped. issues/guest-tests-a-loaded-host-can-turn-red.md records the guest tests the audit found whose fixed wall-clock bounds a loaded host can spend, each marked unmeasured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The owner approved the fixture use-case table on #848 as proposed. The rule is the principle its rows share: a recording from another OS or a reference implementation is an oracle only when committed with its provenance and read hermetically; a timing held to Linux's only on the same machine; a policy another OS chose, or its copy of a fact a primary source states, never. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
The audit on #848 found tests whose oracle reviewer.md's Tests now refuses. They are two defects: - libc host tests that call the host's C library at run time as their oracle (strtonum, internet addresses, socket options, and, missed by the audit, modf/logb and strnlen). long_double.rs is not one: its oracle is compiler-builtins, pinned by the lockfile. - two T14 rows held to Linux's HWP request, a governor policy, and to Linux's definition-block count, a copy of what the firmware's tables state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
Review of #848 at e483415 (merge base a5242a4). Net: +131 −8. Production 0, tests 0. Prompts: +17 −8 across Earlier BLOCKERs
BLOCKER
NOTE
SEND BACK |
…hapes have files reviewer.md's Tests: another OS's or a reference implementation's output, recorded or live, is an oracle only when committed with its provenance and read hermetically, so a live host TCP peer is refused for a claim about TCP behaviour (approved row 10). A timing is held to another's only when read on the same machine, not only Linux's (row 2's reason is general). A policy another OS chose is never an oracle for what ToyOS's own setting must be, so row 6 (toyos-cpuvuln reproducing Linux's lines) and row 3 (the turbostat floor) stand while row 4 (the HWP request) is refused. guest-tests-a-loaded-host-can-turn-red: kind tooling (unmeasured), the connect-storm pace and the stop budget cited to the issues that already own them, the netcase load reds cited, the liveness ceiling given an exit, and a timing claim owed a METAL row rather than deletion. ci-runs-the-whole-guest-suite: a question with status owner, since the tier is the owner's to name. suite-rows-that-are-measurements-or-make-several-claims: audit items 6, 7, 10, 11, 12 and 20-25, with an owner and an exit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Review of #848 at 5ae1d43 (merge base a5242a4). Net: +179 −8. Production 0, tests 0. Prompts: +19 −8 across Earlier BLOCKERs
Earlier NOTEs, all addressed:
BLOCKERNone. NOTENone. LAND |
, #851 and #853, into consent tests/common/qemu.rs: Profile carries both Desktop and MetalNoX2apic (#841). userland/compositor/src/session.rs: the crate import takes main's list, without CURSOR_PX (#814), plus the branch's PROMPT_POLL_HANDLES. Cargo.lock is main's, re-resolved: grants, and filepicker's toyos-manifest and the build's toyos-desktop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
The owner's ruling on tests, written into the two prompts that already speak of tests; five issue files recording what the audit found, each with an owner and an exit; and an audit of every existing test and metal row against the ruling. This branch changes no test. The audit, and the fixture use cases the owner approved, are the second half of this body. Both open questions are answered (below, "Owner's answers").
What changed
.claude/agents/reviewer.md, Tests. A test is hermetic, deterministic and makes one claim, in setup, act and assert under a name that says the claim, and it reaches no internet. However loaded the host, no guest test goes red: a guest test's only clock is a hang ceiling, and timing is valid only on metal. A timing row on the T14 is a test, and it goes red past its bound (the owner's answer to the first open question). A measurement is a number held to no bound — throughput, latency, a comparison against Ubuntu. It is not a test: it is posted on the pull request as evidence and never becomes a suite row. Another OS's or a reference implementation's output, recorded or live, is an oracle only when it is committed with its provenance and read hermetically, and a timing held to another's only when it was read on the same machine. A policy another OS chose is never an oracle for what ToyOS's own setting must be, and its copy of a fact a primary source states is never an oracle (the owner's approval of the fixture table, written as the principle its 12 rows share rather than as a list). A test or row the diff adds or changes that breaks this is a BLOCKER.reviewer.md, Growth and Guest tests, the two cut rules. Both now let a measurement be cut. Without that, "any other cut is a BLOCKER" would block the deletions the audit proposes.reviewer.md, Guest tests, unchanged in substance. The sentence "guest tests run on demand, and CI's small tier holds only the tests that must break…" is not written. No such tier exists, so it would be false of the tree and give a reviewer nothing to check.issues/ci-runs-the-whole-guest-suite.mdrecords the ruling and owns the reshape; it is aquestionwithstatus: owner, since which tests belong in the tier is the owner's to name. Its exit changes rootCLAUDE.md's "The guest suite is theirguest / suitecheck" and this bullet in the same diff that declares the tier.issues/guest-tests-a-loaded-host-can-turn-red.md,kind: tooling, since every bound in it is a mechanism and a measurement owed, not an observed red. Each guest test the audit found whose fixed wall-clock bound a loaded host can spend (items 19, 27-32 below), and the liveness ceiling (34), with an exit for each: a test waits on its event under the scaled hang ceiling,kill_ends_every_wait'sHELDtiming claim is owed aMETALrow rather than deletion, and the liveness ceiling is priced on the host's speed while each test runs. It citesissues/a-netcase-boot-under-host-load-said-nothing-past-the-firmwares-screen-clears.mdfor the netcase reds under load.sched_stress's connect-storm pace and the stop's budget (33) are not recorded twice: it citesissues/timing-verdicts-ruled-off-qemu-have-no-metal-arm.mdandissues/a-woken-klogd-can-wait-seconds-on-an-idle-cpu-under-hvf.md, which own them.issues/suite-rows-that-are-measurements-or-make-several-claims.md. Audit items 6 and 7 (measurements asMETALrows), the one-claim splits of items 10, 11 and 12, and items 20-25 (guest binaries that make several claims), with an owner and an exit.issues/libc-host-tests-take-the-host-c-library-as-their-oracle.mdandissues/t14-rows-hold-toyos-to-linuxs-choice-or-copy.md. The reshapes the fixture rule asks for, as the orchestrator directed: one new file per defect, and no existing issue edited. They are two because the subjects are separate. One is five host tests of libc whose oracle is the host's C library at run time (case 8). The other is two T14 rows held to Linux's governor policy (case 4) and to Linux's copy of the firmware's definition-block count (case 5)..claude/agents/implementer.md, Measure, build, test. One bullet: "A test you write holds toreviewer.md's Tests." The rule is written once, inreviewer.md.CLAUDE.mdandtests/CLAUDE.mdare unchanged. Under the ruling as now written, none of their sentences is false. Root's "Timing and audio verdicts come only from metal" and "A QEMU test asserts … never how long something took" say what the T14-timing-row sentence does.origin/mainat a5242a4. It merged clean. Main deletedvirtio_sound_counts(item 26) and addedspawn_cost(item 6).Owner's answers
issues/suite-rows-that-are-measurements-or-make-several-claims.md.Gates
cargo run -- --ci host, at head 5ae1d43: EXIT=0, "Host: 79 step(s), all green". Full log: part 1, part 2, part 3, part 4, part 5, part 6, part 7, part 8. Home and temporary paths in the log are replaced by<worktree>,<dev>,$TMPDIRand~. It reads every tracked file throughsrc/sourcegate.rs, the issue files and the prompt paragraph included. This branch's own diff is two prompts and five issue files.No guest test is reached: the branch's own diff touches no test, source or manifest.
What I am unsure of
Item 16 is not refused.
long_double.rsis not a host-libc differential. Its oracle is compiler-builtins'__extenddftf2, pinned by the lockfile, so it is committed and hermetic and stays out of the libc issue. Item 8, which the orchestrator first named, istlb_shootdown_waits, a T14 timing row the table does not rule out.The audit missed two host-libc differentials. Filing the libc issue turned up
fparts_differential.rs(modf,logb) andtext_differential.rs'sstrnlen, both held to the host's C library at run time. The issue lists them, and this confirms the scope note below: the host-test pass was a search, not a full read.Scope of the audit. The audit covers:
METALrow;tests/common/metal.rs;MACHINE_TESTS/SCREEN_TESTSregistration;mainthat calls many checks;I did not read every host test for multiple claims, and a multi-claim host test missing from the list below is a gap in the audit, not a clean bill.
Approved: fixtures recorded from another OS or a reference implementation
The owner's answer to the second question: "depends. explain use cases. for example for timings only on the same machine." Each row below is a use case the audit found in the tree, or one ToyOS plausibly has, with allowed or not allowed and one line of reason. The owner approved every row: "Approve all as proposed." The Tests sentence in
reviewer.mdis the principle they share.issues/toyos-beats-linuxs-latency-on-the-t14.md(rtla timerlat),issues/no-program-measures-toyos-against-linux-on-one-machine.mdcounters_on_metal,tests/t14-linux/turbostat-*.txt(item 10)counters_on_metalagainsttests/t14-linux/hwp-request.txt(item 10)acpi_tables_loaded,T14_DEFINITION_BLOCKS(item 11)toyos-cpuvuln/src/tests.rswithfixtures/t14/andfixtures/t14.txt;issues/each-boot-prints-the-vulnerabilities-lines-linux-prints.mdtests/libc-arch/src/text_differential.rs(GLIBC)strtonum_differential.rs,internet_addresses.rs,socket_options.rs,long_double.rs(items 13-16)toyos-ssh/tests/fixtures/*.transcript(OpenSSH, seeded)netstack_streams,libc_sockets(item 18)bcachefs/tests/upstream_fixture.rs(bcachefs-tools v1.39.4,NOTICE)toyos-acpi/fixtures/Audit: what breaks the ruling, and a proposed action for each
Line numbers are at
origin/main819b308, the base the review checked.spawn_costarrived with the merge and is cited at the head.A. The CI tier
.github/workflows/guest.yml:65(cargo run -- --ci guest, called fromci.yml:54andnightly.yml:78;src/ci.rs:969guest): every ready pull request and the merge queue run the whole guest suite.issues/ci-runs-the-whole-guest-suite.md. Which tests belong in the tier is the owner's call.B. Timing thresholds and measurements as suite rows
Under the first ruling, a timing row on the T14 is a test that can go red. Items 2-5, 8 and 9 therefore conform, and the proposals from the earlier round to delete them are withdrawn.
src/metaltimings.rs:23-29(ceiling),tests/common/metal.rs:1127-1138and:1245-1259, withtests/metal/lenovo-20w0003amz.toml: every metal boot reds past twice the committed record ofcomplete_ms,panel_usandpanel_max_us. Keep: a T14 timing row.tests/toyos.rs:700-712tlb_shootdown_cost: p50 and p99 are held to the record. Keep: a T14 timing row.tests/toyos.rs:713-722latency_wake: cyclictest's p99 is held to the record. Keep: a T14 timing row.tests/toyos.rs:485-491wake_storm_cost, withwake_storm_cost.rs:237: 64 waiters cost at most 5× 16 waiters, inrdtsccycles. Keep: a T14 timing row. It skips where the instrument cannot resolve it (:225), which a T14 run is not expected to hit.tests/toyos.rs:606-619syscall_cost, andspawn_cost(headtests/toyos.rs:641,spawn_cost.rs): each asserts only that its numbers were printed. Each is a measurement as a suite row. Reshape: hold each to the T14 record, as rows 3 and 4 do. Otherwise delete the row and its binary together, since a binary no row runs is exercised by nothing. Recorded inissues/suite-rows-that-are-measurements-or-make-several-claims.md.tests/toyos.rs:633-650trace_record_cost: the cost is "read, not held", so it is a measurement. Buttrace_flood.rs:25-53also asserts the ring's accounting: the newest record read back, lost ≥ overwritten, one CPU's records. Reshape: name the row for the ring-accounting claim, and drop the cost print or hold it to the record. Recorded inissues/suite-rows-that-are-measurements-or-make-several-claims.md.tests/toyos.rs:620-632tlb_shootdown_waits(metal-only): a clock floor of half a staged 20 ms delay. Keep: a T14 timing row.tests/common/metal.rs:1145-1146(deadline_on_time,lockup_on_time,:324-354): each sampler fires within one period of its bound, derived fromQUANTUM_NSandHARD_LOCKUP_SAMPLE_NS. Keep: a T14 timing row.C. Comparisons against Linux, and rows asserting several claims
tests/toyos.rs:566-575counters, withcounters_on_metal:5102-5283andtests/t14-linux/. The row asserts several unrelated claims::5234);:5147-5171);:5249-5281).Reshape into one row per claim. Under the approved table, the turbostat floor stands as its own row (case 3), and the HWP comparison with Linux is ruled out (case 4): the row holds the kernel's own declared constant. Recorded in
issues/t14-rows-hold-toyos-to-linuxs-choice-or-copy.md(the HWP oracle) and inissues/suite-rows-that-are-measurements-or-make-several-claims.md(the split).tests/toyos.rs:5324-5326,:5361-5363acpi_tables_loaded: the number of definition blocks is held to Linux's 14. The same row also asserts the MTRR comparison across CPUs (:5364-5372), that nothing was refused, and the AML reads. Reshape:issues/t14-rows-hold-toyos-to-linuxs-choice-or-copy.md.tests/toyos.rs:685-694timer_calibration: two claims, that both clocks calibrated and that the TSC is within 1% of CPUID. Reshape: two rows, each named for its claim. Recorded inissues/suite-rows-that-are-measurements-or-make-several-claims.md.D. Host tests that compare against the host OS at run time
13-15.
tests/libc-arch/src/strtonum_differential.rs,internet_addresses.rs,socket_options.rs, withfparts_differential.rsandtext_differential.rs'sstrnlen, which the audit missed: libc against the host's C library at run time. Ruled out (case 8). Reshape each to a committed table (case 7) or to a specification oracle. Recorded inissues/libc-host-tests-take-the-host-c-library-as-their-oracle.md.16.
long_double.rs: misread in the audit. Its oracle is compiler-builtins'__extenddftf2, pinned by the lockfile, not the host's C library. Keep.E. Peers outside the guest
tests/toyos.rs:331-335(the comment inhttps_fetch): a planned T14 row against a server on the bench.issues/the-lan-is-not-yet-production-grade.md:88-90plans bench tests too. Not flagged: the ruling says internet, and a LAN bench peer is not the internet. Listed so the owner can say otherwise.netstack_streams,netstack_streams_e1000e,libc_sockets,https_fetch,netstack_socket_churn(tests/toyos.rs:314-335): the peer is the harness's own server on the host's loopback, reached through slirp.netstack_streamsandlibc_socketsuse the host kernel's TCP as that peer. Under case 10 they stand while that peer is only a byte pipe. Whether either asserts a TCP behaviour the host's TCP decides has not been read, so neither is listed as ruled out.F. Fixed guest-side ceilings under QEMU that the harness does not scale
usbd_spare.rs:46(CLAIM_RETURN, 2 s) and:228(2 s), run under QEMU byusbd_drives_the_spare. Recorded inissues/guest-tests-a-loaded-host-can-turn-red.md.G. Guest binaries that assert several unrelated claims
Each is a shared-boot member, so splitting one costs binaries, not boots. The owner confirmed the "one claim" reading. Every item is recorded in
issues/suite-rows-that-are-measurements-or-make-several-claims.md.allocator_stress.rs:3: twelve checks. Reshape: one binary per claim.std_process.rs:4: eight std process behaviours. Reshape: one per claim.std_sync.rs:109: mutex, condvar, park and rwlock. Reshape: one per primitive.sched_stress.rs:10-23: io_uring acceptor isolation, the post-wake vruntime lag bound, and a connect storm. Reshape: three binaries.toybox_file_tools.rs:130:cp,mvandhexdump. Reshape: one per tool.std_fs.rs:7: listing, a self-read, NotFound, and file types. Reshape: split offfile_types.Keep, with reason: each of the following makes one claim over a family of crafted cases, and names each case in its failure:
abuse_elf_loader.rs;mmap_prot.rs;fs_escape.rs;abuse_handle_table.rs;abuse_cwd_growth.rs.H. Guest tests that host load could turn red
Items 27-32 and 34 are recorded in
issues/guest-tests-a-loaded-host-can-turn-red.md, which cites the owners of 33. Each was found by reading its clock, and none of the in-guest bounds has been seen red under load: each is a mechanism, except 26.virtio_sound_counts: went red under load. Deleted onmainsince the last round.usbd_drives_the_spare: a claim back, and a halt or reset, each within a fixed 2 s. Unmeasured.kill_ends_every_wait(HELD= 1 s,:57, asserted at:197-202): a timing claim. Unmeasured.sched_stress(:121-125): the ±50 ms lag after a 10 ms sleep, Unmeasured: nobody has seen host steal push the lag past the bound. Its flat 50 ms connect-storm sleep at:146is already a premise ofissues/timing-verdicts-ruled-off-qemu-have-no-metal-arm.md, which the load issue cites.abuse_thread_table(RUN_BOUND10 s). Unmeasured.acpi_mediated_access(RELEASED5 s). Unmeasured.netstack_socket_churn(20 s), andnetstack_streamsandnetstack_streams_e1000e(60 s), plus the harness'sANSWERED(30 s). Unmeasured.klogdand every thread to park. A mechanism, already seen under HVF, owned byissues/timing-verdicts-ruled-off-qemu-have-no-metal-arm.mdandissues/a-woken-klogd-can-wait-seconds-on-an-idle-cpu-under-hvf.md, which the load issue cites rather than records again.issues/a-netcase-boot-under-host-load-said-nothing-past-the-firmwares-screen-clears.mdrecordsnetstack_socket_churnandnetstack_streamsred at boot under host load with ceilings priced at 1.00x. Given an exit in the load issue.Already metal-only, and so outside the load rule:
hda_client_stall,null_sink_client_exits,soundserver_log_stall,audio_idle_suspend,isa_lines,tlb_shootdown_waits,wake_storm_cost,counters_metal,cyclictest,syscall_costandspawn_cost(each inRUST_SKIP).Records outside this branch's fence
issues/timing-verdicts-ruled-off-qemu-have-no-metal-arm.md: lists timing properties owed a metal row. Under the first ruling, a T14 timing row is a test, so the issue stands as written.issues/each-boot-prints-the-vulnerabilities-lines-linux-prints.md: stands. Its host test against Linux's lines on recorded inputs is approved (case 6).Not tests, but in the host gate's path:
userland/doom/build.rs:163-170downloads an archive when its tree has none, and thehostjob'srustupstep reaches the network.🤖 Generated with Claude Code
https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C