diff --git a/CLAUDE.md b/CLAUDE.md index 676cd75cfe3..9763311b5f0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -70,7 +70,7 @@ Vendor firmware a device or CPU verifies by its maker's signature may be shipped - **A red test is a defect**: it is fixed, or deleted with its issue recording the commit that restores it; a flaky test is deleted at once, never re-run. A red seen only under load is no flake: it is a defect, recorded with the host's load. - **A high-risk change names its checks.** Security boundaries, the scheduler, the ABI, filesystems, devices, memory management, concurrency primitives: a negative control where a defect would otherwise land unseen — the *whole* change reverted onto the base the green arm was measured on — and an independent oracle where one exists: an external specification, a differential implementation, real hardware, a third-party checker, a formal model, or a recorded real failure. A second agent is not independence. - **Never truncate command output.** No `| head`, `| tail`, `| grep` to reduce it: long output runs in the background and is read from its file — `[N characters truncated]` means data was lost. -- **Leave the machine as you found it.** The development machine is shared: every agent stops what it started and never another's process, killing only by PID and waiting out a build that holds the global lock, and removes the scratch output it made once it no longer needs it. +- **Leave the machine as you found it.** The development machine is shared: every agent stops what it started and never another's process, killing only by PID and waiting out a build that holds a lock its own needs (a key's in the host's store or its worktree's, `src/buildlock.rs`), and removes the scratch output it made once it no longer needs it. ## Repository layout @@ -82,7 +82,7 @@ The root `Cargo.toml`'s `[workspace]` `members` and `exclude` lists account for - **Never degrade audible or visual quality** — even temporarily, even for a big win elsewhere — without the owner's explicit sign-off. - **Always be empirical.** Read actual output; run the code; investigate root causes instead of guessing. Every written number comes from a command that was run; an estimate or datasheet bound says so. - **Never put the owner's email or any other personal data in a network request or its headers**; any `User-Agent` is `toyos-build (https://github.com/ToyOSOrg/ToyOS)`. Nothing that identifies his machines or network goes into the tree, a commit message or anything posted on GitHub, and one that has to be referred to there is named by where it stands and its kind, with no character of it; `src/sourcegate.rs` names the shapes. -- **One agent, one worktree, one branch.** The primary checkout owns `rust/`, the rustup link and `main`, and is no workspace; `.claude/agents/implementer.md` makes a worktree and `orchestrator.md` removes it. Never make one with `git clone`, and never run `git submodule` in one: either fetches the fork's history again, a clone builds a toolchain that takes the rustup link, and `git submodule` writes `core.worktree` into the fork's shared config, which breaks git in the primary's `rust/`. +- **One agent, one worktree, one branch.** The primary checkout owns `rust/` and `main`, and is no workspace; `.claude/agents/implementer.md` makes a worktree and `orchestrator.md` removes it. Never make one with `git clone`, and never run `git submodule` in one: either fetches the fork's history again, and `git submodule` writes `core.worktree` into the fork's shared config, which breaks git in the primary's `rust/`. - **Commit freely on your branch.** `git commit -F `, never `-m`. An agent does what it wants with the commits it made and has not pushed, in a fork clone too. A remote branch of this repository other than `main` that one implementer owns is that implementer's to amend, rebase and force-push; a fork's branches are append-only. - **Never touch `main`.** It moves only through a merged pull request, by its required merge queue, and is protected — no push, force-push, deletion or bypass. A pull request's title and body become the merge commit's: write them as `main`'s record. A branch lands after a review against `.claude/agents/reviewer.md`. A modify/delete conflict is resolved by accounting for every hunk of the modified side, never by checking its headings survived. A merge that deletes a document also deletes every citation to it in the same merge, found by searching the bare name as well as the path. An ABI change lands with the work that needs it: every worktree builds the toolchain its own sources name, so branches that change the ABI run side by side. Every merge leaves `main`'s tip compiling. diff --git a/README.md b/README.md index 84db74bab8b..53175ad69d6 100644 --- a/README.md +++ b/README.md @@ -252,7 +252,8 @@ cargo run -- --build-only # build everything, boot nothing cargo test # boot the OS and run the integration suite ``` -The first run initializes submodules and bootstraps the custom Rust toolchain. +The first run initializes submodules and bootstraps the custom Rust toolchain +into `~/.cache/toyos`, where every checkout on the machine finds it. Later runs rebuild only what changed; a `std`-only change is a few seconds. ## Running it on real hardware diff --git a/examples/imgstat.rs b/examples/imgstat.rs index a3bda833daf..7297efcedb3 100644 --- a/examples/imgstat.rs +++ b/examples/imgstat.rs @@ -94,15 +94,9 @@ fn main() { } } -/// Which of the four things on ROOT an entry is. -/// -/// The order matters: `bin/rustc` is the toolchain's, not userland's. +/// Which of the three things on ROOT an entry is. fn group_of(name: &str) -> &'static str { - if name.starts_with("lib/") { - "hosted rustc lib/" - } else if name.starts_with("bin/rustc") { - "hosted rustc bin/" - } else if name.starts_with("bin/") { + if name.starts_with("bin/") { "userland bin/" } else if name.starts_with("share/") { "assets share/" diff --git a/issues/a-compiler-fixtures-git-commit-could-not-create-a-temporary-file.md b/issues/a-compiler-fixtures-git-commit-could-not-create-a-temporary-file.md index 8736ad7ce66..99923d0f5fb 100644 --- a/issues/a-compiler-fixtures-git-commit-could-not-create-a-temporary-file.md +++ b/issues/a-compiler-fixtures-git-commit-could-not-create-a-temporary-file.md @@ -6,9 +6,10 @@ opened: 2026-09-29 # A compiler fixture's git commit could not create a temporary file -`src/compiler.rs`'s `a_missing_primary_record_is_refused_and_builds_nothing` -went red in 1 of 200 full runs of the toyos-build lib test binary at -`e3a1cdc8`. Those runs went beside a `cargo test --workspace --exclude +`src/compiler.rs`'s `estate` fixture, which +`one_compiler_per_key_whichever_checkout_names_it` and every other compiler +and LLVM test build on, failed under a test since deleted in 1 of 200 full +runs of the toyos-build lib test binary at `e3a1cdc8`. Those runs went beside a `cargo test --workspace --exclude toyos-build` loop as host load, with the 1-minute load average at 62.82 for that run. The fixture's own git failed, not the code under test: @@ -22,5 +23,5 @@ that repository's object store failed with EINVAL. ## Exit condition -The EINVAL is traced to a cause the test can rule out by construction, and a -loop of 200 loaded runs shows the test green each time. +The EINVAL is traced to a cause the fixture can rule out by construction, and +a loop of 200 loaded runs shows every test that builds on it green each time. diff --git a/issues/a-compiler-key-reads-no-symbolic-link.md b/issues/a-compiler-key-reads-no-symbolic-link.md index 79076367f20..55e9487ce6f 100644 --- a/issues/a-compiler-key-reads-no-symbolic-link.md +++ b/issues/a-compiler-key-reads-no-symbolic-link.md @@ -6,17 +6,21 @@ opened: 2026-10-01 # A compiler key reads no symbolic link -`compiler::key` reads `compiler/`, `src/tools/`, `src/stage0` and -`Cargo.lock` through `sysroot::tree_identity` with `Links::Skipped`, so a -symbolic link there is in no key: retargeting one, or editing what it names -outside those trees, keeps the old compiler. Refusing a link there, as the -freestanding key does, refuses every compiler build: -`git -C rust ls-files -s compiler src/tools src/stage0 Cargo.lock` at fork -commit `aca5f527f` lists 5 entries of mode `120000`, all under `src/tools` -(clippy's and rust-analyzer's `LICENSE-APACHE` and `LICENSE-MIT`, and -rust-analyzer's `AGENTS.md`). Hashing a link's target text instead moves the -key of every compiler of a worktree's own once. +`compiler::key` reads the paths of `compiler::KEYED` through +`sysroot::tree_identity` with `Links::Skipped`, so a symbolic link there is +in no key: retargeting one, or editing what it names outside those paths, +keeps the old compiler. Refusing a link there, as the freestanding key does, +refuses every compiler build: `git -C rust ls-files -s` over `KEYED`'s eleven +paths at fork commit `6d6ad8c7190` lists 5 entries of mode `120000`, all +under `src/tools` (`rustc_tools_util`'s and `lsp-server`'s `LICENSE-APACHE` +and `LICENSE-MIT`, and rust-analyzer's `AGENTS.md`). Hashing a link's target +text instead moves the key of every compiler once. -**Exit**: `Links::Skipped` is deleted and the compiler key hashes a link's -target text, landed with the next change to `compiler.rs`'s `RECIPE`, which -moves every compiler key anyway. +Owner: the first step of +`issues/the-forks-pin-is-a-file-and-a-worktree-checks-no-fork-out.md` ("The +fork's objects are the store's"), which reads a key's fork parts as git tree +ids, where a link's target text is in the key by construction, and moves +every compiler key anyway; the orchestrator briefs it. + +**Exit**: `Links::Skipped` is deleted, and a test in which retargeting a link +under a keyed path moves the compiler's key. diff --git a/issues/a-compilers-build-reads-an-environment-and-a-cargo-configuration-its-key-does-not-name.md b/issues/a-compilers-build-reads-an-environment-and-a-cargo-configuration-its-key-does-not-name.md new file mode 100644 index 00000000000..2a2a8a01621 --- /dev/null +++ b/issues/a-compilers-build-reads-an-environment-and-a-cargo-configuration-its-key-does-not-name.md @@ -0,0 +1,43 @@ +--- +status: open +kind: tooling +opened: 2026-10-08 +--- + +# A compiler's build reads an environment and a cargo configuration its key does not name + +Two inputs of `compiler::build_in_fork` are outside the fork checkout, and +`compiler::key` reads neither: + +- **The caller's environment.** `toolchain::x_build_compiler` gives bootstrap + the whole environment of the process that asked, less `GITHUB_ACTIONS` and + `CI`, where the LLVM build gets `PATH` and `TMPDIR` alone (`llvm::clear`). + `RUSTFLAGS`, `CC`, `CXX`, `CARGO_*` and `MACOSX_DEPLOYMENT_TARGET` reach the + compiler's build; the C and C++ compilers a compiler's key names are the + ones its LLVM's key resolved in the cleared environment, which a `CC` in the + caller's makes another. +- **The worktree's cargo configuration.** Bootstrap runs cargo in + `/rust`, and cargo reads every `.cargo/config.toml` above its + working directory: the worktree's, and through its `include` the untracked + `.cargo/local.toml` in which `.claude/agents/implementer.md` has a fork clone + under edit listed. The compiler's workspace patches four crates to ToyOS + forks (`libloading`, `memmap2`, `stacker`, `getrandom`); a `local.toml` + redirecting one to a local clone builds a compiler from that clone under the + key of the one the lockfile names. The committed `config.toml` holds only + guest-target tables today, which a host-only build does not read. + +A compiler built under either is stored and served to every checkout naming +its key. By reading (`src/toolchain.rs`, cargo's documented configuration +search); no compiler was built under a differing environment or a +`local.toml` to measure it. Both are `main`'s shape since worktrees built +compilers of their own; the host's store widens who is served. + +Owner: the step of +`issues/the-forks-pin-is-a-file-and-a-worktree-checks-no-fork-out.md` that +builds from an export ("A pinned build reads an export"), which decides where +a compiler's build runs; the orchestrator briefs it. + +**Exit**: a compiler's build runs with the environment `llvm::clear` leaves +and in a directory no worktree's `.cargo` is above, or its key names what of +either it reads; and a host test in which a variable set in the caller's +environment does not reach the command the build runs. diff --git a/issues/a-finder-file-in-a-store-directory-panics-its-sweep.md b/issues/a-finder-file-in-a-store-directory-panics-its-sweep.md deleted file mode 100644 index 3ee324dca57..00000000000 --- a/issues/a-finder-file-in-a-store-directory-panics-its-sweep.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-09-30 ---- - -# A Finder file in a store directory panics its sweep - -`keystore::sweep_by` takes every entry of a store directory as `` or -`.`, so a `.DS_Store` the macOS Finder writes there names the key -`""`, and `buildlock::keyed_idle` then opens the lock directory itself: -`build lock: open /.git/toyos-build-locks/llvm/: Is a directory (os -error 21)`. It happened after an LLVM placement, whose product was whole, so -the next build went on; the sweep had removed nothing. - -**Exit**: a sweep passes over a store entry that names no key, with a test. diff --git a/issues/a-killed-keystore-writer-leaves-an-orphan-temp-in-target.md b/issues/a-killed-keystore-writer-leaves-an-orphan-temp-in-target.md deleted file mode 100644 index 1d16051be45..00000000000 --- a/issues/a-killed-keystore-writer-leaves-an-orphan-temp-in-target.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-09-29 ---- - -# A killed keystore writer leaves an orphan temp in `target/` - -`record_by` in `src/keystore.rs` writes a uniquely named temp beside the record -and renames it over. A writer killed between the write and the rename leaves -that temp behind, and because its name is unique nothing later overwrites it: one -orphan per kill. - -Exit condition: whatever next writes that record removes the orphans of its -name. diff --git a/issues/a-worktree-cannot-build-a-hosted-rustc-of-its-own.md b/issues/a-worktree-cannot-build-a-hosted-rustc-of-its-own.md deleted file mode 100644 index 29f1b4b9a5e..00000000000 --- a/issues/a-worktree-cannot-build-a-hosted-rustc-of-its-own.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-09-30 ---- - -# A worktree cannot build a hosted rustc of its own - -The ToyOS-hosted rustc is built by the primary checkout alone -(`src/toolchain.rs`'s `ensure`, under `Owner::Us`), from the primary's `rust/` -under the primary's `write_config`. A worktree's image with `hosted-rustc = -true` carries that one: refused by name when the worktree builds a compiler of -its own (`src/build.rs`, `env.sysroot.primary_compiler`), and taken without a word when -only the worktree's `write_config` differs, so the image then carries a rustc -another tree's recipe built. - -So no branch can put the hosted rustc it changes into a guest before it lands. -Carrying LLVM instead of Cranelift changes `write_config`'s hosted target and -the fork's `compiler/rustc_llvm`, and its guest test could run only after the -merge. - -**Exit**: a worktree whose `write_config`, or whose fork's `compiler/` or -`src/bootstrap`, differs from the primary's builds a hosted rustc of its own, -keyed as `src/compiler.rs` keys a compiler, and its image carries that one. diff --git a/issues/an-llvms-key-names-of-bootstrap-only-src-bootstrap.md b/issues/an-llvms-key-names-of-bootstrap-only-src-bootstrap.md new file mode 100644 index 00000000000..5f46f3a22b4 --- /dev/null +++ b/issues/an-llvms-key-names-of-bootstrap-only-src-bootstrap.md @@ -0,0 +1,28 @@ +--- +status: open +kind: tooling +opened: 2026-10-08 +--- + +# An LLVM's key names, of bootstrap, only `src/bootstrap` + +`llvm::key` reads the fork's bootstrap as the committed tree of +`src/bootstrap`. The bootstrap that builds an LLVM is also `src/build_helper`, +its path dependency (`src/bootstrap/Cargo.toml`), and the launchers +`toolchain::x_build_with` runs, `x` and `x.py`; a fork commit that moves only +one of them keeps the stored LLVM. A compiler's key reads all three +(`compiler::KEYED`); the LLVM's was left as it is because naming them moves +every LLVM key, which is one cold LLVM on each host (15:19 on an idle +development host, measured once) and three on CI. + +Read in the fork at `6d6ad8c7190`, not measured: no LLVM was built from a +fork whose `src/build_helper` differed. + +Owner: the first step of +`issues/the-forks-pin-is-a-file-and-a-worktree-checks-no-fork-out.md` ("The +fork's objects are the store's"), which reads every key's fork parts as git +tree ids and so moves every LLVM key anyway; the orchestrator briefs it. + +**Exit**: `llvm::key` names `src/build_helper`, `x` and `x.py` as it names +`src/bootstrap`, refused while one holds what no commit does, and a test in +which a commit moving each alone moves the key. diff --git a/issues/every-build-still-removes-an-llvm-no-build-makes-in-tree-any-more.md b/issues/every-build-still-removes-an-llvm-no-build-makes-in-tree-any-more.md index 8fc7f04ffff..73700366c44 100644 --- a/issues/every-build-still-removes-an-llvm-no-build-makes-in-tree-any-more.md +++ b/issues/every-build-still-removes-an-llvm-no-build-makes-in-tree-any-more.md @@ -7,18 +7,16 @@ opened: 2026-09-29 # Every build still removes an in-tree LLVM that no build makes any more `llvm::retire_in_tree` removes a bootstrap build directory's own `/llvm`, -`/lld`, `/ci-llvm` and `cache/llvm-*`. It runs at four sites on -every build: the primary's `reassemble`, `compiler::place`, `compiler::choose` -on the way back to the primary's compiler, and the std build -(`sysroot::prepare_std_build`). +`/lld`, `/ci-llvm` and `cache/llvm-*`. It runs at two sites: +`compiler::place`, and the std build (`sysroot::prepare_std_build`) on every +sysroot build. Since the LLVM store, no build makes any of these. Every compiler build links the store's `llvm-config`, and the std build sets `download-ci-llvm = false`. -What the four sites remove is what a build directory kept from before the +What the two sites remove is what a build directory kept from before the store. After a checkout's first build at that code, they remove nothing, but -they keep asking, on every build, a `read_dir` of `cache/` and a `stat` of -each name. +they keep asking, a `read_dir` of `cache/` and a `stat` of each name. -Exit: delete `retire_in_tree`, `in_tree` and the four call sites once no +Exit: delete `retire_in_tree`, `in_tree` and the two call sites once no checkout that builds holds a build directory made before the store. A host cannot know that for any other host, so the owner sets the date. diff --git a/issues/no-test-covers-the-pid-in-a-keystore-records-temp-name.md b/issues/no-test-covers-the-pid-in-a-keystore-records-temp-name.md deleted file mode 100644 index 91a406b4732..00000000000 --- a/issues/no-test-covers-the-pid-in-a-keystore-records-temp-name.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-09-29 ---- - -# No test covers the pid in a keystore record's temp name - -`record_by` in `src/keystore.rs` names its temp file -`...new` so that concurrent writers of one record never -share one. The counter distinguishes threads of one process; the pid is what -distinguishes two processes in one worktree, and nothing tests it: dropping -`std::process::id()` from the format leaves `cargo test --lib keystore` green -(exit 0, measured by that mutation and that command) -while two processes writing the same record share a temp name again. - -Exit condition: a test arm that races a child process against the parent on one -record, using the re-exec pattern of `rerun` in `src/buildlock.rs`, and goes red -when the pid is dropped from the format. diff --git a/issues/nothing-builds-the-toyos-hosted-rustc.md b/issues/nothing-builds-the-toyos-hosted-rustc.md new file mode 100644 index 00000000000..5791347533c --- /dev/null +++ b/issues/nothing-builds-the-toyos-hosted-rustc.md @@ -0,0 +1,35 @@ +--- +status: open +kind: tooling +opened: 2026-10-08 +--- + +# Nothing builds the ToyOS-hosted rustc + +No build makes a rustc that runs on ToyOS, and no config can ask for one. The +primary checkout's in-place build of it, the `hosted-rustc` key of +`system.toml` and the image's collection of its files went with the primary's +in-place compiler: the owner, of the in-place hosted rustc build and the rustup +link, "Drop both for now (Recommended)". No tracked config set the key. + +What that build left owed, which the one that replaces it owes too: + +- It was the primary's alone, built from the primary's `rust/` under the + primary's configuration, so no branch could put the hosted rustc it changes + into a guest before it landed, and a worktree whose configuration alone + differed shipped the primary's without a word. +- Nothing keyed it on std's sources: an edit to `sdk/std` or to the fork's + `library/` left a built one standing, and an image carried its `libstd-*.so` + beside the rlibs of the build's own sysroot. +- It was unmeasured since std's ToyOS backend left the fork for `sdk/std`: no + hosted rustc was built whose std reaches the backend through the fork's + `#[path]` arms. +- Nothing read its licences: a config that set the key was refused for that. + +Owner: `issues/toyos-builds-itself.md`, M3. + +**Exit**: a hosted rustc is a product of the host's store (`src/keystore.rs`), +keyed as `src/compiler.rs` keys a compiler and on the sources of the std it +runs on, and built by whichever checkout first names it; an image whose config +asks for it carries that one, with its licences read; and the `libstd-*.so` it +carries holds `sdk/std/sys/` paths and no `sys/pal/toyos`. diff --git a/issues/nothing-keys-the-hosted-rustc-on-stds-sources.md b/issues/nothing-keys-the-hosted-rustc-on-stds-sources.md deleted file mode 100644 index c14284e8b93..00000000000 --- a/issues/nothing-keys-the-hosted-rustc-on-stds-sources.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-10-07 ---- - -# Nothing keys the hosted rustc on std's sources - -Nothing keys the hosted rustc on std's sources. `src/toolchain.rs`'s -`hosted_rustc_owed` reads a stamp and whether `bin/rustc` is there, and no std -source: an edit to `sdk/std`, or to the fork's `library/`, leaves a built -hosted rustc standing, and `src/build.rs`'s `collect_hosted_rustc` ships its -`lib/*.so`, the `libstd-*.so` that rustc itself runs on among them, beside the -rlibs of the build's own sysroot. The compiler's key, which is what forgets a -hosted rustc, reads the fork's `compiler/`, `src/tools`, `src/stage0` and -`Cargo.lock` and nothing of `library/`. - -It is also unmeasured since std's ToyOS backend left the fork for `sdk/std`: -no hosted rustc has been built whose std reaches the backend through the -fork's `#[path]` arms. Only the primary checkout builds one, in its own -`rust/` (`issues/a-worktree-cannot-build-a-hosted-rustc-of-its-own.md`), so -the branch that moved the backend could not; no tracked config sets -`hosted-rustc = true`, and the primary held no hosted `stage2` when the move -landed. By reading, bootstrap compiles `library/std` in place and the arms -resolve to the primary's `sdk/std`. - -Owner: the build system (`src/toolchain.rs`). - -**Exit:** on the primary at a `main` that carries the move, a build whose -config asks for the hosted rustc exits 0, and the `libstd-*.so` under -`rust/build/x86_64-unknown-toyos/stage2/lib` carries `sdk/std/sys/` paths and -no `sys/pal/toyos`. diff --git a/issues/record-reads-compiler-source-after-the-bootstrap-it-records.md b/issues/record-reads-compiler-source-after-the-bootstrap-it-records.md deleted file mode 100644 index 9300b15baea..00000000000 --- a/issues/record-reads-compiler-source-after-the-bootstrap-it-records.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-09-28 ---- - -# `record` reads `compiler/` as it stands after the bootstrap it records - -`compiler::record` (`src/compiler.rs`) is called once `reassemble` has finished -building `stage2`, and it computes `source(rust_dir)` at that point — the -`compiler/` tree, working diff and untracked files as they read *then*, not as -they read when the bootstrap it is recording began. A `compiler/` edit made -while that bootstrap was running (`x.py build` takes minutes) is folded into -the record even though `stage2` was built without it, so the next build sees -`primary_is_current` return true for a `stage2` that does not contain the edit, -and skips a bootstrap that is actually owed. - -Exit condition: `record` (or whoever calls it) captures `source(rust_dir)` -before the bootstrap starts, not after, and a test edits `compiler/` mid-build -(a `bootstrap` closure that writes a file before returning) and asserts the -next `primary_is_current` is false. - -Owner: whoever next touches `compiler::record` or `rebuild_compiler`. diff --git a/issues/rustc-llvm-cannot-build-for-a-toyos-host.md b/issues/rustc-llvm-cannot-build-for-a-toyos-host.md index 2fa571118d2..2aff884c593 100644 --- a/issues/rustc-llvm-cannot-build-for-a-toyos-host.md +++ b/issues/rustc-llvm-cannot-build-for-a-toyos-host.md @@ -20,7 +20,7 @@ run, builds it for `x86_64-unknown-toyos` wrongly three ways: The C++ runtime and the C library are ToyOS arms at existing dispatch sites, written as upstream would take them. The paths are no arm, and rustc does not -change for them: ToyOS's build hands the hosted rustc's build a host LLVM whose +change for them: ToyOS's build is to hand a hosted rustc's build a host LLVM whose paths hold the host triple where the ToyOS host's LLVM's hold `x86_64-unknown-toyos`, as bootstrap's own build directory lays the two out. diff --git a/issues/the-build-runs-host-tools-outside-rust-and-qemu.md b/issues/the-build-runs-host-tools-outside-rust-and-qemu.md index a0701d04800..437568af777 100644 --- a/issues/the-build-runs-host-tools-outside-rust-and-qemu.md +++ b/issues/the-build-runs-host-tools-outside-rust-and-qemu.md @@ -18,7 +18,7 @@ arrives and is not one. M4 and M5 are stages of `issues/toyos-builds-itself.md`. | CMake | rustc's bootstrap, for LLVM and clang; `src/llvm.rs`, for the LLVM's key; `src/libcxx.rs`, for the C++ runtime in every sysroot build | admitted: no Rust tool does the job, LLVM, clang and LLD are described in CMake, and upstream's only other descriptions are a GN overlay it does not support and a Bazel one | M5 runs it in the guest | | `sh` running LLVM's `config.guess`, and the POSIX tools and `cc` it runs | LLVM's CMake, whenever this host builds an LLVM, and the C++ runtime's, in every sysroot build, ask it the host's triple, unconditionally (`get_host_triple` in `rust/src/llvm-project/llvm/cmake/modules/GetHostTriple.cmake`, which runs `sh` by name) | refused: a Rust tool does the shell's part, brush 0.4.0: on the development host (macOS, arm64) `config.guess` printed `/bin/sh`'s triple under it, `arm64-apple-darwin27.0.0`, exit 0 each. The script runs `sed`, `uname`, `mktemp`, `grep`, `rm`, `rmdir` and `cc` there under either shell, and that `cc` is the `cc` rows'. Five of the other six are refused, uutils' doing each: under brush with sed 0.2.0, grep 0.2.0 and coreutils 0.12.0's `mktemp`, `rm` and `rmdir`, and nothing else on `PATH` but the host's `uname` and `cc`, it printed that triple, exit 0. `uname` is admitted: coreutils 0.12.0's answers `-p` with `unknown` where macOS's answers `arm`, and `config.guess` reads that as PowerPC, `powerpc-apple-darwin27.0.0`, exit 0 | CMake finds brush as its `sh`, uutils' `sed`, `grep`, `mktemp`, `rm` and `rmdir`, and a Rust `uname` that answers `-p` as the host's does; or M5 runs it in the guest | | `git` for worktrees, submodules, checkouts, fixtures and rustc's bootstrap | adds worktrees (`src/sysroot.rs`); updates submodules (`src/lib.rs`, `src/sysroot.rs`); fetches the fork from the primary's and checks it out (`src/sysroot.rs`); makes the tests' fixture repositories; runs inside rustc's bootstrap | admitted: no Rust tool does the job, gitoxide 0.85 adds, removes and prunes no worktree, updates no submodule, stages, resets and pushes nothing, checks out only a fresh clone and fetches a local path by spawning `git`; a fixture must be what `git` makes, and bootstrap runs `git` itself | M4 runs it in the guest | -| `git` for reads, a config write, a commit's paths written out, and clones and fetches over HTTPS | `rev-parse`, `merge-base`, `ls-tree`, `ls-files`, `cat-file`, `config --get-regexp`, `worktree list`, `status`, `diff`, `ls-remote` and `grep`, in the build system and its tests; `config --global --add safe.directory` in CI's containers; `checkout -- ` through an index of its own, which writes the C++ runtime's sources out of the LLVM commit into the stored LLVM (`src/llvm.rs`); every workflow's checkout | refused: a Rust tool does it, gitoxide 0.85, which reads refs, objects, the index, config, worktrees and status, adds a value to a config file and writes it (gix-config 0.58's `File::section_mut_or_create_new`, `SectionMut::push`, `File::write_to`), walks history, diffs, and lists, fetches and clones a remote over HTTPS; `grep` is a search of the files its index names; and gitoxide's CLI 0.59 (gix 0.88) wrote the runtimes' sources of LLVM `849da7d6` into an empty directory, each path's tree through `gix rev parse`, `gix index from-tree` and `gix free index checkout-exclusive`, exit 0 each: the 18759 files `git` writes there, byte for byte and mode for mode | those are gitoxide's | +| `git` for reads, a config write, a commit's paths written out, and clones and fetches over HTTPS | `rev-parse`, `merge-base`, `ls-tree`, `ls-files`, `cat-file`, `config --get-regexp`, `config --blob`, `status`, `diff`, `ls-remote` and `grep`, in the build system and its tests; `config --global --add safe.directory` in CI's containers; `checkout -- ` through an index of its own, which writes the C++ runtime's sources out of the LLVM commit into the stored LLVM (`src/llvm.rs`); every workflow's checkout | refused: a Rust tool does it, gitoxide 0.85, which reads refs, objects, the index, config, worktrees and status, adds a value to a config file and writes it (gix-config 0.58's `File::section_mut_or_create_new`, `SectionMut::push`, `File::write_to`), walks history, diffs, and lists, fetches and clones a remote over HTTPS; `grep` is a search of the files its index names; and gitoxide's CLI 0.59 (gix 0.88) wrote the runtimes' sources of LLVM `849da7d6` into an empty directory, each path's tree through `gix rev parse`, `gix index from-tree` and `gix free index checkout-exclusive`, exit 0 each: the 18759 files `git` writes there, byte for byte and mode for mode | those are gitoxide's | | `cc`, `c++` and `ar` on a Linux host, `build-essential` in CI's containers | rustc links every host binary through `cc`; `cc` and `c++` compile LLVM, clang, LLD and `rustc_llvm` (`src/llvm.rs` names both to bootstrap); `cc` compiles ring's C and assembly in every build of the build system itself, whose HTTP agent is rustls on ring (`src/release.rs`); `ar` archives what `cc::Build` compiles, ring's included | admitted: no Rust tool compiles C or C++, or takes rustc's host link; ring is the one TLS provider the tree takes (owner, 2026-10-02), over a provider in Rust alone | M5: no host in the loop | | the toolchain's own `clang`, `llvm-ar`, `rust-lld` and `llvm-config`, built from `ToyOSOrg/llvm-project` | rustc links every guest binary with `rust-lld`; `clang` compiles the C corpus (`tests/common/compile.rs`) and, with `llvm-ar`, doomgeneric through `cc::Build` (`src/clang.rs`); rustc's bootstrap asks `llvm-config` how to link LLVM | admitted: our fork's C++, which ToyOS can one day build and run; no Rust tool compiles C, `cc::Build` archives with an `ar`, bootstrap reads LLVM through `llvm-config`, and `CLAUDE.md` links everything with `rust-lld` | M5: no host in the loop | | `ovmf-generic`, `qemu-efi-aarch64` | the x86-64 and AArch64 UEFI firmware of CI's guest containers (`src/firmware.rs`), packaged by Debian apart from QEMU | admitted: QEMU's own firmware, and no Rust firmware does its job | the instrument's QEMU carries its own firmware | diff --git a/issues/the-build-system-does-not-compile-on-windows.md b/issues/the-build-system-does-not-compile-on-windows.md index f04e1982845..5a4ef6b3fa5 100644 --- a/issues/the-build-system-does-not-compile-on-windows.md +++ b/issues/the-build-system-does-not-compile-on-windows.md @@ -8,23 +8,30 @@ opened: 2026-08-19 `src/tether.rs`: `std::os::unix` and a pseudo-terminal per child, behind a Linux and macOS `cfg` pair with no Windows arm. -## The judge, and it needs no Windows host and no download +## The judge needs no Windows host, and stops at a dependency's C before it reaches the build system ``` -__CARGO_TESTS_ONLY_SRC_ROOT= CARGO_TARGET_DIR= \ - cargo +toyos check -Z build-std=std,panic_abort \ +__CARGO_TESTS_ONLY_SRC_ROOT=/src/library CARGO_TARGET_DIR= \ + RUSTUP_TOOLCHAIN= cargo check -Z build-std=std,panic_abort \ --target x86_64-pc-windows-msvc --offline -p toyos-build --all-targets ``` -`` is `src/CLAUDE.md`'s std-src-root recipe with one addition: a -workspace `Cargo.toml` whose members are `library/std`, `library/sysroot`, -`library/proc_macro`, `library/panic_abort` and `library/test`, and whose -`[patch.crates-io]` is `library/Cargo.toml`'s four entries with `library/` -prepended to each path. It works because the fork vendors `library/windows-sys` -and `library/windows_link`, so a Windows `std` builds from the tree — a plain -`cargo check --target x86_64-pc-windows-msvc` instead says *"the -`x86_64-pc-windows-msvc` target may not be installed"* and asks for -`rustup target add`. It resolves crates.io through the cargo cache. +`` and `` are `src/CLAUDE.md`'s std type-check recipe's, with +no addition: the copied library is its own workspace, with its patches and +its lockfile, and a manifest written above it does not resolve offline. The +fork vendors `library/windows-sys` and `library/windows_link`, so a Windows +`std` builds from the tree — a plain `cargo check --target +x86_64-pc-windows-msvc` instead says *"the `x86_64-pc-windows-msvc` target may +not be installed"* and asks for `rustup target add`. It resolves crates.io +through the cargo cache. + +Run that way it builds `core`, `std` and `test` for Windows and exits 101 at +the 37th crate, before any source of `toyos-build` is checked: `failed to run +custom build command for ring v0.17.14`, whose build script compiles C for +the target with the host's `cc` and finds no `assert.h`. The judge reaches +`src/tether.rs` only with the target's C headers, the C runtime's and the +Windows SDK's, which no host here has and which are a download; or once +nothing `toyos-build` depends on compiles C for its target. ## Compiling is not working, and that is why the cheap half is refused @@ -32,9 +39,8 @@ Each wants a Windows call whose semantics differ in kind from the Unix one it replaces, and none can be run by anybody here: - `std::os::windows::fs::symlink_dir` needs the privilege or developer mode - Windows does not grant by default, so `link_host_target` and - `provision_toolchain_cargo` would compile and fail at run time — the quieter - kind of broken. + Windows does not grant by default, so `provision_toolchain_cargo` would + compile and fail at run time — the quieter kind of broken. - `flock` is advisory and whole-file; `LockFileEx` is mandatory and byte-range. `buildlock` is what serialises the shared sysroot across every worktree. diff --git a/issues/the-forks-pin-is-a-file-and-a-worktree-checks-no-fork-out.md b/issues/the-forks-pin-is-a-file-and-a-worktree-checks-no-fork-out.md new file mode 100644 index 00000000000..36a5998026e --- /dev/null +++ b/issues/the-forks-pin-is-a-file-and-a-worktree-checks-no-fork-out.md @@ -0,0 +1,64 @@ +--- +status: assigned +kind: track +opened: 2026-10-08 +--- + +# The fork's pin is a file, and a worktree checks no fork out + +**Owner**: the orchestrator, who briefs each step below as a branch of its +own. + +The `rust` gitlink becomes a pinned commit in a file, and a worktree that does +not edit the fork holds no checkout of it. The host's store +(`src/keystore.rs`) and every compiler in it came first; a worktree still +makes a fork checkout, 61,627 files and 399 MB as an export, to read its keys from, and still builds std in +it. + +What is left, each step landing with `main` working: + +- **The fork's objects are the store's.** A bare repository in the store + holds the fork, `library/backtrace` and `src/llvm-project`, each fetched by + commit, and every key's fork parts are git tree ids read from the pinned + commit there, with no checkout. The pin is still the gitlink. + *Exit*: a fresh worktree whose sources are `main`'s runs `cargo run -- + --build-only` with no `rust/` made in it and builds no compiler and no + sysroot. +- **A pinned build reads an export.** Std, the compiler and LLVM are built + from the pinned commit exported with no `.git`, beside copies of the + worktree's `toyos-abi`, `toyos` and `sdk/std`, and bootstrap is given its + source directory, so no build runs `git submodule`. A `rust/` that exists is + the agent's fork work and is built as it stands; every build with one that is + not the pin says so. + *Exit*: `issues/the-fork-checkout-runs-git-submodule-in-a-linked-worktree.md`'s + exit, and the sysroot such a build makes has the symbol tables of one built + in a checkout. +- **The pin is a file.** Owner: "Yes to the pin file". The repository's URL + and a 40-digit commit, in place of the gitlink and `.gitmodules`; `rust/` is + ignored; `fork_checkout`, `ensure_submodule`, `ensure_shallow_fork` and + `toolchain::Owner` go, and a runner keys and finds its sysroot as a dev host + does. A full commit id counts as pinned by hash (orchestrator). + *Exit*: the tree holds no gitlink, and root `CLAUDE.md` no longer forbids + `git clone` or `git submodule` in a worktree. +- **A pin off the fork's `main` is refused.** The gate fetches the fork's + `main` commits-only to a bounded depth, and can only say "within N of + `main`" (orchestrator). + *Exit*: `issues/seven-commits-of-mains-history-pin-a-rust-commit-no-repository-holds.md`'s. + +Measured once each, on a host whose load average was 20 to 35, at fork +`cc9c8b1be68`: + +- The fork by commit, trees only (`--depth=1 --filter=blob:none`): 2.09 MB in + 1.0 s, and every key's tree id is in it. With blobs: 49.3 MB in 12.1 s. + LLVM with blobs is about 280 MB. +- A trees-only repository fetches a blob on a plain read: reading + `.gitmodules` went to the network by itself. Either every git command on + the store carries the request's environment, or it runs with + `GIT_NO_LAZY_FETCH=1` and fetches blobs by id. +- The fork's `main` commits-only: 19.7 KB at depth 10, 13.6 MB at depth 50, + 83.1 MB at depth 200, since depth counts upstream's merge history too. +- Std built from an export with `--src` ran no git command, and its `libstd` + rlib had the 1,758 defined symbols of the one built in a checkout, none + differing in size, type or name. +- An export must carry no `git-commit-info` file, which makes bootstrap take + it for a tarball. diff --git a/issues/the-hosted-rustc-names-a-linker-toyos-does-not-have.md b/issues/the-hosted-rustc-names-a-linker-toyos-does-not-have.md index df4c842d09e..19be1fd6fcb 100644 --- a/issues/the-hosted-rustc-names-a-linker-toyos-does-not-have.md +++ b/issues/the-hosted-rustc-names-a-linker-toyos-does-not-have.md @@ -6,12 +6,12 @@ opened: 2026-09-26 # The ToyOS-hosted rustc names a linker ToyOS does not have -`x86_64-unknown-toyos` names `rust-lld`, and the rustc `src/toolchain.rs` -builds for a ToyOS host carries that target spec into the guest, where no -`rust-lld` exists: LLD runs on ToyOS only once clang and libc++ do, and no -other linker runs there. No image ships the hosted rustc -today — `src/build.rs` refuses `hosted-rustc = true` until its licences are -read — so nothing links through it yet. +`x86_64-unknown-toyos` names `rust-lld`, and a rustc built for a ToyOS host +carries that target spec into the guest, where no `rust-lld` exists: LLD runs +on ToyOS only once clang and libc++ do, and no other linker runs there. No +image ships a hosted rustc today, because nothing builds one +(`issues/nothing-builds-the-toyos-hosted-rustc.md`), so nothing links through +it yet. Exit: the hosted rustc links a program inside ToyOS through a linker the image carries, and a guest test compiles and runs one. diff --git a/issues/the-hosted-rustcs-stage2-carries-seventeen-proc-macro-libraries-no-image-needs.md b/issues/the-hosted-rustcs-stage2-carries-seventeen-proc-macro-libraries-no-image-needs.md index d4f017de8b3..0b5b42612b8 100644 --- a/issues/the-hosted-rustcs-stage2-carries-seventeen-proc-macro-libraries-no-image-needs.md +++ b/issues/the-hosted-rustcs-stage2-carries-seventeen-proc-macro-libraries-no-image-needs.md @@ -14,12 +14,8 @@ comment 5968053849). The Linux-hosted stage2 beside it holds `librustc_driver` alone, 594,483,472 bytes, which `issues/toyos-builds-itself.md` measures at 184.7 MB stripped. -`collect_hosted_rustc` (`src/build.rs`) puts every `lib/*.so` of that -directory, and `bin/rustc`, into ROOT as built. `hosted-rustc` is `false` in -`system.toml` and a config that sets it is refused until the compiler's -licences are read (`build::shipped`), so no image carries them today. #629 -deletes the function, the key and the refusal, and this paragraph goes in its -merge. +No image carries them today: nothing builds a hosted rustc +(`issues/nothing-builds-the-toyos-hosted-rustc.md`). Owner: `issues/toyos-builds-itself.md`, M3. diff --git a/issues/the-kernel-still-parses-what-userland-writes.md b/issues/the-kernel-still-parses-what-userland-writes.md index a538ba56576..4574801b669 100644 --- a/issues/the-kernel-still-parses-what-userland-writes.md +++ b/issues/the-kernel-still-parses-what-userland-writes.md @@ -30,7 +30,7 @@ have no bound at all; and two of those ceilings are *already* exceeded by artifacts this tree builds. **It has a deadline.** Nothing shipped is dynamically linked today, so the move -is pure deletion. The day `hosted-rustc` turns on, a very large shared object is +is pure deletion. The day an image ships a hosted rustc, a very large shared object is dlopened into a kernel whose cache never evicts, and every one of those bounds becomes load-bearing at once. Do it after the completion architecture lands and before that day. Independent of everything else; may run as soon as a slot frees. diff --git a/issues/the-primary-rebuilds-its-compiler-on-compiler-alone.md b/issues/the-primary-rebuilds-its-compiler-on-compiler-alone.md deleted file mode 100644 index 911ceab50a3..00000000000 --- a/issues/the-primary-rebuilds-its-compiler-on-compiler-alone.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -status: assigned -kind: tooling -opened: 2026-09-26 ---- - -# The primary rebuilds its compiler on `rust/compiler/` alone - -`src/toolchain.rs` rebuilds the primary's toolchain when the stamp over -`rust/compiler/` changes, and `compiler::record` writes that tree as the -compiler the primary's `stage2` is. A fork commit that moves only -`rust/src/bootstrap`, `rust/src/tools`, `rust/src/stage0`, `rust/Cargo.lock` -leaves the primary on the compiler it had, and a worktree whose `rust/compiler/` -matches the record is handed that compiler too. A worktree's own compiler is -keyed on all of them (`compiler::key`, PR #524), so the two answers to "which -compiler do these sources name" differ. - -**Owner**: the ARM64 track (`issues/toyos-runs-on-arm64.md`), whose -PR #524 made the worktree key; it is closed before that track's stage 4 -lands. - -**Exit condition**: the primary's rebuild and its record read the same sources -`compiler::key` does, and a worktree compares against that, shown by a test -in which a fork moving only `rust/src/tools` gets a compiler of its own. diff --git a/issues/the-primary-still-holds-the-toolchain-nothing-reads-any-more.md b/issues/the-primary-still-holds-the-toolchain-nothing-reads-any-more.md new file mode 100644 index 00000000000..411aed89ae1 --- /dev/null +++ b/issues/the-primary-still-holds-the-toolchain-nothing-reads-any-more.md @@ -0,0 +1,56 @@ +--- +status: assigned +kind: tooling +opened: 2026-10-08 +--- + +# The primary still holds the toolchain nothing reads any more + +Every LLVM, compiler and sysroot is a product of the host's store +(`src/keystore.rs`), and nothing was migrated into it. What the layout before +it left on a development host is read by nothing once the last worktree +still on that layout has merged `main` or gone, and no code removes it, +because no code knows whether such a worktree is still building: + +- **The primary's `rust/build`**, 20 G by `du -sh` on the day this was + filed: the in-place compiler and its build tree (`/`, `host`, + `bootstrap/`, `cache/`, `tmp/`, `lock`), the stores that were in it + (`llvm/`, `compilers/`, `freestanding/`, `sysroots/`), and + `toyos-compiler`, the record of the in-place compiler, which is a file. + `toyos-std/` there is still the primary's std build directory. +- **`toyos-compiler` being a file refuses the primary's first compiler + build**: `compiler::place` creates `rust/build/toyos-compiler/`, before it + empties anything, and the OS answers `File exists` on that path, naming no + cause. It bites the + first time the primary is the first checkout to name a compiler: a fork + bump built there, or a host tool update that moves the LLVM's key. +- **`toyos-build-locks/` in the primary's git directory**: the locks of the + stores above and the global lock. +- **The rustup toolchain `toyos`**, linked to the primary's in-place + `stage2`. Nothing rebuilds that compiler or its std, so `cargo +toyos` goes + on running the last ones built there without a word. + +**Owner**: the orchestrator, who knows which worktrees are left. + +A worktree that has not merged the store still builds with these: it reads +the record to find its compiler, keeps its products in the stores under the +primary's `rust/build` and takes its locks in `toyos-build-locks/`. So +nothing here is safe to +remove while `git worktree list` in the primary names a worktree whose +`src/keystore.rs` has no `pub fn host`, and none while a build runs in the +primary. Removing `toyos-compiler` alone before that, to let the primary +build a compiler, leaves each such worktree refused for want of the record +until it merges. Then, in the primary: + + chmod -R u+w rust/build && rm -rf rust/build + rm -rf "$(git rev-parse --git-common-dir)/toyos-build-locks" + rustup toolchain uninstall toyos + +The `chmod` is for the old LLVM store, which is read-only. The first build in +the primary afterwards makes `rust/build/toyos-std` again and finds +everything else in the store. + +**Exit**: in the primary, `test ! -e rust/build/toyos-compiler -o -d +rust/build/toyos-compiler`, `test ! -e rust/build/sysroots` and `test ! -e +"$(git rev-parse --git-common-dir)/toyos-build-locks"` each exit 0, and +`rustup toolchain list` prints no `toyos`. diff --git a/issues/the-primarys-compiler-record-is-written-by-truncate-then-write.md b/issues/the-primarys-compiler-record-is-written-by-truncate-then-write.md deleted file mode 100644 index 80ef1a72c99..00000000000 --- a/issues/the-primarys-compiler-record-is-written-by-truncate-then-write.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -status: open -kind: tooling -opened: 2026-09-29 ---- - -# The primary's compiler record is written by truncate-then-write - -`record` in `src/compiler.rs` writes `build/toyos-compiler` with `fs::write`, -which truncates and then writes. A linked worktree reads that file in -`primary_is` without the primary's lock, so a read between the two sees an empty -or partial record. It compares unequal to what the worktree's source names, and -the worktree builds a compiler of its own that the primary already has. - -Exit condition: the record is written whole or not at all (a temp beside it, -renamed over), as `keystore::record` does for its records. diff --git a/issues/toyos-builds-itself.md b/issues/toyos-builds-itself.md index 1172af237d2..02a51e6940b 100644 --- a/issues/toyos-builds-itself.md +++ b/issues/toyos-builds-itself.md @@ -130,7 +130,7 @@ measures it. **What M3 adds: a rustc that carries that LLVM**, after all of M2's. - Build: `issues/rustc-llvm-cannot-build-for-a-toyos-host.md`. - Link: `issues/a-rust-std-binary-cannot-link-the-cxx-runtime.md`. -- Test: `issues/a-worktree-cannot-build-a-hosted-rustc-of-its-own.md`. +- Test: `issues/nothing-builds-the-toyos-hosted-rustc.md`. M3's exit then waits on a linker in the guest (`issues/the-hosted-rustc-names-a-linker-toyos-does-not-have.md`). diff --git a/issues/toyos-runs-on-arm64.md b/issues/toyos-runs-on-arm64.md index 7257d925c4c..1f1f1ec64cf 100644 --- a/issues/toyos-runs-on-arm64.md +++ b/issues/toyos-runs-on-arm64.md @@ -216,8 +216,10 @@ Each stage names its exit; "measured" means a number from a run. 1. **`aarch64-unknown-toyos` in the rust fork.** A target spec: `aarch64-unknown-none-elf`, PIC, frame pointers. Std pal `_start` and TLS - for variant I. **Exit**: `cargo +toyos build --target aarch64-unknown-toyos` - builds `std` plus a hello-world and the whole Rust userland. + for variant I. **Exit**: `RUSTUP_TOOLCHAIN= cargo build --target + aarch64-unknown-toyos`, with `` as `src/CLAUDE.md`'s std + type-check recipe names it, builds `std` plus a hello-world and the whole + Rust userland. 2. **The UEFI loader on AArch64** (`aarch64-unknown-uefi`, tier 2 upstream, no fork work needed). Entry is `extern "C"` rather than `sysv64`. diff --git a/issues/two-builds-of-one-llvm-key-differ-in-their-bytes.md b/issues/two-builds-of-one-llvm-key-differ-in-their-bytes.md index d9540f0334b..755715a1bc6 100644 --- a/issues/two-builds-of-one-llvm-key-differ-in-their-bytes.md +++ b/issues/two-builds-of-one-llvm-key-differ-in-their-bytes.md @@ -7,19 +7,9 @@ opened: 2026-09-30 # Two builds of one LLVM key differ in their bytes `src/llvm.rs` holds that an LLVM is a function of its key. Two builds of one -key by `build_in_fork`, in two fork checkouts, differed in three things the key +key by `build_in_fork`, in two fork checkouts, differ in two things the key does not name: -- **The LLVM checkout's `origin`.** LLVM's CMake writes it into - `VCSRevision.h` as `LLVM_REPOSITORY` (`get_source_info` in - `llvm/cmake/modules/VersionFromVCS.cmake`), and clang and LLD put it in their - version (`clang/lib/Basic/Version.cpp`, `lld/Common/Version.cpp`), so in the - guest's bytes: LLD writes it into the `.comment` of the `libstd` a sysroot - links, and clang into that of each C and C++ object of its `c/lib/libc++.a`. - Bootstrap's `update_submodule` sets a checkout's origin from the fork's - `.gitmodules`, `ToyOSOrg`, when it moves the checkout to the gitlink's commit, - and leaves it when the checkout holds that commit already. A worktree's - checkout is cloned from the URL the fork's shared config names, `ToyOSOrg`. - **The build directory.** `lld`'s `LC_RPATH` and `llvm-config`'s object and source roots name the `build/toyos-llvm` of the checkout that built it. - **Archive dates.** The members of every static archive carry the time they @@ -30,12 +20,10 @@ which holds the LLVM fixed and varies rustc's inputs: every checkout on a host links the one LLVM its key names, so that gate sees no origin, and a comparison of two LLVM installs sees no panic path rustc writes. -**Exit**: `config_text` sets `LLVM_FORCE_VC_REPOSITORY` and -`LLVM_FORCE_VC_REVISION`, since the repository alone drops the revision, and a -host test that runs `llvm/cmake/modules/GenerateVersionFromVCS.cmake` with -`config_text`'s defines on two checkouts of one commit whose `origin`s differ -writes one header from both, where today each names its own origin. What only -a configure or a build writes, the build directory and the archive dates, a -nightly check sees: it builds one key in two fork checkouts at different paths -whose origins differ, checks after each build that they still do, and finds -every file of the two installs byte-identical. +The checkout's `origin` was the third, until `config_text` named the revision +and the repository (`llvm::stamp`): of that, only the configuration's text is +tested, and no test builds two checkouts whose `origin`s differ. + +**Exit**: a nightly check builds one key in two fork checkouts at different +paths whose origins differ, checks after each build that they still do, and +finds every file of the two installs byte-identical. diff --git a/issues/two-refusals-of-a-product-whose-sources-moved-have-no-test.md b/issues/two-refusals-of-a-product-whose-sources-moved-have-no-test.md new file mode 100644 index 00000000000..6f3c2fd07b5 --- /dev/null +++ b/issues/two-refusals-of-a-product-whose-sources-moved-have-no-test.md @@ -0,0 +1,24 @@ +--- +status: open +kind: tooling +opened: 2026-10-08 +--- + +# Two refusals of a product whose sources moved have no test + +`src/sysroot.rs`'s `build` and `build_freestanding` each read their key again +once the product is built and refuse to publish it when the sources moved +meanwhile, as `compiler::place` and `llvm::place` do. Those two have a test +whose stand-in build moves a source; these two run bootstrap inline +(`build_std`), so no test can stand in for it, and deleting either `assert!` +passes every test. A sysroot or freestanding libraries built while +`toyos-abi` or the fork's `library/` was edited would then be served to every +checkout naming the key of the sources before the edit. + +Owner: the step of `issues/the-forks-pin-is-a-file-and-a-worktree-checks-no-fork-out.md` +that rewrites the std build ("A pinned build reads an export"), which the +orchestrator briefs. + +**Exit**: `build` and `build_freestanding` take the std build they run as +`compiler::choose` takes its compiler build, and a test each, whose stand-in +edits a keyed source before it returns, reds when the refusal is deleted. diff --git a/rust b/rust index cc9c8b1be68..6d6ad8c7190 160000 --- a/rust +++ b/rust @@ -1 +1 @@ -Subproject commit cc9c8b1be686c6c01f70bda99e59fb743513923f +Subproject commit 6d6ad8c71906c4e7ddb67e34720f297704d6c282 diff --git a/src/CLAUDE.md b/src/CLAUDE.md index 2729d46d48f..4c69aebe497 100644 --- a/src/CLAUDE.md +++ b/src/CLAUDE.md @@ -13,11 +13,10 @@ Loads when you read a file under `src/` — the root cargo project, package name ## Worktrees -- Everything under a worktree — targets, images, `.build-locks/`, its fork checkout — is its own; the object stores, the compiler and the rustup link are the primary checkout's, and ownership is derived from `git rev-parse --git-common-dir`, never recorded. +- Everything under a worktree — targets, images, `.build-locks/`, its fork checkout — is its own; the object stores are the primary checkout's, and ownership is derived from `git rev-parse --git-common-dir`, never recorded. - **A linked worktree's `main` ref is only as current as the primary's last `git pull`: anything asking "does this branch differ from main" diffs against `origin/main`.** -- **Type-checking a std edit without building a sysroot**: point `__CARGO_TESTS_ONLY_SRC_ROOT` at a tree holding a copy of `rust/library`, a workspace `Cargo.toml` naming `library/std`, and symlinks to `toyos-abi`, `toyos` and `sdk` — through the `sdk` link the backend's two `#[path]`s back into the fork read `os_str.rs` and `common.rs` from the worktree's own `rust/library`, never the copy, so that checkout must exist and an edit to either is made there; then `CARGO_TARGET_DIR= cargo +toyos build -Z build-std=std,panic_abort --target x86_64-unknown-toyos --offline`. Delete `/**/.fingerprint/std-*` between runs — cargo does not re-fingerprint std under `-Zbuild-std`. +- **Type-checking a std edit without building a sysroot**: copy `rust/library` to `/src/library`, put symlinks to `toyos-abi`, `toyos` and `sdk` in ``, where std's `../../../` paths look for them, and point `__CARGO_TESTS_ONLY_SRC_ROOT` at the copy itself, whose own workspace and `Cargo.lock` cargo then resolves with — a manifest written above it has no lockfile and does not resolve offline; through the `sdk` link the backend's two `#[path]`s back into the fork read `os_str.rs` and `common.rs` from the worktree's own `rust/library`, never the copy, so that checkout must exist and an edit to either is made there; then, in a scratch crate outside the worktree, `RUSTUP_TOOLCHAIN= CARGO_TARGET_DIR= cargo build -Z build-std=std,panic_abort --target x86_64-unknown-toyos --offline`, where `` is the store's `sysroots/` and `` is what the worktree's `target/.deps-stamp` gives `x86_64-unknown-toyos` once it has built. Delete `/**/.fingerprint/std-*` between runs — cargo does not re-fingerprint std under `-Zbuild-std`. ## Caveats that bite every agent - **A workflow job that runs in a container adds `safe.directory` itself** — `actions/checkout` sets it into a temporary global config it discards when its step ends, so the first git command a container step runs after checkout dies on a dubiously-owned repository. -- **A `stage1-std//dist/deps` temp-dir error means a concurrent build**, never a broken checkout; never repair or force-rebuild the toolchain. diff --git a/src/build.rs b/src/build.rs index 8c8c90eba7e..c58297577dd 100644 --- a/src/build.rs +++ b/src/build.rs @@ -108,8 +108,6 @@ struct SystemConfig { #[serde(default)] symlinks: BTreeMap, #[serde(default)] - hosted_rustc: bool, - #[serde(default)] assets: Vec, /// What `/system/bin/supervisor` starts at boot. Program *keys*, never paths — a path /// here is a second spelling of a `[programs]` key and is what let a boot @@ -229,7 +227,6 @@ fn invalidate_stale( targets: &[PathBuf], ) { lock.act_if( - buildlock::Scope::Worktree, "clean crate targets against changed external deps", || { let work: Vec<(PathBuf, Stale)> = @@ -329,7 +326,7 @@ pub const PROFILE: &str = "toyos"; /// What every guest `cargo` and `rustc` here runs with: the toolchain directory /// of the sysroot this checkout's sources name, as `RUSTUP_TOOLCHAIN`, which -/// carries the linker too. Never the `toyos` rustup name, which is the primary's. +/// carries the linker too. struct GuestEnv { /// Owned, so the sysroot is held in use for as long as anything here runs /// against it. @@ -729,22 +726,6 @@ fn build_and_assemble( build_programs(root, config, env, quiet, arch, &mut root_files); root_files.push((toyos_manifest::PATH.to_string(), render_manifest(config))); - if config.hosted_rustc { - assert!( - arch == toolchain::HOSTED_ARCH, - "hosted-rustc is built to run on {}, and this image is for {}", - toolchain::HOSTED_ARCH.name(), - arch.name() - ); - assert!( - env.sysroot.primary_compiler, - "hosted-rustc ships the primary checkout's hosted compiler, and this worktree builds with \ - a compiler of its own (src/compiler.rs): the image would carry a rustc that is not the \ - one its programs were built with" - ); - collect_hosted_rustc(root, env.sysroot.dir(), &mut root_files); - } - if !config.assets.is_empty() { let programs: BTreeSet<&str> = config.programs.keys().map(String::as_str).collect(); root_files.extend(assets::collect(&config.assets, &programs)); @@ -759,11 +740,7 @@ fn build_and_assemble( .map(|(k, v)| (k.clone(), v.clone())) .collect(); - let mut programs: BTreeSet<&str> = config.programs.keys().map(String::as_str).collect(); - if config.hosted_rustc { - // `collect_hosted_rustc` puts it there and no row can. - programs.insert("rustc"); - } + let programs: BTreeSet<&str> = config.programs.keys().map(String::as_str).collect(); // Targets are inventoried beside the files: `bin/ls -> /system/bin/ghost` reaches a // program as surely as a file would, and the files alone walk past it. let targets: Vec = @@ -1102,21 +1079,12 @@ pub struct Shipped { } /// [`Shipped`], read out of the modes' configs the way [`build`] reads them. -/// -/// **A config that ships the hosted compiler is refused**: its dependencies are -/// the rust fork's `compiler/` workspace, which no reader of this answer walks. -pub fn shipped(root: &Path) -> Result { +pub fn shipped(root: &Path) -> Shipped { let mut crates = BTreeSet::new(); let mut programs = BTreeSet::new(); let mut assets = BTreeSet::new(); for boot in [Boot::shipped(root), Boot::diag(root), Boot::console(root)] { let config = parse_config(&boot.config); - if config.hosted_rustc { - return Err(format!( - "{} sets hosted-rustc, and nothing reads the licences of the compiler it ships", - boot.config.display() - )); - } for c in config_crates(root, &config) { if c.built == Built::Member { programs.insert(c.dir.clone()); @@ -1125,7 +1093,7 @@ pub fn shipped(root: &Path) -> Result { } assets.extend(config.assets.iter().map(|dir| root.join(dir))); } - Ok(Shipped { crates, programs, assets }) + Shipped { crates, programs, assets } } /// The parameters an image built for flashing may carry: the kernel's own boot @@ -1764,7 +1732,7 @@ fn shipped_parts(root: &Path, boot: &Boot, plan: &Plan) -> (Vec, Vec, Ve // this worktree's crate targets can land inside this build. let mut lock = buildlock::shared(root, "build"); let config = parse_config(&boot.config); - let env = GuestEnv::new(toolchain::ensure(root, &mut lock, config.hosted_rustc)); + let env = GuestEnv::new(toolchain::ensure(root, &mut lock)); invalidate_stale(&mut lock, &env.sysroot.identity, &[root.to_path_buf()]); @@ -1978,7 +1946,7 @@ pub fn build_test_parts( // back after the userland build, and a clean landing in between is the // same defect as one landing mid-compile. let mut lock = buildlock::shared(root, "test image"); - let env = GuestEnv::new(toolchain::ensure(root, &mut lock, config.hosted_rustc)); + let env = GuestEnv::new(toolchain::ensure(root, &mut lock)); invalidate_stale(&mut lock, &env.sysroot.identity, &[root.to_path_buf()]); @@ -2182,7 +2150,7 @@ struct TestBuild { impl TestBuild { fn begin(root: &Path, arch: Arch, what: &str, stale_targets: &[PathBuf]) -> Self { let mut lock = buildlock::shared(root, what); - let env = GuestEnv::new(toolchain::ensure(root, &mut lock, false)); + let env = GuestEnv::new(toolchain::ensure(root, &mut lock)); invalidate_stale(&mut lock, &env.sysroot.identity, stale_targets); let artifact = buildlock::artifact(root); TestBuild { target: arch.userland(), env, _lock: lock, _artifact: artifact } @@ -2202,67 +2170,6 @@ pub fn build_toyos_bin(root: &Path, arch: Arch, crate_path: &Path, name: &str, q // --- Internal helpers --- -/// The ToyOS-hosted rustc, and the target libraries it compiles against: this -/// build's own sysroot's, so the compiler on the image links what the image's -/// programs link. The hosted compiler itself is the primary's, read under the -/// lock its rebuild takes. -fn collect_hosted_rustc(root: &Path, toolchain: &Path, root_files: &mut Vec<(String, Vec)>) { - let _compiler = buildlock::compiler_shared(root, "reading the hosted rustc"); - let target = toolchain::HOSTED_ARCH.userland(); - let sysroot = toolchain::rust_dir(root).join(format!("build/{target}/stage2")); - assert!( - sysroot.exists(), - "Hosted rustc sysroot missing: {}", - sysroot.display() - ); - - let rustc = sysroot.join("bin/rustc"); - assert!( - rustc.exists(), - "Hosted rustc binary missing: {}", - rustc.display() - ); - root_files.push(("bin/rustc".to_string(), fs::read(&rustc).unwrap())); - - if let Ok(entries) = fs::read_dir(sysroot.join("lib")) { - for entry in entries.flatten() { - let path = entry.path(); - if path.extension().is_some_and(|e| e == "so") { - let name = path.file_name().unwrap().to_str().unwrap().to_string(); - let data = fs::read(&path).unwrap(); - root_files.push((format!("lib/{name}"), data)); - } - } - } - - let backends = sysroot.join(format!("lib/rustlib/{target}/codegen-backends")); - if backends.exists() { - for entry in fs::read_dir(&backends).into_iter().flatten().flatten() { - let path = entry.path(); - if path.extension().is_some_and(|e| e == "so") { - let name = path.file_name().unwrap().to_str().unwrap().to_string(); - let data = fs::read(&path).unwrap(); - root_files.push(( - format!("lib/rustlib/{target}/codegen-backends/{name}"), - data, - )); - } - } - } - - let rlibs = toolchain.join(format!("lib/rustlib/{target}/lib")); - for entry in fs::read_dir(&rlibs).unwrap_or_else(|e| panic!("read {}: {e}", rlibs.display())) { - let path = entry.unwrap_or_else(|e| panic!("read {}: {e}", rlibs.display())).path(); - if path.extension().is_some_and(|e| e == "rlib" || e == "rmeta") { - let name = path.file_name().unwrap().to_str().unwrap().to_string(); - root_files.push(( - format!("lib/rustlib/{target}/lib/{name}"), - fs::read(&path).unwrap(), - )); - } - } -} - #[cfg(test)] mod tests { use super::*; @@ -2324,7 +2231,7 @@ mod tests { #[test] fn every_modes_crates_and_the_supervisor_ship() { let root = Path::new(env!("CARGO_MANIFEST_DIR")); - let shipped = shipped(root).expect("the modes' configs"); + let shipped = shipped(root); for (dir, features) in [ ("userland/console", Features::Default), ("userland/supervisor", Features::Default), diff --git a/src/buildlock.rs b/src/buildlock.rs index efec388bd8c..95920dbee39 100644 --- a/src/buildlock.rs +++ b/src/buildlock.rs @@ -1,5 +1,5 @@ //! Serialising the build system's stateful phases across the builds running -//! against this repository. +//! on this host. //! //! Cargo's own build lock cannot do this job. `src/build.rs`'s `invalidate_stale` //! runs the `clean` that removes what a moved sysroot left stale of a `target/`, @@ -14,31 +14,25 @@ //! //! - **shared** — "I am building against the state as it stands". Any number //! at once. -//! - **exclusive** — "I am replacing it": the rust bootstrap, this worktree's -//! std build, the cleans of stale targets, the making or moving of its fork checkout. -//! One at a time, and never while a build holds the shared mode. +//! - **exclusive** — "I am replacing it": a compiler's, an LLVM's or a std's +//! build in this worktree's fork checkout, the cleans of stale targets, the +//! making or moving of that checkout. One at a time, and never while a build +//! holds the shared mode. //! -//! And two [`Scope`]s: a crate target directory is shared by the builds in one -//! worktree, while the primary's `rust/build` — the compiler every sysroot is -//! cloned from and compiled by — is shared by every worktree at once. A build -//! holds its worktree's lock shared for its whole length and the global one not -//! at all: it compiles against its own content-addressed sysroot -//! (`src/sysroot.rs`), which nothing rewrites. Only a sysroot being *made* -//! reads the compiler, and it holds [`compiler_shared`] while it does. -//! -//! A sysroot's own lock ([`keyed_building`], [`keyed_using`]) is per key, -//! so two worktrees with different ABIs never meet in it, and two with the same -//! one build it once. A compiler a worktree's fork checkout names apart from the -//! primary's (`src/compiler.rs`) is locked the same way under its own key, and -//! neither it nor a sysroot built from it takes the global lock. +//! A build holds its worktree's lock shared for its whole length. What it +//! shares with every other checkout on the host is the store +//! (`src/keystore.rs`), whose products nothing rewrites once they are placed, +//! each locked under its own key ([`keyed_made`], [`keyed_idle`]) in the store +//! itself: two worktrees with different ABIs never meet in a sysroot's lock, +//! and two with the same one build it once. A key's lock file is never +//! removed, and its modification time is when the key was last used or made. //! //! A compiler key's lock → a sysroot key's lock → a freestanding key's lock → the -//! worktree build lock → the global one → an LLVM key's lock → artifact. A -//! compiler's, a sysroot's or a freestanding key lock is taken with the -//! worktree lock put down ([`Held::without_shared`]), because the -//! key's builder takes the worktree lock exclusively; an LLVM key's is taken -//! inside the worktree or global lock covering the fork build directory its -//! builder writes. +//! worktree build lock → an LLVM key's lock → artifact. A compiler's, a +//! sysroot's or a freestanding key lock is taken with the worktree lock put +//! down ([`Held::without_shared`]), because the key's builder takes the +//! worktree lock exclusively; an LLVM key's is taken inside the worktree lock +//! covering the fork build directory its builder writes. //! //! Holder death: `flock` is released by the kernel when the open file //! description closes, so a builder that is SIGKILLed mid-phase — routine here @@ -74,31 +68,6 @@ const HEARTBEAT: Duration = Duration::from_secs(30); const HEARTBEAT: Duration = Duration::from_millis(300); const LOCK_DIR: &str = ".build-locks"; -/// Inside the git common directory: the one place every worktree of this -/// repository names identically, and one the build system never cleans. -const GLOBAL_LOCK_DIR: &str = "toyos-build-locks"; - -/// The one directory every worktree of this repository names identically. -fn git_lock_dir(root: &Path) -> PathBuf { - crate::git_common_dir(root).join(GLOBAL_LOCK_DIR) -} - -/// Which shared state a phase replaces, and so which lock has to serialise it. -/// -/// Stated at every call site rather than inferred, because the two are not -/// interchangeable in either direction: a toolchain phase taken in the worktree -/// scope serialises nothing across worktrees, and a target-directory clean -/// taken in the global scope stalls builds it has no business stalling. -#[derive(Clone, Copy, PartialEq)] -pub enum Scope { - /// State every worktree shares: the primary's `rust/` build tree — the - /// compiler every sysroot is made with — and the machine-global rustup link. - Global, - /// State this worktree alone owns — its crate target directories and its - /// fork checkout. Two worktrees cleaning their own have nothing to say to - /// each other. - Worktree, -} /// A held lock. Releasing it is closing the file. #[must_use] @@ -118,12 +87,9 @@ impl Drop for Guard { } /// The worktree's build lock, held in shared mode for the length of one build so -/// no clean of its crate targets lands inside it. The global lock is not held: -/// what a build reads of the shared tree is its own sysroot, which nothing -/// rewrites once it is made. +/// no clean of its crate targets lands inside it. pub struct Held { worktree_dir: PathBuf, - global_dir: PathBuf, what: String, /// `None` only while [`Held::without_shared`] has it put down, which is the /// whole reason this is an `Option`. @@ -137,7 +103,6 @@ pub struct Held { pub fn shared(root: &Path, what: &str) -> Held { let mut held = Held { worktree_dir: root.join(LOCK_DIR), - global_dir: git_lock_dir(root), what: what.to_string(), guard: None, }; @@ -146,8 +111,8 @@ pub fn shared(root: &Path, what: &str) -> Held { } impl Held { - /// Ask `decide`, and if it reports work, do that work under `scope`'s - /// exclusive lock. + /// Ask `decide`, and if it reports work, do that work under this worktree's + /// lock held exclusively. /// /// `decide` runs first under the shared lock this value holds, so a phase /// with nothing to do costs no serialisation at all. When it does report @@ -155,13 +120,8 @@ impl Held { /// asked **again**: whatever it saw a moment ago may have been done by the /// process that held the lock in between, and only this second answer is /// acted on. Serialising the action alone would still double-clean. - /// - /// The shared lock goes down whichever scope is escalated: holding it while - /// queueing for the global one is a deadlock with a process holding the - /// global one that wants this worktree's. pub fn act_if( &mut self, - scope: Scope, phase: &str, decide: impl Fn() -> Option, act: impl FnOnce(W), @@ -169,10 +129,7 @@ impl Held { if decide().is_none() { return; } - let dir = match scope { - Scope::Global => self.global_dir.clone(), - Scope::Worktree => self.worktree_dir.clone(), - }; + let dir = self.worktree_dir.clone(); self.without_shared(|| { let _exclusive = acquire(&dir, LOCK_EX, phase, BUILD); if let Some(work) = decide() { @@ -201,20 +158,6 @@ pub fn worktree_exclusive(root: &Path, what: &str) -> Guard { acquire(&root.join(LOCK_DIR), LOCK_EX, what, BUILD) } -/// The global lock in shared mode: "I am reading the primary's compiler". A -/// sysroot build holds it from its std compile to its clone of `stage2`, so a -/// toolchain rebuild does not land inside either. -pub fn compiler_shared(root: &Path, what: &str) -> Guard { - acquire(&git_lock_dir(root), LOCK_SH, what, BUILD) -} - -/// The global lock exclusively: what the primary holds, inside its worktree -/// lock held exclusively, while it moves its fork checkout, which its -/// [`Scope::Global`] phases build from with the worktree lock put down. -pub fn global_exclusive(root: &Path, what: &str) -> Guard { - acquire(&git_lock_dir(root), LOCK_EX, what, BUILD) -} - /// Exclusive lock over the shared cargo artifact paths. /// /// Cargo keys an artifact path on (crate, target, profile) and nothing else, so @@ -228,8 +171,7 @@ pub fn artifact(root: &Path) -> Guard { /// A content-addressed product of the host, locked per key: a sysroot, the /// freestanding targets' libraries it carries (`src/sysroot.rs`), a compiler -/// a worktree's fork checkout names (`src/compiler.rs`), or the LLVM a -/// compiler links (`src/llvm.rs`). +/// (`src/compiler.rs`), or the LLVM a compiler links (`src/llvm.rs`). #[derive(Clone, Copy)] pub enum Keyed { Sysroot, @@ -248,6 +190,11 @@ impl Keyed { } } + /// Where `store` keeps every product of this kind, one directory per key. + pub fn store(self, store: &Path) -> PathBuf { + store.join(self.dir()) + } + pub(crate) fn name(self) -> &'static str { match self { Keyed::Sysroot => "sysroot", @@ -260,15 +207,16 @@ impl Keyed { /// Make what `key` names: exclusive, and waited for by every other process that /// wants the same key, which then finds it made. -fn keyed_building(root: &Path, kind: Keyed, key: &Key) -> Guard { +fn keyed_building(store: &Path, kind: Keyed, key: &Key) -> Guard { let lock = format!("{} lock", kind.name()); - exclusive(&keyed_lock_path(root, kind, key), &lock, &format!("building {} {key}", kind.name())) + exclusive(&keyed_lock_path(store, kind, key), &lock, &format!("building {} {key}", kind.name())) } /// Use what `key` names: shared, so any number of builds use it at once, a -/// builder of it is waited for, and a sweep cannot remove it. -fn keyed_using(root: &Path, kind: Keyed, key: &Key) -> Guard { - let path = keyed_lock_path(root, kind, key); +/// builder of it is waited for, and a sweep cannot remove it; and dated now, +/// which a sweep reads as its last use. +fn keyed_using(store: &Path, kind: Keyed, key: &Key) -> Guard { + let path = keyed_lock_path(store, kind, key); let file = open_lock_file(&path); if !try_lock(&file, LOCK_SH) { let lock = format!("{} lock", kind.name()); @@ -278,6 +226,7 @@ fn keyed_using(root: &Path, kind: Keyed, key: &Key) -> Guard { announce(&lock, &what, &holder); take_lock_announcing(&file, LOCK_SH, &path, &lock, &what); } + file.set_modified(SystemTime::now()).unwrap_or_else(|e| panic!("build lock: date {}: {e}", path.display())); Guard { file, records_holder: false } } @@ -286,19 +235,19 @@ fn keyed_using(root: &Path, kind: Keyed, key: &Key) -> Guard { /// not. A `make` that leaves it not whole is refused by that defect rather than /// run again. pub fn keyed_made( - root: &Path, + store: &Path, kind: Keyed, key: &Key, defect: impl Fn() -> Option, mut make: impl FnMut(), ) -> Guard { loop { - let using = keyed_using(root, kind, key); + let using = keyed_using(store, kind, key); if defect().is_none() { return using; } drop(using); - let _building = keyed_building(root, kind, key); + let _building = keyed_building(store, kind, key); if defect().is_some() { make(); if let Some(defect) = defect() { @@ -310,13 +259,24 @@ pub fn keyed_made( /// What `key` names, exclusively and only if nobody is making or using it: what /// a sweep holds while it removes one. -pub fn keyed_idle(root: &Path, kind: Keyed, key: &Key) -> Option { - let file = open_lock_file(&keyed_lock_path(root, kind, key)); +pub fn keyed_idle(store: &Path, kind: Keyed, key: &Key) -> Option { + let file = open_lock_file(&keyed_lock_path(store, kind, key)); try_lock(&file, LOCK_EX).then_some(Guard { file, records_holder: false }) } -fn keyed_lock_path(root: &Path, kind: Keyed, key: &Key) -> PathBuf { - git_lock_dir(root).join(kind.dir()).join(key) +impl Guard { + /// Whether the key this holds was used or made less than `kept` ago. A key + /// nothing had locked before is dated by the lock that asks. + pub(crate) fn used_within(&self, kept: Duration) -> bool { + let used = self.file.metadata().and_then(|meta| meta.modified()).unwrap_or_else(|e| panic!("build lock: stat: {e}")); + !used.elapsed().is_ok_and(|unused| unused >= kept) + } +} + +/// The lock of `kind`'s `key`: in the store, where every checkout on the host +/// names it alike, and in none of its products. +pub(crate) fn keyed_lock_path(store: &Path, kind: Keyed, key: &Key) -> PathBuf { + store.join("locks").join(kind.dir()).join(key) } /// One lock file, taken exclusively and held until the guard drops. @@ -636,34 +596,13 @@ pub(crate) mod tests { Elsewhere::hold("buildlock::tests::child_role", &env) } - /// A [`Scope::Global`] phase of `root`, as a bootstrap holds it, in a process of its own. - pub(crate) fn global_phase_elsewhere(root: &Path) -> Elsewhere { - held_elsewhere(root, "global-phase") - } - - /// Whether an exclusive acquirer of `root`'s global lock is queued for it. - pub(crate) fn global_queued(root: &Path) -> bool { - !try_lock(&open_lock_file(&git_lock_dir(root).join("intent")), LOCK_SH) - } - /// What `role` of [`child_role`] takes in `root`, held by a process of its own. fn held_elsewhere(root: &Path, role: &str) -> Elsewhere { Elsewhere::hold("buildlock::tests::child_role", &[(ROLE, OsStr::new(role)), (ROOT, root.as_os_str())]) } - /// A git repository, because the global scope is keyed on the common - /// directory and a scratch tree that is not one would exercise a path the - /// build system never takes. fn scratch(name: &str) -> TempDir { - let dir = TempDir::new(&format!("buildlock-{name}")); - let ok = Command::new("git") - .args(["init", "-q"]) - .current_dir(&dir) - .status() - .expect("git init") - .success(); - assert!(ok, "git init in {}", dir.display()); - dir + TempDir::new(&format!("buildlock-{name}")) } fn worktree_lock_dir(root: &Path) -> PathBuf { @@ -740,12 +679,11 @@ pub(crate) mod tests { match role.as_str() { "hold-exclusive" => { let mut held = shared(&root, "child"); - held.act_if(Scope::Worktree, "child exclusive phase", || Some(()), |()| hold_until_released()); + held.act_if("child exclusive phase", || Some(()), |()| hold_until_released()); } "hold-exclusive-forever" => { let mut held = shared(&root, "child"); held.act_if( - Scope::Worktree, "child exclusive phase", || Some(()), |()| { @@ -761,7 +699,7 @@ pub(crate) mod tests { } "want-exclusive" => { let mut held = shared(&root, "child"); - held.act_if(Scope::Worktree, "queued exclusive phase", || Some(()), |()| note(&root, "ex")); + held.act_if("queued exclusive phase", || Some(()), |()| note(&root, "ex")); } "want-shared" => { let _held = shared(&root, "child"); @@ -784,10 +722,6 @@ pub(crate) mod tests { let _using = keyed_using(&root, Keyed::Sysroot, &Key::parse(&std::env::var(KEY).unwrap()).unwrap()); hold_until_released(); } - "global-phase" => { - let mut held = shared(&root, "child"); - held.act_if(Scope::Global, "child global phase", || Some(()), |()| hold_until_released()); - } "clean" | "clean-unlocked" => { touch(&root.join("cleaner-ready")); assert!(appeared(&root.join("builder-mid"), Duration::from_secs(20))); @@ -795,7 +729,6 @@ pub(crate) mod tests { if role == "clean" { let mut held = shared(&root, "child"); held.act_if( - Scope::Worktree, "clean the crate target", || target.exists().then_some(()), |()| fs::remove_dir_all(&target).unwrap(), @@ -858,70 +791,6 @@ pub(crate) mod tests { assert!(!try_lock(&third, LOCK_EX), "a clean got in while a build was running"); } - /// Two worktrees of one repository must name one global lock file and two - /// worktree ones. - /// - /// Getting either half backwards is silent — every build still runs. One - /// global file per worktree means the phases that replace the shared sysroot - /// stop excluding each other, which is the defect worktrees were introduced - /// without; one worktree file for all of them means a clean of a target - /// directory stalls builds that cannot see it. - #[test] - fn worktrees_share_the_global_lock_and_not_the_worktree_one() { - let root = scratch("worktrees"); - fs::write(root.join("f"), b"x").unwrap(); - git(&root, &["add", "f"]); - git(&root, &["commit", "-qm", "init"]); - let linked = root.join("wt"); - git(&root, &["worktree", "add", "-q", linked.to_str().unwrap(), "-b", "wt"]); - - let mine = shared(&root, "primary"); - let theirs = shared(&linked, "linked"); - assert_eq!( - mine.global_dir, theirs.global_dir, - "two worktrees disagree about where the global lock lives" - ); - assert_ne!( - mine.worktree_dir, theirs.worktree_dir, - "two worktrees share one target-directory lock" - ); - - // Naming one path is not yet excluding on it: `flock` conflicts between - // open file descriptions, so a second handle on the shared file is the - // question a second process would ask. A build compiles against its own - // sysroot and reads no compiler, so a toolchain rebuild waits for no - // build in either worktree; a sysroot being made reads the compiler, so - // the rebuild waits for that. - let global = open_lock_file(&git_common_lock_dir(&root).join("state")); - assert!(try_lock(&global, LOCK_EX), "a build kept the toolchain from being rebuilt"); - drop(global); - drop(theirs); - drop(mine); - let making = compiler_shared(&linked, "a sysroot build in the worktree"); - let global = open_lock_file(&git_common_lock_dir(&root).join("state")); - assert!( - !try_lock(&global, LOCK_EX), - "a toolchain rebuild could land inside a sysroot build in another worktree" - ); - drop(making); - } - - fn git_common_lock_dir(root: &Path) -> PathBuf { - crate::git_common_dir(root).join(GLOBAL_LOCK_DIR) - } - - fn git(dir: &Path, args: &[&str]) { - let ok = Command::new("git") - .args(["-c", "commit.gpgsign=false", "-c", "user.email=t@t", "-c", "user.name=t"]) - .args(crate::gitfixture::NO_AUTO_MAINTENANCE) - .args(args) - .current_dir(dir) - .status() - .expect("run git") - .success(); - assert!(ok, "git {args:?} in {}", dir.display()); - } - /// `flock` alone would let a stream of builds starve the rebuild they are /// all waiting for; the `intent` file is what stops that, and this is the /// gate on it. diff --git a/src/compiler.rs b/src/compiler.rs index 2377bbb6384..af2c78d9462 100644 --- a/src/compiler.rs +++ b/src/compiler.rs @@ -1,67 +1,71 @@ -//! The compiler a worktree's sysroot is cloned from and compiled by: the -//! primary's, or one of its own, content-addressed. +//! The compiler a worktree's sysroot is cloned from and compiled by, +//! content-addressed: one per key on this host, shared by every checkout whose +//! fork checkout names it. //! -//! **Every worktree builds with the compiler its own fork checkout names.** The -//! primary's `stage2` is built from what the primary's fork checkout holds, and -//! [`record`] writes which that is ([`source`]). A linked worktree whose fork -//! checkout names the same compiler compiles with that one. One whose compiler -//! sources differ — a new target spec, a codegen change — gets its own: built -//! by bootstrap in its own fork checkout, under that checkout's -//! `build/toyos-compiler/`, and placed at `rust/build/compilers//`, where -//! the key ([`key`]) is the identity (`src/identity.rs`) of the checkout's -//! [`KEYED`] sources, the key of the LLVM it links, which names -//! `src/bootstrap`, and the build itself ([`build_text`]). Nothing writes that -//! directory after its [`SOURCE`] file exists, and two worktrees naming the -//! same compiler share one copy. +//! **Every worktree builds with the compiler its own fork checkout names, and +//! no checkout's is special.** The key ([`key`]) is the identity +//! (`src/identity.rs`) of the checkout's [`KEYED`] sources, the key of the LLVM +//! it links, which names `src/bootstrap`, and the build itself +//! ([`build_text`]). A compiler nobody on the host has built is built by +//! bootstrap in the fork checkout that first names it, under that checkout's +//! `build/toyos-compiler/`, which starts from nothing unless the build and the +//! LLVM that filled it are this one's ([`place`]), and placed at +//! `compilers//` in the store (`src/keystore.rs`). Nothing writes that +//! directory after its [`SOURCE`] file exists. A sysroot is named by its +//! directory, never by a toolchain name, so no rustup toolchain is linked. //! //! **LLVM is the host's, built from `src/llvm-project`** (`src/llvm.rs`), and -//! linked through its `llvm-config`. [`source`] names that LLVM's key, so -//! another LLVM is another compiler, the primary's among them, and an LLVM -//! checkout or a `src/bootstrap` holding what no commit does names none. A -//! worktree records the LLVM its compiler links for as long as it builds with -//! that compiler. -//! -//! **A compiler of a worktree's own never touches what the others build with**: -//! not the primary's `stage2`, not its record, not the machine-global rustup -//! `toyos` link — a sysroot is named by its directory, never by a toolchain -//! name, so no link is made. The global lock is not taken either; nothing of -//! the primary's is read but the LLVM store, under its key's own lock. +//! linked through its `llvm-config`. The key names that LLVM's key, so another +//! LLVM is another compiler, and an LLVM checkout or a `src/bootstrap` holding +//! what no commit does names none. //! //! Locks, in the one order every acquirer takes them: the key's //! (`buildlock::keyed_*` with [`Keyed::Compiler`]), with this worktree's build -//! lock put down, held shared for as long as a sysroot is being made from it; +//! lock put down once the key is read under it ([`resolve`]), held shared for +//! as long as a sysroot is being made from it; //! then, to build, this worktree's exclusively, because its fork build //! directory is written; then the LLVM key's, held shared while it is linked. //! -//! A compiler no worktree names any more is removed by `keystore::sweep`, which -//! every placement runs: each build records the key it used in its -//! worktree's `target/`, and a key no registered worktree records, that nobody -//! is making or using, goes. -//! -//! What such a worktree's image cannot carry is the ToyOS-hosted rustc: that is -//! the primary's, built from the primary's `compiler/`, so `hosted-rustc` in a -//! worktree building with its own compiler is refused by name -//! (`src/build.rs`). +//! A compiler nothing has used for the store's keep time is removed by +//! `keystore::sweep`, which every placement runs. use std::fs; use std::path::{Path, PathBuf}; use crate::buildlock::{self, Guard, Held, Keyed}; use crate::keystore::{self, Key}; -use crate::sysroot::{clone_tree, git_bytes, git_out, short, tree_identity, Links}; +use crate::sysroot::{clone_tree, forget_another_compiler, tree_identity, Links}; use crate::toolchain::{self, host_triple}; /// What changes how a key's sources become a compiler and is neither them nor /// [`config_text`]: the build below. Moving it moves every key. -const RECIPE: &str = "bootstrap stage 2 of compiler/rustc and library, profile compiler, host only, with rust-lld, host linker pinned, LLVM, clang and LLD from the host's LLVM, no LLVM tool copied, rustc without debuginfo; 6"; - -/// What a compiler's key is the identity of, in its fork checkout. -const KEYED: [&str; 4] = ["compiler", "src/tools", "src/stage0", "Cargo.lock"]; - -/// What a compiler build is beyond its sources, as every key and record of one -/// reads it: [`RECIPE`], the configuration bootstrap is given ([`config_text`]) -/// with no path of this host in it, and the tools `clang::provision` puts beside -/// the compiler. +const RECIPE: &str = "bootstrap stage 2 of compiler/rustc and library, profile compiler, host only, with rust-lld, host linker pinned, LLVM, clang and LLD from the host's LLVM, no LLVM tool copied, rustc without debuginfo, no link to the checkout's sources; 7"; + +/// What a compiler's key is the identity of, in its fork checkout, beside the +/// `src/bootstrap` its LLVM's key names: what bootstrap compiles, which is +/// `library` too, for the host's std its `stage2` carries and every guest +/// crate's build scripts and proc macros link; the workspace it compiles them +/// in, with its profiles and patches; the version it gives rustc and the +/// channel file it reads beside it; and what runs bootstrap, with the crate +/// bootstrap itself is built with. +const KEYED: [&str; 11] = [ + "compiler", + "library", + "src/tools", + "src/stage0", + "Cargo.lock", + "Cargo.toml", + "src/version", + "src/ci/channel", + "src/build_helper", + "x", + "x.py", +]; + +/// What a compiler build is beyond its sources, as every key reads it: +/// [`RECIPE`], the configuration bootstrap is given ([`config_text`]) with no +/// path of this host in it, and the tools `clang::provision` puts beside the +/// compiler. fn build_text() -> String { let config = config_text(Path::new(""), &host_triple(), Path::new("")); let provisioned: Vec<&str> = crate::clang::tools().collect(); @@ -72,42 +76,32 @@ fn build_text() -> String { /// bootstrap builds from that commit, so its content is never read. pub(crate) const LLVM: &str = "src/llvm-project"; -/// Where a fork checkout builds a compiler of its own. +/// Where a fork checkout builds a compiler. const BUILD_DIR: &str = "build/toyos-compiler"; -/// The file a finished compiler carries last, naming what it was built from. A -/// directory without it is a build that did not finish. +/// The file a finished compiler carries last, naming its key. A directory +/// without it is a build that did not finish. const SOURCE: &str = "SOURCE"; +/// The links bootstrap puts in a `stage2` to the checkout that built it, where +/// rustup's `rust-src` and `rustc-dev` components would be. A stored compiler +/// carries neither: the checkout goes and the compiler stays, and rustc reads +/// one only to translate a library source's path to or from its remapped form +/// (`rustc_session`'s `real_source_base_dir`), which no build here asks for. +const CHECKOUT_LINKS: [&str; 2] = ["lib/rustlib/src/rust", "lib/rustlib/rustc-src/rust"]; + /// A compiler, held in use for as long as this lives. pub struct Compiler { /// Its toolchain directory: `bin/rustc`, `lib/`. pub stage2: PathBuf, - /// The file naming what it was built from. - record: PathBuf, - /// Whether it is the primary's, the one the hosted rustc and the rustup - /// link are built from. - pub primary: bool, - _using: Option, + key: Key, + _using: Guard, } impl Compiler { - /// The primary's `stage2`. - pub fn primary(rust_dir: &Path) -> Self { - Self { stage2: toolchain::stage2(rust_dir), record: primary_record(rust_dir), primary: true, _using: None } - } - - /// The compiler as a sysroot's key sees it: the source it was built from, - /// and the driver that build left, so a rebuild of the same source is a new - /// compiler too. + /// The compiler as a sysroot's key sees it: its key, and the driver its + /// build left, so one made again under its key is a new compiler too. pub fn identity(&self) -> String { - let source = fs::read_to_string(&self.record).unwrap_or_else(|_| { - panic!( - "{} is missing, so no sysroot can say which compiler it was built with.\n\ - The primary checkout writes the primary's: run `cargo run -- --build-only` there once.", - self.record.display(), - ) - }); let lib = self.stage2.join("lib"); let driver = fs::read_dir(&lib) .unwrap_or_else(|e| panic!("read {}: {e}", lib.display())) @@ -120,171 +114,49 @@ impl Compiler { .ok() .and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok()) .map_or(0, |d| d.as_nanos()); - format!("{} {} {} {mtime}", source.trim(), driver.file_name().to_string_lossy(), meta.len()) + format!("{} {} {} {mtime}", self.key, driver.file_name().to_string_lossy(), meta.len()) } - /// What a store built with this compiler is keyed by: its record, which - /// names what builds it and is known before it is built ([`primary_key`]). - pub fn key(&self) -> Key { - Key::of(&fs::read(&self.record).unwrap_or_else(|e| { - panic!("{} cannot be read ({e}), so nothing says which compiler a store is built with", self.record.display()) - })) - } -} - -/// [`Compiler::key`] of the primary's compiler `rust_dir`'s sources name, -/// whether or not it is built. -pub(crate) fn primary_key(rust_dir: &Path) -> Key { - Key::of(source(rust_dir).as_bytes()) -} - -/// The primary's record of which `compiler/` its `stage2` was built from. -pub(crate) fn primary_record(rust_dir: &Path) -> PathBuf { - rust_dir.join("build/toyos-compiler") -} - -/// Every compiler of a worktree's own on this host. -pub fn compilers_dir(rust_dir: &Path) -> PathBuf { - rust_dir.join("build/compilers") -} - -/// What `checkout`'s compiler is built from, as the primary records it: the -/// build ([`build_text`]), [`compiler_source`], and the key of the LLVM it links. -pub fn source(checkout: &Path) -> String { - source_with(checkout, &crate::llvm::key(checkout)) -} - -/// [`source`], with the key of the LLVM `checkout` names. -fn source_with(checkout: &Path, llvm: &Key) -> String { - format!("{}\n{} llvm {llvm}", build_text(), compiler_source(checkout)) -} - -/// What `checkout`'s [`KEYED`] sources are: each one's entry at its commit, and -/// whatever the working tree changes in them — an edit, or a file git does not -/// track yet, which is what a new target spec is before its commit. -fn compiler_source(checkout: &Path) -> String { - let in_keyed = |args: &[&'static str]| [args, &KEYED[..]].concat(); - let tree = git_out(checkout, &in_keyed(&["ls-tree", "HEAD", "--"])); - let mut local = git_bytes(checkout, &in_keyed(&["diff", "HEAD", "--"])); - let untracked = git_bytes(checkout, &in_keyed(&["ls-files", "-z", "--others", "--exclude-standard", "--"])); - for name in untracked.split(|b| *b == 0).filter(|n| !n.is_empty()) { - let path = checkout.join(String::from_utf8_lossy(name).as_ref()); - local.extend_from_slice(name); - local.push(0); - local.extend(fs::read(&path).unwrap_or_else(|e| panic!("read {}: {e}", path.display()))); - local.push(0); - } - if local.is_empty() { - tree.trim().to_string() - } else { - format!("{} with local changes {}", tree.trim(), short(&local)) - } -} - -/// Record which compiler the primary's `stage2` is. The primary calls this -/// after a toolchain build. -pub fn record(rust_dir: &Path) { - let at = primary_record(rust_dir); - let want = source(rust_dir); - if fs::read_to_string(&at).ok().as_deref() != Some(want.as_str()) { - fs::write(&at, &want).unwrap_or_else(|e| panic!("write {}: {e}", at.display())); - } -} - -/// Remove the record of which compiler the primary's `stage2` is. The primary -/// calls this before a toolchain build. -pub fn forget(rust_dir: &Path) { - let at = primary_record(rust_dir); - match fs::remove_file(&at) { - Err(e) if e.kind() != std::io::ErrorKind::NotFound => panic!("remove {}: {e}", at.display()), - _ => {} - } -} - -/// Whether the primary's `stage2` is the compiler `checkout` names — `Err` when -/// nothing records which compiler that is. -/// -/// **The source's content, never its files' times**: a checkout that rewrites a -/// file with the bytes it had is no new compiler. -fn primary_is(rust_dir: &Path, checkout: &Path, llvm: &Key) -> Result { - let names = source_with(checkout, llvm); - Ok(fs::read_to_string(primary_record(rust_dir))?.trim() == names) -} - -/// Whether the primary's `stage2` is the compiler its own fork checkout names: -/// false until a bootstrap has finished and [`record`]ed it. -pub fn primary_is_current(rust_dir: &Path) -> bool { - match primary_is(rust_dir, rust_dir, &crate::llvm::key(rust_dir)) { - Ok(current) => current, - Err(e) if e.kind() == std::io::ErrorKind::NotFound => false, - Err(e) => panic!("read {}: {e}", primary_record(rust_dir).display()), + /// What a store built with this compiler is keyed by. + pub fn key(&self) -> &Key { + &self.key } } /// The key of the compiler `fork`'s sources name: their content, and the build. pub fn key(fork: &Path) -> Key { - key_with(fork, &crate::llvm::key(fork)) -} - -/// [`key`], with the key of the LLVM `fork` names. -fn key_with(fork: &Path, llvm: &Key) -> Key { - key_of(fork, &build_text(), llvm) + key_of(fork, &build_text(), &crate::llvm::key(fork)) } -/// [`key`], with the build it reads. +/// [`key`], with the build it reads and the key of the LLVM `fork` names. fn key_of(fork: &Path, build: &str, llvm: &Key) -> Key { let parts = [build, &tree_identity(fork, &KEYED, Links::Skipped), llvm.as_str()]; Key::of(parts.join("\n\0\n").as_bytes()) } -/// The compiler `root`'s fork checkout at `fork` names: the primary's where its -/// `compiler/` is the one the primary's was built from, and otherwise its own, -/// built if nobody has built it, held in use for as long as the returned value -/// lives. -pub fn resolve(root: &Path, rust_dir: &Path, fork: &Path, lock: &mut Held) -> Compiler { - lock.without_shared(|| choose(root, rust_dir, fork, |fork| build_in_fork(root, rust_dir, fork))) +/// Why `dir` is not a compiler a build finished, if it is not: it carries no +/// [`SOURCE`]. +pub(crate) fn unplaced(dir: &Path) -> Option { + (!dir.join(SOURCE).is_file()).then(|| format!("{} carries no {SOURCE}", dir.display())) } -/// [`resolve`] with the build that makes a compiler's `stage2` passed in, so a -/// test can stand in for bootstrap: `build` compiles the fork checkout it is -/// given and returns the `stage2` it left there. -fn choose(root: &Path, rust_dir: &Path, fork: &Path, build: impl Fn(&Path) -> PathBuf) -> Compiler { - if fork == rust_dir { - return Compiler::primary(rust_dir); - } - let llvm = crate::llvm::key(fork); - let names_primary = primary_is(rust_dir, fork, &llvm).unwrap_or_else(|e| { - panic!( - "{} cannot be read ({e}), so nothing says which compiler the primary's stage2 is, \ - and no worktree can know whether it names that one.\n\ - The primary checkout writes it: run `cargo run -- --build-only` there once.", - primary_record(rust_dir).display(), - ) - }); - // The LLVM record follows the compiler's: a compiler links the LLVM its key - // names, and the primary's is recorded by the primary. - if names_primary { - keystore::forget(root, Keyed::Compiler); - keystore::forget(root, Keyed::Llvm); - let build = fork.join(BUILD_DIR); - if !crate::llvm::in_tree(&build).is_empty() { - let _worktree = buildlock::worktree_exclusive(root, "removing the LLVM its compiler build built"); - crate::llvm::retire_in_tree(&build); - } - return Compiler::primary(rust_dir); - } - let key = key_with(fork, &llvm); - let dir = compilers_dir(rust_dir).join(&key); - keystore::record(root, Keyed::Llvm, &llvm); - let using = keystore::made( - root, - Keyed::Compiler, - &compilers_dir(rust_dir), - &key, - || (!dir.join(SOURCE).is_file()).then(|| format!("{} carries no {SOURCE}", dir.display())), - || place(root, fork, &key, &dir, &build), - ); - Compiler { stage2: dir.join("stage2"), record: dir.join(SOURCE), primary: false, _using: Some(using) } +/// The compiler `root`'s fork checkout at `fork` names, in `store`: built if +/// nobody on the host has built it, and held in use for as long as the +/// returned value lives. +pub fn resolve(root: &Path, store: &Path, fork: &Path, lock: &mut Held) -> Compiler { + // Under the shared lock: a bootstrap of this worktree, which holds it + // exclusively, rewrites the fork's lockfiles for as long as it runs. + let key = key(fork); + lock.without_shared(|| choose(root, store, fork, key, |fork| build_in_fork(root, store, fork))) +} + +/// [`resolve`] of the compiler `key` names, with the build that makes its +/// `stage2` passed in, so a test can stand in for bootstrap: `build` compiles +/// the fork checkout it is given and returns the `stage2` it left there. +fn choose(root: &Path, store: &Path, fork: &Path, key: Key, build: impl Fn(&Path) -> PathBuf) -> Compiler { + let dir = Keyed::Compiler.store(store).join(&key); + let using = keystore::made(store, Keyed::Compiler, &key, || unplaced(&dir), || place(root, fork, &key, &dir, &build)); + Compiler { stage2: dir.join("stage2"), key, _using: using } } /// Build the compiler `key` names from `fork` and put it at `dir`. The caller @@ -292,14 +164,20 @@ fn choose(root: &Path, rust_dir: &Path, fork: &Path, build: impl Fn(&Path) -> Pa fn place(root: &Path, fork: &Path, key: &Key, dir: &Path, build: &impl Fn(&Path) -> PathBuf) { let what = format!("building compiler {key}"); let _worktree = buildlock::worktree_exclusive(root, &what); - eprintln!("Building compiler {key} in {}: its compiler/ is not the one the primary's was built from", fork.display()); + eprintln!("Building compiler {key} in {}: nobody on this host has", fork.display()); + let build_dir = fork.join(BUILD_DIR); + fs::create_dir_all(&build_dir).unwrap_or_else(|e| panic!("create {}: {e}", build_dir.display())); + // Bootstrap and cargo reuse what the directory holds whatever configuration + // and LLVM built it; only a source that moved do they see. + forget_another_compiler(&build_dir, &host_triple(), &format!("{}\n{}", build_text(), crate::llvm::key(fork))); let stage2 = build(fork); - crate::llvm::retire_in_tree(&fork.join(BUILD_DIR)); + crate::llvm::retire_in_tree(&build_dir); let partial = dir.with_extension("partial"); - if partial.exists() { - fs::remove_dir_all(&partial).unwrap_or_else(|e| panic!("remove {}: {e}", partial.display())); - } + keystore::remove(&partial); clone_tree(&stage2, &partial.join("stage2")); + for link in CHECKOUT_LINKS { + keystore::remove(&partial.join("stage2").join(link)); + } // The sources the key named are the ones built, or this is not that key's. let again = self::key(fork); assert!( @@ -315,12 +193,11 @@ fn place(root: &Path, fork: &Path, key: &Key, dir: &Path, build: &impl Fn(&Path) /// Bootstrap's build of the compiler in `fork`, into its own build directory, /// against the LLVM `fork` names (`src/llvm.rs`), and the `stage2` it made, /// with the cargo and the clang every toolchain directory carries. -fn build_in_fork(root: &Path, rust_dir: &Path, fork: &Path) -> PathBuf { +fn build_in_fork(root: &Path, store: &Path, fork: &Path) -> PathBuf { crate::ensure_submodule(fork, "library/backtrace"); - let llvm = crate::llvm::resolve(root, rust_dir, fork); + let llvm = crate::llvm::resolve(root, store, fork); let host = host_triple(); let build_dir = fork.join(BUILD_DIR); - fs::create_dir_all(&build_dir).unwrap_or_else(|e| panic!("create {}: {e}", build_dir.display())); let config = build_dir.join("bootstrap.toml"); fs::write(&config, config_text(&build_dir, &host, &llvm.dir)).unwrap_or_else(|e| panic!("write {}: {e}", config.display())); let config = config.to_str().unwrap_or_else(|| panic!("{} is not UTF-8", config.display())); @@ -336,10 +213,24 @@ fn build_in_fork(root: &Path, rust_dir: &Path, fork: &Path) -> PathBuf { stage2 } -/// Bootstrap's configuration for every compiler, the primary's and a -/// worktree's own, built in `build_dir`: for the host alone, since every guest -/// target's libraries are the sysroot's to build, linking the LLVM at `llvm`. -pub(crate) fn config_text(build_dir: &Path, host: &str, llvm: &Path) -> String { +/// What the host rustc links its own binaries with, pinned off: with +/// `lld = true` bootstrap otherwise makes `rust-lld` the default linker of +/// `x86_64-unknown-linux-gnu`, and of no other host. +const HOST_LINKER_PIN: &str = "default-linker-linux-override = \"off\""; + +/// The `[rust]` options of a compiler build beyond its profile's: no LLVM tool +/// copied into the compiler's sysroot, since `clang::provision` puts there the +/// ones a build runs; no debuginfo in rustc, which no build reads; and no +/// codegen test, for which bootstrap demands LLVM's `FileCheck` beside +/// `llvm-config` (`src/bootstrap/src/core/sanity.rs`). +const LEAN: &str = "llvm-tools = false\ndebuginfo-level-rustc = 0\ncodegen-tests = false"; + +/// Bootstrap's configuration for a compiler built in `build_dir`: for the host +/// alone, since every guest target's libraries are the sysroot's to build, +/// linking the LLVM at `llvm` (`clang::LLVM_CONFIG`, `llvm::host_lines`). +/// `lld = true` is what puts `rust-lld` in every stage's sysroot, where rustc +/// finds the linker every guest target names. +fn config_text(build_dir: &Path, host: &str, llvm: &Path) -> String { format!( r#"change-id = "ignore" profile = "compiler" @@ -355,16 +246,14 @@ target = ["{host}"] [rust] incremental = true lld = true -{lean} +{LEAN} [target.{host}] -{pin} +{HOST_LINKER_PIN} {external} "#, build_dir = build_dir.display(), llvm = crate::clang::LLVM_CONFIG, - lean = toolchain::LEAN, - pin = toolchain::HOST_LINKER_PIN, external = crate::llvm::host_lines(llvm), ) } @@ -377,6 +266,7 @@ pub(crate) mod tests { use std::process::Command; use super::*; + use crate::keystore::tests::{last_used, LONG_AGO}; pub(crate) fn git(dir: &Path, args: &[&str]) -> String { let out = Command::new("git") @@ -396,31 +286,16 @@ pub(crate) mod tests { fs::write(path, text).unwrap(); } - /// Every file under `dir` with its bytes, for "nothing here changed". - fn snapshot(dir: &Path) -> Vec<(PathBuf, Vec)> { - let mut out = Vec::new(); - let mut stack = vec![dir.to_path_buf()]; - while let Some(at) = stack.pop() { - for entry in fs::read_dir(&at).unwrap().flatten() { - let path = entry.path(); - if path.is_dir() { - stack.push(path); - } else { - out.push((path.clone(), fs::read(&path).unwrap())); - } - } - } - out.sort(); - out - } - /// The LLVM commits the fixtures' forks record. Nothing reads their content. pub(crate) const LLVM_A: &str = "1111111111111111111111111111111111111111"; pub(crate) const LLVM_B: &str = "2222222222222222222222222222222222222222"; - /// A primary whose `rust` pins fork commit `C0` and has built a compiler - /// from it, and three linked worktrees: `same` pins `C0`, `a` and `b` each - /// pin a commit whose `compiler/` is its own. + /// The URL the fixtures' forks name for their LLVM. + pub(crate) const LLVM_URL: &str = "https://llvm.invalid/llvm-project.git"; + + /// A primary whose `rust` pins fork commit `C0`, a store nothing is in + /// yet, and three linked worktrees: `same` pins `C0`, `a` and `b` each pin + /// a commit whose `compiler/` is its own. pub(crate) fn estate(scratch: &Path) -> (PathBuf, PathBuf, [PathBuf; 3]) { let base = fs::canonicalize(scratch).unwrap(); @@ -434,6 +309,7 @@ pub(crate) mod tests { write(&fork.join("x.py"), "\n"); write(&fork.join("library/std/src/lib.rs"), "pub fn a() {}\n"); write(&fork.join(".gitignore"), "/build\n"); + write(&fork.join(".gitmodules"), &format!("[submodule \"{LLVM}\"]\n\tpath = {LLVM}\n\turl = {LLVM_URL}\n")); git(&fork, &["add", "-A"]); git(&fork, &["update-index", "--add", "--cacheinfo", &format!("160000,{LLVM_A},{LLVM}")]); // What an uninitialised submodule leaves, so `commit -a` keeps the gitlink. @@ -457,9 +333,6 @@ pub(crate) mod tests { git(&primary, &["add", "-A"]); git(&primary, &["commit", "-qm", "pins C0"]); let rust_dir = primary.join("rust"); - write(&toolchain::stage2(&rust_dir).join("bin/rustc"), "the primary's rustc"); - write(&toolchain::stage2(&rust_dir).join("lib/librustc_driver-0.dylib"), "the primary's driver"); - record(&rust_dir); let mut linked = Vec::new(); for (name, pin) in [("same", c0.as_str()), ("a", pins[0].as_str()), ("b", pins[1].as_str())] { @@ -469,131 +342,153 @@ pub(crate) mod tests { git(&rust_dir, &["worktree", "add", "-q", "--detach", wt.join("rust").to_str().unwrap(), pin]); linked.push(wt); } - (primary, rust_dir, linked.try_into().unwrap()) + (primary, base.join("store"), linked.try_into().unwrap()) + } + + /// A compiler placed in `store` under the key of `name`: a driver, and no + /// more. + pub(crate) fn compiler(store: &Path, name: &str) -> Compiler { + let key = Key::of(name.as_bytes()); + let dir = Keyed::Compiler.store(store).join(&key); + let using = keystore::made(store, Keyed::Compiler, &key, || unplaced(&dir), || { + write(&dir.join("stage2/lib/librustc_driver-1.dylib"), "a driver"); + write(&dir.join(SOURCE), &format!("{key}\n")); + }); + Compiler { stage2: dir.join("stage2"), key, _using: using } } - /// **Two worktrees with different compilers build side by side, and the - /// primary's toolchain is untouched by either.** Each gets its own compiler - /// at its own key, built once and found again; one that names the primary's - /// `compiler/` builds nothing and gets the primary's; the primary's `stage2`, - /// its record and the rustup `toyos` link are byte-for-byte what they were; - /// a sweep takes a compiler only once no worktree names it. + /// **One compiler per key, whichever checkout names it.** The primary and a + /// worktree naming one `compiler/` share one compiler, built once and found + /// again; worktrees naming others each get their own, side by side; an + /// uncommitted file in `compiler/` is a new compiler and committing it is + /// not another; and a sweep takes a compiler only once nothing has used it + /// for the store's keep time and nobody uses it. #[test] - fn worktrees_with_different_compilers_coexist_and_the_primary_s_is_untouched() { + fn one_compiler_per_key_whichever_checkout_names_it() { let scratch = TempDir::new("compiler"); - let (primary, rust_dir, [same, a, b]) = estate(&scratch); - let before = snapshot(&rust_dir.join("build")); - let link = toolchain::rustup_link(); + let (primary, store, [same, a, b]) = estate(&scratch); + let compilers = Keyed::Compiler.store(&store); let builds = Cell::new(0); let fake = |fork: &Path| { builds.set(builds.get() + 1); fake_build(fork) }; - let mine = choose(&same, &rust_dir, &same.join("rust"), fake); - assert!(mine.primary && mine.stage2 == toolchain::stage2(&rust_dir)); - assert_eq!(builds.get(), 0, "a worktree naming the primary's compiler built one"); + let mine = chosen(&same, &store, &same.join("rust"), fake); + let primarys = chosen(&primary, &store, &primary.join("rust"), fake); + assert_eq!((&primarys.stage2, builds.get()), (&mine.stage2, 1), "one compiler/ named two compilers"); + for link in CHECKOUT_LINKS { + assert!(same.join("rust").join(BUILD_DIR).join("stage2").join(link).is_dir(), "the stand-in build made no {link}"); + assert!(fs::symlink_metadata(mine.stage2.join(link)).is_err(), "a stored compiler links the checkout that built it at {link}"); + } - let ca = choose(&a, &rust_dir, &a.join("rust"), fake); - let cb = choose(&b, &rust_dir, &b.join("rust"), fake); - assert_eq!(builds.get(), 2); - assert!(!ca.primary && !cb.primary); + let ca = chosen(&a, &store, &a.join("rust"), fake); + let cb = chosen(&b, &store, &b.join("rust"), fake); + assert_eq!(builds.get(), 3); assert_ne!(ca.stage2, cb.stage2, "two compilers were given one directory"); - assert!(ca.stage2.starts_with(compilers_dir(&rust_dir)) && cb.stage2.starts_with(compilers_dir(&rust_dir))); + assert!(ca.stage2.starts_with(&compilers) && cb.stage2.starts_with(&compilers)); assert!(fs::read_to_string(ca.stage2.join("bin/rustc")).unwrap().contains("aarch64")); assert!(fs::read_to_string(cb.stage2.join("bin/rustc")).unwrap().contains("riscv")); assert_ne!(ca.identity(), cb.identity()); - assert_ne!(ca.identity(), Compiler::primary(&rust_dir).identity()); + assert_ne!(ca.identity(), mine.identity()); // Found again, not rebuilt; and still both there. - let again = choose(&a, &rust_dir, &a.join("rust"), fake); - assert_eq!((again.stage2, builds.get()), (ca.stage2.clone(), 2)); + let again = chosen(&a, &store, &a.join("rust"), fake); + assert_eq!((again.stage2, builds.get()), (ca.stage2.clone(), 3)); assert!(ca.stage2.join("bin/rustc").is_file() && cb.stage2.join("bin/rustc").is_file()); // An uncommitted file in `compiler/` is a new compiler too, and // committing it is not another one. let pinned = git(&a.join("rust"), &["rev-parse", "HEAD"]); write(&a.join("rust/compiler/rustc_target/src/new_target.rs"), "pub fn t() {}\n"); - let ca2 = choose(&a, &rust_dir, &a.join("rust"), fake); + let ca2 = chosen(&a, &store, &a.join("rust"), fake); assert_ne!(ca2.stage2, ca.stage2, "an untracked target spec kept the old compiler"); git(&a.join("rust"), &["add", "-A"]); git(&a.join("rust"), &["commit", "-qm", "the target, committed"]); - let committed = choose(&a, &rust_dir, &a.join("rust"), fake); - assert_eq!((committed.stage2, builds.get()), (ca2.stage2.clone(), 3), "a commit rebuilt the compiler"); + let committed = chosen(&a, &store, &a.join("rust"), fake); + assert_eq!((committed.stage2, builds.get()), (ca2.stage2.clone(), 4), "a commit rebuilt the compiler"); git(&a.join("rust"), &["checkout", "-q", &pinned]); - // The primary's own: nothing under its `build/` but `compilers/` moved. - let after: Vec<_> = snapshot(&rust_dir.join("build")) - .into_iter() - .filter(|(p, _)| !p.starts_with(compilers_dir(&rust_dir))) - .collect(); - assert_eq!(after, before, "the primary's stage2 or its record was written"); - assert_eq!(git(&rust_dir, &["rev-parse", "HEAD"]), git(&primary, &["rev-parse", "HEAD:rust"])); - assert_eq!(toolchain::rustup_link(), link, "the machine-global toyos link moved"); - - // A sweep takes the compiler nobody names, and only that one — and - // not while it is still in use, though nobody names it any more. + // A sweep takes the compiler nothing has used for the keep time, and + // only that one — and not while it is still in use. let spec = a.join("rust/compiler/rustc_target/src/orphan.rs"); write(&spec, "pub fn o() {}\n"); - let orphan = compilers_dir(&rust_dir).join(key(&a.join("rust"))); - let user = chosen_elsewhere(&a, &rust_dir); + let unused = key(&a.join("rust")); + let orphan = compilers.join(&unused); + let user = chosen_elsewhere(&a, &store); fs::remove_file(&spec).unwrap(); - let kept = choose(&a, &rust_dir, &a.join("rust"), fake); - let sweep = |root: &Path, rust_dir: &Path| keystore::sweep(root, Keyed::Compiler, &compilers_dir(rust_dir)); - assert_eq!(sweep(&primary, &rust_dir), Vec::::new(), "the sweep took a compiler still in use"); - assert!(orphan.is_dir() && ca2.stage2.is_dir()); + last_used(&store, Keyed::Compiler, &unused, LONG_AGO); + let sweep = || keystore::sweep(&store, Keyed::Compiler); + assert_eq!(sweep(), Vec::::new(), "the sweep took a compiler still in use"); + assert!(orphan.is_dir()); user.release(); - drop(cb); - assert_eq!(sweep(&primary, &rust_dir), [orphan], "the sweep took a compiler a worktree names, or left one nobody does"); - assert!(kept.stage2.is_dir() && ca2.stage2.is_dir()); + assert_eq!(sweep(), [orphan], "the sweep took a compiler in use, or left one nothing had used for the keep time"); + assert!(ca.stage2.is_dir() && ca2.stage2.is_dir() && mine.stage2.is_dir()); - // A placement sweeps too: the compiler an edit replaces goes once nobody - // uses it, and the one another worktree names stays. + // A placement sweeps too: the compiler an edit replaces goes once + // nothing has used it for the keep time. let spec = a.join("rust/compiler/rustc_target/src/another.rs"); write(&spec, "pub fn u() {}\n"); - let replaced = compilers_dir(&rust_dir).join(key(&a.join("rust"))); - chosen_elsewhere(&a, &rust_dir).release(); - let named = choose(&b, &rust_dir, &b.join("rust"), fake).stage2; + let replaced = key(&a.join("rust")); + chosen_elsewhere(&a, &store).release(); + last_used(&store, Keyed::Compiler, &replaced, LONG_AGO); write(&spec, "pub fn v() {}\n"); - let ca3 = choose(&a, &rust_dir, &a.join("rust"), fake); - assert!(!replaced.exists(), "placing a compiler left the one it replaced, which nobody names"); - assert!(ca3.stage2.is_dir() && named.is_dir()); + let ca3 = chosen(&a, &store, &a.join("rust"), fake); + assert!(!compilers.join(&replaced).exists(), "placing a compiler left one nothing had used for the keep time"); + assert!(ca3.stage2.is_dir() && cb.stage2.is_dir()); } - /// **A worktree names the LLVM of the compiler it builds with, and only - /// that one**: one it placed or found placed, never one it has gone back to - /// the primary's from; and its build directory keeps no LLVM of its own - /// either way. + /// **A fork build directory whose compiler links the host's LLVM keeps + /// none of its own**: placing the compiler removes it, from a directory + /// this build and this LLVM filled too. #[test] - fn the_llvm_record_follows_the_compiler_in_use() { - let scratch = TempDir::new("compiler-llvm-record"); - let (primary, rust_dir, [_same, a, _b]) = estate(&scratch); + fn a_compiler_s_build_directory_keeps_no_llvm_of_its_own() { + let scratch = TempDir::new("compiler-in-tree-llvm"); + let (_primary, store, [_same, a, _b]) = estate(&scratch); let fork = a.join("rust"); - let store = crate::llvm::store(&rust_dir); - let llvm = store.join(crate::llvm::key(&fork)); - let sweep = || keystore::sweep(&primary, Keyed::Llvm, &store); - + drop(chosen(&a, &store, &fork, fake_build)); let own = fork.join(BUILD_DIR).join(host_triple()).join("llvm"); write(&own.join("bin/llvm-config"), "the build directory's own"); - drop(choose(&a, &rust_dir, &fork, fake_build)); + write(&fork.join("compiler/rustc_target/src/lib.rs"), "pub fn targets() { edited() }\n"); + let seen = Cell::new(false); + let fake = |fork: &Path| { + seen.set(own.is_dir()); + fake_build(fork) + }; + drop(chosen(&a, &store, &fork, fake)); + assert!(seen.get(), "the LLVM went before the build, with the directory: this test shows nothing"); assert!(!own.exists(), "a compiler built against the store left the LLVM its build directory built"); - fs::create_dir_all(&llvm).unwrap(); - assert_eq!(sweep(), Vec::::new(), "the LLVM of a worktree's own compiler was swept"); - - keystore::forget(&a, Keyed::Llvm); - let never = |_: &Path| -> PathBuf { panic!("a placed compiler was built again") }; - drop(choose(&a, &rust_dir, &fork, never)); - assert_eq!(keystore::recorded(&a, Keyed::Llvm), Some(crate::llvm::key(&fork)), "a placed compiler's LLVM went unrecorded"); - - git(&fork, &["checkout", "-q", &git(&rust_dir, &["rev-parse", "HEAD"])]); - write(&own.join("bin/llvm-config"), "the build directory's own, from before the store"); - assert!(choose(&a, &rust_dir, &fork, never).primary); - assert_eq!(sweep(), [llvm], "the LLVM of a compiler the worktree no longer builds with stayed"); - assert!(!own.exists(), "a worktree back on the primary's compiler kept the LLVM its build directory built"); + } + + /// **A build directory is built in again only by the build and the LLVM + /// that filled it**: a source edit keeps what the last build left there, + /// and a compiler linking another LLVM starts from nothing. + #[test] + fn a_build_directory_another_llvm_filled_starts_from_nothing() { + let scratch = TempDir::new("compiler-build-dir"); + let (_primary, store, [_same, a, _b]) = estate(&scratch); + let fork = a.join("rust"); + let left = fork.join(BUILD_DIR).join(host_triple()).join("stage1-rustc/left-by-the-last-build"); + let found = Cell::new(false); + let fake = |fork: &Path| { + found.set(left.is_file()); + write(&left, "what bootstrap would reuse"); + fake_build(fork) + }; + drop(chosen(&a, &store, &fork, fake)); + + write(&fork.join("compiler/rustc_target/src/lib.rs"), "pub fn targets() { edited() }\n"); + drop(chosen(&a, &store, &fork, fake)); + assert!(found.get(), "a source edit emptied the build directory, so no compiler build is incremental"); + + git(&fork, &["update-index", "--add", "--cacheinfo", &format!("160000,{LLVM_B},{LLVM}")]); + git(&fork, &["commit", "-qm", "another LLVM"]); + drop(chosen(&a, &store, &fork, fake)); + assert!(!found.get(), "a compiler linking another LLVM was built over what the last one left"); } const WORKTREE: &str = "TOYOS_COMPILER_TEST_WORKTREE"; - const RUST_DIR: &str = "TOYOS_COMPILER_TEST_RUST_DIR"; + const STORE: &str = "TOYOS_COMPILER_TEST_STORE"; /// The competing process for the test above: the compiler the worktree in /// [`WORKTREE`] names, chosen and held in use until released. @@ -602,94 +497,63 @@ pub(crate) mod tests { fn child_role() { let worktree = std::env::var(WORKTREE).unwrap_or_else(|_| panic!("child_role ran without {WORKTREE}; it is not a test")); let worktree = PathBuf::from(worktree); - let rust_dir = PathBuf::from(std::env::var(RUST_DIR).unwrap()); - let chosen = choose(&worktree, &rust_dir, &worktree.join("rust"), fake_build); - assert!(!chosen.primary, "the holder was given the primary's compiler, which holds no key"); + let store = PathBuf::from(std::env::var(STORE).unwrap()); + let _chosen = chosen(&worktree, &store, &worktree.join("rust"), fake_build); buildlock::tests::hold_until_released(); } /// The compiler `worktree` names, made if nobody has and held in use by a /// process of its own. - fn chosen_elsewhere(worktree: &Path, rust_dir: &Path) -> buildlock::tests::Elsewhere { - let env = [(WORKTREE, worktree.as_os_str()), (RUST_DIR, rust_dir.as_os_str())]; + fn chosen_elsewhere(worktree: &Path, store: &Path) -> buildlock::tests::Elsewhere { + let env = [(WORKTREE, worktree.as_os_str()), (STORE, store.as_os_str())]; buildlock::tests::Elsewhere::hold("compiler::tests::child_role", &env) } + /// [`choose`] of the compiler `fork` names. + fn chosen(root: &Path, store: &Path, fork: &Path, build: impl Fn(&Path) -> PathBuf) -> Compiler { + choose(root, store, fork, key(fork), build) + } + /// Bootstrap's stand-in: a `stage2` that says which target spec it knows. fn fake_build(fork: &Path) -> PathBuf { let stage2 = fork.join("build/toyos-compiler/stage2"); let spec = fs::read_to_string(fork.join("compiler/rustc_target/src/lib.rs")).unwrap(); write(&stage2.join("bin/rustc"), &format!("a rustc knowing {spec}")); write(&stage2.join("lib/librustc_driver-1.dylib"), &spec); + for link in CHECKOUT_LINKS { + let link = stage2.join(link); + fs::create_dir_all(link.parent().unwrap()).unwrap(); + let _ = fs::remove_file(&link); + std::os::unix::fs::symlink(fork, &link).unwrap(); + } stage2 } - /// **A primary with no record of its compiler is refused by name**, never - /// read as "no compiler": that reading made every worktree build its own. + /// **Only a compiler built from what its key names is stored**: one whose + /// sources moved while it was being built is refused, and nothing is at + /// the key the build began under. #[test] - fn a_missing_primary_record_is_refused_and_builds_nothing() { - let scratch = TempDir::new("compiler-record"); - let (_primary, rust_dir, [same, _, _]) = estate(&scratch); - fs::remove_file(primary_record(&rust_dir)).unwrap(); - let builds = Cell::new(0); - let fake = |fork: &Path| { - builds.set(builds.get() + 1); - fork.join("build/toyos-compiler/stage2") + fn a_compiler_whose_sources_moved_while_it_was_built_is_not_stored() { + let scratch = TempDir::new("compiler-moved"); + let (_primary, store, [_same, a, _b]) = estate(&scratch); + let fork = a.join("rust"); + let named = key(&fork); + let moving = |fork: &Path| { + let stage2 = fake_build(fork); + write(&fork.join("compiler/rustc_target/src/lib.rs"), "pub fn targets() { moved() }\n"); + git(fork, &["commit", "-qam", "moved while built"]); + stage2 }; - let refused = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { - choose(&same, &rust_dir, &same.join("rust"), fake); - })); - let why = refused.expect_err("a worktree resolved a compiler with no primary record"); - let why = why.downcast_ref::().cloned().unwrap_or_default(); - assert!(why.contains("toyos-compiler cannot be read"), "{why}"); - assert_eq!(builds.get(), 0, "a missing record built a compiler"); - } - - /// **The primary bootstraps when its `compiler/` holds other content, and - /// never because its files' times moved**: every file rewritten with its own - /// bytes is the compiler just recorded. - #[test] - fn only_the_compiler_s_content_makes_the_primary_bootstrap() { - let scratch = TempDir::new("compiler-current"); - let (_primary, rust_dir, _) = estate(&scratch); - assert!(primary_is_current(&rust_dir), "the compiler just recorded is not current"); - - let files = snapshot(&rust_dir.join("compiler")); - let later = std::time::SystemTime::now() + std::time::Duration::from_secs(3600); - for (file, bytes) in &files { - fs::write(file, bytes).unwrap(); - fs::File::options().write(true).open(file).unwrap().set_modified(later).unwrap(); - assert_eq!(fs::metadata(file).unwrap().modified().unwrap(), later); - } - assert!(!files.is_empty()); - assert!( - primary_is_current(&rust_dir), - "every file of compiler/ was rewritten with its own bytes, and the primary would bootstrap" - ); - - let spec = rust_dir.join("compiler/rustc_target/src/lib.rs"); - let held = fs::read(&spec).unwrap(); - write(&spec, "pub fn targets() { riscv() }\n"); - assert!(!primary_is_current(&rust_dir), "a change to compiler/ kept the old stage2"); - fs::write(&spec, held).unwrap(); - assert!(primary_is_current(&rust_dir), "compiler/ put back is not the compiler recorded"); - - write(&rust_dir.join("compiler/rustc_target/src/new_target.rs"), "pub fn t() {}\n"); - assert!(!primary_is_current(&rust_dir), "an untracked file in compiler/ kept the old stage2"); - fs::remove_file(rust_dir.join("compiler/rustc_target/src/new_target.rs")).unwrap(); - - fs::remove_file(primary_record(&rust_dir)).unwrap(); - assert!(!primary_is_current(&rust_dir), "a stage2 nothing recorded was taken for current"); - } - - /// Write the primary's record as it was written before its compiler linked - /// the host's LLVM: its `compiler/` and its LLVM commit. - pub(crate) fn record_before_the_store(rust_dir: &Path) { - fs::write(primary_record(rust_dir), format!("{} llvm {}", compiler_source(rust_dir), crate::sysroot::gitlink(rust_dir, LLVM))).unwrap(); + let said = refusal("a compiler whose sources moved while it was built was stored", || { + chosen(&a, &store, &fork, moving); + }); + assert!(said.contains("moved while compiler"), "{said}"); + let stored: Vec<_> = fs::read_dir(Keyed::Compiler.store(&store)).unwrap().flatten().map(|e| e.file_name()).collect(); + assert_eq!(stored, [std::ffi::OsString::from(format!("{named}.partial"))], "a compiler its key does not name was kept"); } /// `fork`'s LLVM checked out at a commit of its own. - pub(crate) fn llvm_checkout(fork: &Path) -> PathBuf { + fn llvm_checkout(fork: &Path) -> PathBuf { let llvm = fork.join(LLVM); git(&llvm, &["init", "-q"]); write(&llvm.join("llvm/lib/IR/Core.cpp"), "int core;\n"); @@ -705,11 +569,11 @@ pub(crate) mod tests { } /// An LLVM checkout holding what no commit does, an edit or a file git does - /// not track, names no compiler of a worktree's. + /// not track, names no compiler. #[test] fn an_uncommitted_llvm_edit_is_refused() { let scratch = TempDir::new("compiler-llvm-edit"); - let (_primary, rust_dir, [same, _, _]) = estate(&scratch); + let (_primary, store, [same, _, _]) = estate(&scratch); let fork = same.join("rust"); let llvm = llvm_checkout(&fork); let committed = key(&fork); @@ -721,7 +585,7 @@ pub(crate) mod tests { write(&llvm.join("llvm/lib/IR/Core.cpp"), "int core_edited;\n"); let said = refusal("an uncommitted LLVM edit named a compiler", || { - choose(&same, &rust_dir, &fork, fake); + chosen(&same, &store, &fork, fake); }); assert!(said.contains("holds changes no commit does"), "{said}"); git(&llvm, &["commit", "-qam", "the edit"]); @@ -729,125 +593,83 @@ pub(crate) mod tests { write(&llvm.join("llvm/lib/IR/Untracked.cpp"), "int untracked;\n"); let said = refusal("an untracked file in LLVM named a compiler", || { - choose(&same, &rust_dir, &fork, fake); + chosen(&same, &store, &fork, fake); }); assert!(said.contains("holds changes no commit does"), "{said}"); assert_eq!(builds.get(), 0, "an LLVM checkout no commit holds built a compiler"); } - /// The primary records no LLVM edit as the commit it is an edit of. + /// Every source a compiler is built from moves its key: the library its + /// host std is built from, the tools, the stage-0 pin, the lockfile, the + /// workspace's manifest, the version, the channel, bootstrap's helper crate + /// and what runs bootstrap by content, committed or not, and LLVM by the + /// commit its gitlink records. #[test] - fn the_primary_records_no_uncommitted_llvm_edit() { - let scratch = TempDir::new("compiler-llvm-primary"); - let (_primary, rust_dir, _) = estate(&scratch); - let llvm = llvm_checkout(&rust_dir); - let before = fs::read_to_string(primary_record(&rust_dir)).unwrap(); - write(&llvm.join("llvm/lib/IR/Core.cpp"), "int core_edited;\n"); - let said = refusal("the primary recorded an uncommitted LLVM edit as its commit", || record(&rust_dir)); - assert!(said.contains("holds changes no commit does"), "{said}"); - assert_eq!(fs::read_to_string(primary_record(&rust_dir)).unwrap(), before); - } - - /// Every source a compiler is built from moves its key: LLVM by commit, - /// the tools by content. - #[test] - fn llvm_and_the_tools_move_the_key() { + fn every_source_of_a_compiler_moves_its_key() { let scratch = TempDir::new("compiler-key"); - let (_primary, _rust_dir, [same, _, _]) = estate(&scratch); + let (_primary, _store, [same, _, _]) = estate(&scratch); let fork = same.join("rust"); + let sources = [ + ("library/std/src/lib.rs", "pub fn b() {}\n"), + ("src/tools/lld-wrapper/src/main.rs", "fn main() {}\n"), + ("src/stage0", "compiler_version=nightly\n"), + ("Cargo.lock", "# relocked\n"), + ("Cargo.toml", "[profile.release]\nopt-level = 3\n"), + ("src/version", "1.0.0\n"), + ("src/ci/channel", "beta\n"), + ("src/build_helper/src/lib.rs", "pub fn helper() {}\n"), + ("x", "#!/bin/sh\n"), + ("x.py", "# edited\n"), + ]; + for (file, text) in sources { + let before = key(&fork); + write(&fork.join(file), text); + let edited = key(&fork); + assert_ne!(edited, before, "{file} kept the key"); + git(&fork, &["add", "-A"]); + git(&fork, &["commit", "-qm", file]); + assert_eq!(key(&fork), edited, "committing {file} moved the key"); + } let before = key(&fork); - write(&fork.join("src/tools/lld-wrapper/src/main.rs"), "fn main() { 1; }\n"); - let tools = key(&fork); - assert_ne!(tools, before, "a tool's source did not move the key"); git(&fork, &["update-index", "--add", "--cacheinfo", &format!("160000,{LLVM_B},{LLVM}")]); let said = refusal("a gitlink staged and not committed named a compiler", || { key(&fork); }); assert!(said.contains("stages"), "{said}"); git(&fork, &["commit", "-qm", "another LLVM"]); - assert_ne!(key(&fork), tools, "another LLVM commit did not move the key"); + assert_ne!(key(&fork), before, "another LLVM commit did not move the key"); } - /// **The primary's record and every compiler's key read the whole build**: - /// the configuration bootstrap is given and the tools put beside the - /// compiler are in both, and another configuration is another key; the - /// record names every source of [`KEYED`], as the key does. What - /// [`record`] writes is the key a store is filed under before the - /// compiler is built ([`primary_key`]). + /// **Every compiler's key reads the whole build**: the configuration + /// bootstrap is given and the tools put beside the compiler are in it, and + /// another configuration is another key. #[test] - fn a_compiler_is_keyed_and_recorded_by_its_whole_build() { + fn a_compiler_is_keyed_by_its_whole_build() { let scratch = TempDir::new("compiler-build"); - let (_primary, rust_dir, [same, _, _]) = estate(&scratch); + let (_primary, _store, [same, _, _]) = estate(&scratch); let fork = same.join("rust"); let llvm = crate::llvm::key(&fork); assert_eq!(key_of(&fork, &build_text(), &llvm), key(&fork)); let config = config_text(Path::new(""), &host_triple(), Path::new("")); let tools: Vec<&str> = crate::clang::tools().collect(); - for part in [config.as_str(), toolchain::LEAN, toolchain::HOST_LINKER_PIN, &tools.join(" ")] { + for part in [config.as_str(), LEAN, HOST_LINKER_PIN, &tools.join(" ")] { assert!(build_text().contains(part), "a compiler's key reads no {part:?}"); - assert!(source(&fork).contains(part), "the primary's record names no {part:?}"); } let more = build_text().replace("debuginfo-level-rustc = 0", "debuginfo-level-rustc = 1"); assert_ne!(key_of(&fork, &more, &llvm), key(&fork), "another configuration kept the key"); - - let sources = [ - ("src/tools/lld-wrapper/src/main.rs", "fn main() {}\n"), - ("src/stage0", "compiler_version=nightly\n"), - ("Cargo.lock", "# relocked\n"), - ]; - for (file, text) in sources { - let before = source(&fork); - write(&fork.join(file), text); - assert_ne!(source(&fork), before, "{file} kept the primary's record"); - git(&fork, &["add", "-A"]); - git(&fork, &["commit", "-qm", file]); - assert_ne!(source(&fork), before, "{file}, committed, kept the primary's record"); - } - - record(&rust_dir); - assert_eq!(Compiler::primary(&rust_dir).key(), primary_key(&rust_dir)); } - /// **A compiler of a worktree's own is built as the primary's is**: for - /// the host alone, against the host's LLVM, copying none of its tools, with - /// rustc without debuginfo and no codegen test. + /// **Every compiler is built lean against the host's LLVM**: for the host + /// alone, linking that LLVM through its `llvm-config` and taking its LLD as + /// `rust-lld`, copying none of its tools, with rustc without debuginfo and + /// no codegen test, and the host's own linker left as it is. #[test] - fn a_worktree_compiler_is_built_lean_against_the_host_s_llvm() { + fn every_compiler_is_built_lean_against_the_host_s_llvm() { let config = config_text(Path::new("/b"), "h", Path::new("/llvm")); - let lean = "\nlld = true\nllvm-tools = false\ndebuginfo-level-rustc = 0\ncodegen-tests = false\n"; + let lean = "\n[rust]\nincremental = true\nlld = true\nllvm-tools = false\ndebuginfo-level-rustc = 0\ncodegen-tests = false\n"; assert!(config.contains(lean) && config.contains("\ntarget = [\"h\"]\n"), "{config}"); - assert!(config.contains("\nllvm-config = \"/llvm/bin/llvm-config\"\n"), "{config}"); - } - - /// A primary record naming another LLVM, or none, is another compiler. - #[test] - fn another_llvm_is_another_compiler() { - let scratch = TempDir::new("compiler-llvm"); - let (_primary, rust_dir, [same, _, _]) = estate(&scratch); - let builds = Cell::new(0); - let fake = |fork: &Path| { - builds.set(builds.get() + 1); - let stage2 = fork.join("build/toyos-compiler/stage2"); - write(&stage2.join("bin/rustc"), "a rustc"); - write(&stage2.join("lib/librustc_driver-2.dylib"), "a driver"); - stage2 - }; - let fork = same.join("rust"); - assert!(choose(&same, &rust_dir, &fork, fake).primary, "the primary's own compiler/ and LLVM built one"); - - let record = primary_record(&rust_dir); - let recorded = fs::read_to_string(&record).unwrap(); - let (compiler, llvm) = recorded.rsplit_once(" llvm ").expect("the record names its LLVM"); - assert_eq!(llvm, crate::llvm::key(&rust_dir).as_str()); - fs::write(&record, compiler).unwrap(); - assert!(!choose(&same, &rust_dir, &fork, fake).primary, "a record naming no LLVM was taken for this one"); - assert_eq!(builds.get(), 1); - fs::write(&record, &recorded).unwrap(); - - git(&fork, &["update-index", "--add", "--cacheinfo", &format!("160000,{LLVM_B},{LLVM}")]); - git(&fork, &["commit", "-qm", "the ToyOS LLVM"]); - let mine = choose(&same, &rust_dir, &fork, fake); - assert!(!mine.primary, "a worktree pinning another LLVM took the primary's compiler"); - assert_eq!(builds.get(), 2); + let host = "[target.h]\ndefault-linker-linux-override = \"off\"\nllvm-config = \"/llvm/bin/llvm-config\"\nllvm-has-rust-patches = true\n"; + assert!(config.contains(host), "{config}"); + assert!(!config.contains("\"rust-lld\""), "{config}"); } } diff --git a/src/keystore.rs b/src/keystore.rs index 5b91b1f0c39..4a9c557ef07 100644 --- a/src/keystore.rs +++ b/src/keystore.rs @@ -1,7 +1,11 @@ -//! What every content-addressed product of the host ([`Keyed`]) shares: its -//! [`Key`], the record each worktree keeps of the key it uses, and the sweep -//! that removes a key no registered worktree records and nobody is making or -//! using. +//! What every content-addressed product of the host ([`Keyed`]) shares: the +//! store it lives in ([`host`]), its [`Key`], and the sweep that removes a +//! product nobody has used for [`KEPT`] and nobody is making or using. +//! +//! **One store per host, outside every checkout** ([`host`]). Whichever +//! checkout or clone first needs a product makes it, and every other finds it: +//! nothing in a store names the checkout that made it, in a record or as the +//! target of a link. A runner's store is in its checkout (`src/release.rs`). //! //! **A key hashes what its product's build reads**: its sources, the //! configuration its build is given, the tools that run that build and the keys @@ -10,25 +14,24 @@ //! another run's cache, is the one this tree's build would make. An input a //! build reads and its key does not is a defect of the key. //! -//! A product lives at `//`, whole once its maker renamed it there; -//! any other name beginning `.` is one half-made or half-removed. A whole -//! one is renamed out of the way before anything in it is removed ([`retire`]), -//! so a sweep that is stopped leaves nothing at `/` but what was made. A -//! product may be read-only, directories and all: removing one gives its -//! directories back their write permission first ([`remove`]). A hidden name in -//! a store is a desktop's (Finder writes `.DS_Store` into any directory it +//! A product lives at `///`, whole once its maker renamed it +//! there; any other name beginning `.` is one half-made or half-removed. A +//! whole one is renamed out of the way before anything in it is removed +//! ([`retire`]), so a sweep that is stopped leaves nothing at `/` but what +//! was made. A product may be read-only, directories and all: removing one gives +//! its directories back their write permission first ([`remove`]). A hidden name +//! in a store is a desktop's (Finder writes `.DS_Store` into any directory it //! shows) and the sweep leaves it; any other name no key owns refuses the sweep. -use std::collections::{BTreeMap, BTreeSet}; +use std::collections::BTreeMap; use std::fmt; use std::fs; use std::io::ErrorKind; use std::os::unix::fs::PermissionsExt; use std::path::{Path, PathBuf}; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; use crate::buildlock::{self, Guard, Keyed}; -use crate::sysroot::git_out; /// A product's key: the first 16 hex digits of the SHA-256 of what it is made /// from, and no other string, so no name in a store that is not a key is taken @@ -65,105 +68,63 @@ impl AsRef for Key { } } -/// Where `root`'s builds record the key of `kind` they use. -fn record_path(root: &Path, kind: Keyed) -> PathBuf { - root.join(match kind { - Keyed::Sysroot => "target/toyos-sysroot-key", - Keyed::Freestanding => "target/toyos-freestanding-key", - Keyed::Compiler => "target/toyos-compiler-key", - Keyed::Llvm => "target/toyos-llvm-key", - }) -} - -/// Record that `root` uses `kind`'s `key`: whole or not at all, so a sweep -/// never reads a record half-written. -pub fn record(root: &Path, kind: Keyed, key: &Key) { - record_by(root, kind, key, |path, key| fs::write(path, key).unwrap_or_else(|e| panic!("write {}: {e}", path.display()))); -} - -static TEMPS: AtomicU64 = AtomicU64::new(0); - -/// [`record`], writing with `write`, so a test can stop it. -fn record_by(root: &Path, kind: Keyed, key: &Key, write: impl FnOnce(&Path, &str)) { - let path = record_path(root, kind); - let dir = path.parent().expect("a file under target/"); - fs::create_dir_all(dir).unwrap_or_else(|e| panic!("create {}: {e}", dir.display())); - // Its own name: concurrent writers of one record never share a temp file. - let written = path.with_extension(format!("{}.{}.new", std::process::id(), TEMPS.fetch_add(1, Ordering::Relaxed))); - write(&written, key.as_str()); - fs::rename(&written, &path).unwrap_or_else(|e| panic!("rename {} -> {}: {e}", written.display(), path.display())); +/// How long a product nobody uses stays in a store. +const KEPT: Duration = Duration::from_secs(14 * 24 * 60 * 60); + +/// This host's store: `~/.cache/toyos`. Nothing else in the environment names +/// another, so no build makes a second toolchain beside the first; a home that +/// is no absolute path is refused, since its store would be another directory +/// for every checkout that asks. +pub fn host() -> PathBuf { + let home = std::env::var_os("HOME") + .map(PathBuf::from) + .unwrap_or_else(|| panic!("HOME is unset, so nothing says where this host keeps its toolchains")); + assert!(home.is_absolute(), "HOME is {}, which is no absolute path, and the host's store is under it", home.display()); + home.join(".cache/toyos") } -/// Record that `root` uses `kind`'s `key`, whose product is `store/`, and -/// hold it in use: made by `make` first when `defect` says it is not whole -/// (`buildlock::keyed_made`), and then `store` swept, so the product one -/// replaced goes once nobody names it. -pub fn made( - root: &Path, - kind: Keyed, - store: &Path, - key: &Key, - defect: impl Fn() -> Option, - mut make: impl FnMut(), -) -> Guard { - record(root, kind, key); +/// `kind`'s `key` in `store`, held in use: made by `make` first when `defect` +/// says it is not whole (`buildlock::keyed_made`), and then the store swept of +/// that kind, so what nobody has used for [`KEPT`] goes when something is +/// placed beside it. +pub fn made(store: &Path, kind: Keyed, key: &Key, defect: impl Fn() -> Option, mut make: impl FnMut()) -> Guard { let mut placed = false; - let using = buildlock::keyed_made(root, kind, key, defect, || { + let using = buildlock::keyed_made(store, kind, key, defect, || { make(); placed = true; }); if placed { - for gone in sweep(root, kind, store) { - eprintln!("Removed {} {}: no worktree names it", kind.name(), gone.display()); + for gone in sweep(store, kind) { + eprintln!( + "Removed {} {}: nothing used it for {} days, or it was never whole", + kind.name(), + gone.display(), + KEPT.as_secs() / 86_400 + ); } } using } -/// Record that `root` uses no `kind` of its own. -pub fn forget(root: &Path, kind: Keyed) { - let path = record_path(root, kind); - match fs::remove_file(&path) { - Err(e) if e.kind() != ErrorKind::NotFound => panic!("remove {}: {e}", path.display()), - _ => {} - } -} - -/// The key of `kind` `root` records, if it records one. -pub fn recorded(root: &Path, kind: Keyed) -> Option { - let path = record_path(root, kind); - match fs::read_to_string(&path) { - Ok(text) => { - Some(Key::parse(text.trim()).unwrap_or_else(|| panic!("{} records {text:?}, which is no key", path.display()))) - } - Err(e) if e.kind() == ErrorKind::NotFound => None, - Err(e) => panic!("read {}: {e}", path.display()), - } -} - -/// Remove from `store` every `kind` no registered worktree of `root` records -/// and nobody is making or using, and every half-made or half-removed one -/// nobody is making. Returns what went. A `store` holding a name that is -/// neither a key's nor hidden is refused before anything goes. -pub fn sweep(root: &Path, kind: Keyed, store: &Path) -> Vec { - sweep_by(root, kind, store, remove) +/// Remove from `store` every `kind` nothing has used for [`KEPT`] and nobody is +/// making or using, and every half-made or half-removed one nobody is making. +/// Returns what went. A `store` holding a name that is neither a key's nor +/// hidden is refused before anything goes. +pub fn sweep(store: &Path, kind: Keyed) -> Vec { + sweep_by(store, kind, remove) } /// [`sweep`], removing with `remove`, so a test can stop it. -pub(crate) fn sweep_by(root: &Path, kind: Keyed, store: &Path, remove: impl Fn(&Path) + Copy) -> Vec { - let entries = match fs::read_dir(store) { +pub(crate) fn sweep_by(store: &Path, kind: Keyed, remove: impl Fn(&Path) + Copy) -> Vec { + let dir = kind.store(store); + let entries = match fs::read_dir(&dir) { Ok(entries) => entries, Err(e) if e.kind() == ErrorKind::NotFound => return Vec::new(), - Err(e) => panic!("read {}: {e}", store.display()), + Err(e) => panic!("read {}: {e}", dir.display()), }; - let named: BTreeSet = git_out(root, &["worktree", "list", "--porcelain"]) - .lines() - .filter_map(|l| l.strip_prefix("worktree ")) - .filter_map(|w| recorded(Path::new(w), kind)) - .collect(); let mut by_key: BTreeMap> = BTreeMap::new(); for entry in entries { - let entry = entry.unwrap_or_else(|e| panic!("read {}: {e}", store.display())); + let entry = entry.unwrap_or_else(|e| panic!("read {}: {e}", dir.display())); let name = entry.file_name(); // No key begins with a dot. if name.as_encoded_bytes().starts_with(b".") { @@ -177,7 +138,7 @@ pub(crate) fn sweep_by(root: &Path, kind: Keyed, store: &Path, remove: impl Fn(& panic!( "{} holds {name:?}, which is no {} key's and is not hidden: something other than a \ build writes there, and nothing in it is swept until that is gone", - store.display(), + dir.display(), kind.name(), ); }; @@ -185,15 +146,15 @@ pub(crate) fn sweep_by(root: &Path, kind: Keyed, store: &Path, remove: impl Fn(& } let mut removed = Vec::new(); for (key, mut names) in by_key { - names.retain(|name| name != key.as_str() || !named.contains(&key)); - if names.is_empty() { - continue; + let Some(idle) = buildlock::keyed_idle(store, kind, &key) else { continue }; + // Read under the key's lock: a use that came before it has dated the key. + if idle.used_within(KEPT) { + names.retain(|name| name != key.as_str()); } - let Some(_idle) = buildlock::keyed_idle(root, kind, &key) else { continue }; // The whole one last: its `retire` goes where a stopped one's remains were. names.sort_by_key(|name| name == key.as_str()); for name in names { - let path = store.join(&name); + let path = dir.join(&name); if name == key.as_str() { retire_by(&path, remove); } else { @@ -270,59 +231,81 @@ pub(crate) fn writable(dir: &Path) { } #[cfg(test)] -mod tests { +pub(crate) mod tests { + use std::time::SystemTime; + use toyos_tmpdir::TempDir; use super::*; - use crate::compiler::tests::{git, write}; + use crate::compiler::tests::write; + + /// Date the last use of `kind`'s `key` in `store` at `ago` before now. + pub(crate) fn last_used(store: &Path, kind: Keyed, key: &Key, ago: Duration) { + let lock = buildlock::keyed_lock_path(store, kind, key); + fs::create_dir_all(lock.parent().unwrap()).unwrap(); + let lock = fs::File::options().create(true).truncate(false).write(true).open(lock).unwrap(); + lock.set_modified(SystemTime::now() - ago).unwrap(); + } - /// A key no registered worktree records goes, read-only or not, and so does - /// a half-made or half-removed one; a key a worktree records stays, and so - /// does one somebody is using; an unreadable record is refused, never read - /// as "names nothing". - #[test] - fn a_sweep_removes_what_no_worktree_names_and_nobody_uses() { - let root = TempDir::new("sweep"); - git(&root, &["init", "-q"]); - write(&root.join("f"), "x\n"); - git(&root, &["add", "f"]); - git(&root, &["commit", "-qm", "init"]); - let linked = root.join("linked"); - git(&root, &["worktree", "add", "-q", "-b", "wt", linked.to_str().unwrap()]); + /// Longer ago than a store keeps what nobody uses. + pub(crate) const LONG_AGO: Duration = Duration::from_secs(KEPT.as_secs() + 3600); - let dir = root.join("store"); - let [named, linked_named, in_use, orphan, gone] = - ["named", "linked-named", "in-use", "orphan", "gone"].map(|name| Key::of(name.as_bytes())); + /// **A product goes once nothing has used it for [`KEPT`] and nobody is + /// using it**, read-only or not, and so does a half-made or half-removed + /// one however lately its key was used; one used since stays, one somebody + /// is using stays however long ago that use began, and one no lock ever + /// dated is dated by the sweep that finds it, and stays. + #[test] + fn a_sweep_removes_what_nobody_used_for_the_keep_time_and_nobody_uses() { + let store = TempDir::new("sweep"); + let dir = Keyed::Sysroot.store(&store); + let [used, in_use, undated, unused, gone] = + ["used", "in-use", "undated", "unused", "gone"].map(|name| Key::of(name.as_bytes())); let at = |key: &Key, rest: &str| dir.join(format!("{key}{rest}")); - for (key, rest) in [(&named, ""), (&linked_named, ""), (&in_use, ""), (&orphan, ""), (&named, ".partial"), (&gone, ".swept"), (&orphan, ".swept")] { + for (key, rest) in [(&used, ""), (&in_use, ""), (&undated, ""), (&unused, ""), (&used, ".partial"), (&gone, ".swept"), (&unused, ".swept")] { fs::create_dir_all(at(key, rest).join("sub")).unwrap(); } - for read_only in [at(&orphan, "/sub"), at(&orphan, ""), at(&named, ".partial")] { + for read_only in [at(&unused, "/sub"), at(&unused, ""), at(&used, ".partial")] { fs::set_permissions(read_only, fs::Permissions::from_mode(0o555)).unwrap(); } - record(&root, Keyed::Sysroot, &named); - record(&linked, Keyed::Sysroot, &linked_named); - let user = buildlock::tests::sysroot_used_elsewhere(&root, &in_use); + last_used(&store, Keyed::Sysroot, &used, Duration::from_secs(KEPT.as_secs() - 3600)); + last_used(&store, Keyed::Sysroot, &unused, LONG_AGO); + let user = buildlock::tests::sysroot_used_elsewhere(&store, &in_use); + last_used(&store, Keyed::Sysroot, &in_use, LONG_AGO); - let mut removed = sweep(&root, Keyed::Sysroot, &dir); + let mut removed = sweep(&store, Keyed::Sysroot); removed.sort(); - let mut want = [at(&gone, ".swept"), at(&named, ".partial"), at(&orphan, ""), at(&orphan, ".swept")]; + let mut want = [at(&gone, ".swept"), at(&used, ".partial"), at(&unused, ""), at(&unused, ".swept")]; want.sort(); assert_eq!(removed, want); - for stays in [&named, &linked_named, &in_use] { + for stays in [&used, &in_use, &undated] { assert!(dir.join(stays).is_dir(), "{stays} was swept"); } for gone in want { assert!(!gone.exists(), "{} was reported and kept", gone.display()); } user.release(); - assert_eq!(sweep(&root, Keyed::Sysroot, &dir), [dir.join(&in_use)]); + assert_eq!(sweep(&store, Keyed::Sysroot), [dir.join(&in_use)]); + } - fs::remove_file(record_path(&linked, Keyed::Sysroot)).unwrap(); - fs::create_dir(record_path(&linked, Keyed::Sysroot)).unwrap(); - let unreadable = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| sweep(&root, Keyed::Sysroot, &dir))); - assert!(unreadable.is_err(), "an unreadable record was read as naming nothing"); - assert!(dir.join(&linked_named).is_dir(), "an unreadable record's key was swept"); + /// **A use dates its key**: of two products last used longer ago than + /// [`KEPT`], the one used again stays and the other goes, and a sweep of + /// another kind takes neither. + #[test] + fn a_product_used_again_is_kept() { + let store = TempDir::new("sweep-used"); + let dir = Keyed::Sysroot.store(&store); + let [again, never] = ["used again", "never again"].map(|name| Key::of(name.as_bytes())); + for key in [&again, &never] { + fs::create_dir_all(dir.join(key)).unwrap(); + last_used(&store, Keyed::Sysroot, key, LONG_AGO); + } + assert_eq!(sweep(&store, Keyed::Compiler), Vec::::new()); + // The use is another process's, and one sweep decides both products: + // none here takes a key whose lock this process has held. + buildlock::tests::sysroot_used_elsewhere(&store, &again).release(); + assert_eq!(sweep(&store, Keyed::Sysroot), [dir.join(&never)], "a product was swept after a use, or one nothing used was kept"); + assert!(dir.join(&again).is_dir()); } /// **A key is the 16 lowercase hex digits [`Key::of`] gives, and no other @@ -340,16 +323,17 @@ mod tests { /// the sweep.** #[test] fn a_sweep_leaves_hidden_names_and_refuses_names_no_key_owns() { - let root = TempDir::new("strangers"); - git(&root, &["init", "-q"]); - let dir = root.join("store"); - let orphan = dir.join("0123456789abcdef"); + let store = TempDir::new("strangers"); + let dir = Keyed::Sysroot.store(&store); + let unused = Key::parse("0123456789abcdef").unwrap(); + let orphan = dir.join(&unused); let hidden = [".DS_Store", "._0123456789abcdef"].map(|name| dir.join(name)); for path in &hidden { write(path, "a desktop's"); } fs::create_dir_all(orphan.join("sub")).unwrap(); - assert_eq!(sweep(&root, Keyed::Sysroot, &dir), std::slice::from_ref(&orphan)); + last_used(&store, Keyed::Sysroot, &unused, LONG_AGO); + assert_eq!(sweep(&store, Keyed::Sysroot), std::slice::from_ref(&orphan)); for path in &hidden { assert!(path.is_file(), "{} was swept", path.display()); } @@ -357,55 +341,13 @@ mod tests { for stranger in ["notes", "0123456789abcde", "0123456789abcdeg", "0123456789abcdef0", "0123456789abcdef copy"] { fs::create_dir_all(&orphan).unwrap(); fs::create_dir(dir.join(stranger)).unwrap(); - let refused = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| sweep(&root, Keyed::Sysroot, &dir))); + let refused = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| sweep(&store, Keyed::Sysroot))); assert!(refused.is_err(), "{stranger:?} was taken for a key"); assert!(orphan.is_dir() && dir.join(stranger).is_dir(), "a store holding {stranger:?} was swept"); fs::remove_dir(dir.join(stranger)).unwrap(); } } - /// **A record stopped mid-write leaves the one before it readable**: the - /// new key is written beside it and renamed over it whole. A record that - /// holds no key is refused, never read as one. - #[test] - fn a_stopped_record_leaves_the_one_before_it() { - let root = TempDir::new("record"); - let (before, after) = (Key::of(b"before"), Key::of(b"after")); - record(&root, Keyed::Llvm, &before); - let stopped = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { - record_by(&root, Keyed::Llvm, &after, |path, key| { - fs::write(path, &key[..8]).unwrap(); - panic!("stopped"); - }) - })); - assert!(stopped.is_err(), "the stand-in write was never asked"); - assert_eq!(recorded(&root, Keyed::Llvm), Some(before), "a record stopped mid-write was read"); - record(&root, Keyed::Llvm, &after); - assert_eq!(recorded(&root, Keyed::Llvm), Some(after.clone())); - - fs::write(record_path(&root, Keyed::Llvm), &after.as_str()[..8]).unwrap(); - let refused = std::panic::catch_unwind(|| recorded(&root, Keyed::Llvm)); - assert!(refused.is_err(), "a record holding no key was read as one"); - } - - /// Concurrent records of one kind never share a temp file, so none fails - /// and the record holds one of the keys written whole. - #[test] - fn concurrent_records_of_one_kind_all_land() { - let root = TempDir::new("race"); - let keys: Vec = (0..8u8).map(|i| Key::of(&[i])).collect(); - for _ in 0..50 { - std::thread::scope(|s| { - let handles: Vec<_> = keys.iter().map(|key| s.spawn(|| record(&root, Keyed::Llvm, key))).collect(); - for h in handles { - h.join().expect("a concurrent record panicked"); - } - }); - let got = recorded(&root, Keyed::Llvm).unwrap(); - assert!(keys.contains(&got), "the record holds {got:?}, none of the keys written"); - } - } - /// **A retire stopped halfway leaves nothing at the name it removes**: what /// it removes is out of the way before anything in it goes, so a stopped /// sweep leaves nothing that passes for whole, and the next retire takes diff --git a/src/licence.rs b/src/licence.rs index 5dae94d57a7..1c106d08124 100644 --- a/src/licence.rs +++ b/src/licence.rs @@ -1188,7 +1188,7 @@ struct Doc { pub fn judge(root: &Path) -> Result { // Cargo names every manifest by its canonical path. let root = &std::fs::canonicalize(root).map_err(|e| format!("{}: {e}", root.display()))?; - let shipped = crate::build::shipped(root)?; + let shipped = crate::build::shipped(root); let mut report = Report::default(); let mut roots: Vec<(PathBuf, Features)> = shipped.crates.into_iter().collect(); diff --git a/src/llvm.rs b/src/llvm.rs index cbfc3cfe588..cb55f7a3998 100644 --- a/src/llvm.rs +++ b/src/llvm.rs @@ -6,33 +6,32 @@ //! checkout holding what no commit does and a gitlink staged and not committed), //! the committed tree of its `src/bootstrap` (one holding what no commit does is //! refused), the bootstrap configuration below, [`RECIPE`], and the tools the -//! host builds it with ([`host_tools`]). `rust/build/llvm//` in the primary -//! is what builds read of bootstrap's install of that LLVM and its clang +//! host builds it with ([`host_tools`]). `llvm//` in the store +//! (`src/keystore.rs`) is what builds read of bootstrap's install of that LLVM and its clang //! ([`keep`]), with its LLD in `bin/` beside `llvm-config` and in `src/` the //! runtimes' sources the C++ runtime is built from (`src/libcxx.rs`) as its //! commit holds them, made by whichever build first needs it ([`resolve`]), and //! stored only when it was built from what the key names. Once its [`SOURCE`] //! file exists it is read-only, its directories as well as its files. -//! Every compiler build, the primary's and a worktree's own, names it as the -//! host's `llvm-config` with `llvm-has-rust-patches`, so bootstrap builds no -//! LLVM and takes LLD from beside it as `rust-lld`; `clang::provision` copies its -//! clang. Once a build directory's compiler is built against it, the LLVM that +//! Every compiler build names it as the host's `llvm-config` with +//! `llvm-has-rust-patches`, so bootstrap builds no LLVM and takes LLD from +//! beside it as `rust-lld`; `clang::provision` copies its clang. Once a build directory's compiler is built against it, the LLVM that //! directory built itself goes ([`retire_in_tree`]). //! //! **Nothing of the environment it is asked from reaches it but //! [`ENVIRONMENT`]**: the build and every tool its key asks run with the rest //! cleared ([`clear`]), the configuration names the C and C++ compilers by path, -//! and every host library LLVM would otherwise find and link is turned off -//! ([`NO_HOST_LIBRARIES`]). +//! every host library LLVM would otherwise find and link is turned off +//! ([`NO_HOST_LIBRARIES`]), and the revision and repository LLVM, clang and LLD +//! say they were built from are the commit and the URL the fork's commit names +//! ([`stamp`]), never what git answers in the checkout that built them. //! //! Its lock (`buildlock::keyed_*` with [`Keyed::Llvm`]) is taken inside the -//! worktree or global lock that covers the fork build directory its maker -//! writes, `build/toyos-llvm/`, which is removed once the LLVM is placed. +//! worktree lock that covers the fork build directory its maker writes, +//! `build/toyos-llvm/`, which is removed once the LLVM is placed. //! -//! An LLVM no worktree names any more is removed by `keystore::sweep`, which -//! every placement runs: each worktree records the key of -//! the LLVM its compiler links, and a key no registered worktree records, that -//! nobody is making or using, goes. +//! An LLVM nothing has used for the store's keep time is removed by +//! `keystore::sweep`, which every placement runs. use std::collections::BTreeSet; use std::fs; @@ -124,22 +123,36 @@ pub fn host_lines(dir: &Path) -> String { format!("llvm-config = \"{}\"\nllvm-has-rust-patches = true", dir.join("bin/llvm-config").display()) } -/// Every LLVM on this host. -pub fn store(rust_dir: &Path) -> PathBuf { - rust_dir.join("build/llvm") -} - /// The key of the LLVM `fork` names; refused while its `src/bootstrap` holds /// changes no commit does. pub fn key(fork: &Path) -> Key { let tools = host_tools(); - key_of(fork, RECIPE, &config_text(Path::new(KEYED_BUILD_DIR), &host_triple(), tools), &tools.identity) + key_of(fork, RECIPE, &config_text(Path::new(KEYED_BUILD_DIR), &host_triple(), tools, &stamp(fork)), &tools.identity) +} + +/// What an LLVM, its clang and its LLD say they were built from. +struct Stamp { + /// The commit `fork`'s gitlink records. + revision: String, + /// The URL `fork`'s committed `.gitmodules` names for it. + repository: String, +} + +/// The [`Stamp`] of the LLVM `fork` names. LLVM's CMake otherwise asks git in +/// the checkout it builds, and writes that checkout's `origin` into every +/// binary: an input no key reads. +fn stamp(fork: &Path) -> Stamp { + let name = format!("submodule.{LLVM}.url"); + let url = git_out(fork, &["config", "--blob", "HEAD:.gitmodules", "--get", &name]); + Stamp { revision: gitlink(fork, LLVM), repository: url.trim().to_string() } } +/// [`key`], with what it reads beside `fork`'s committed `src/bootstrap`: +/// `config` names the LLVM's commit ([`stamp`]). fn key_of(fork: &Path, recipe: &str, config: &str, tools: &str) -> Key { refuse_uncommitted_bootstrap(fork); let bootstrap = git_out(fork, &["rev-parse", &format!("HEAD:{BOOTSTRAP}")]); - Key::of([recipe, config, &gitlink(fork, LLVM), bootstrap.trim(), tools].join("\n\0\n").as_bytes()) + Key::of([recipe, config, bootstrap.trim(), tools].join("\n\0\n").as_bytes()) } /// Give `command` nothing of this process's environment but [`ENVIRONMENT`]. @@ -205,34 +218,20 @@ fn on_path(name: &str) -> PathBuf { fs::canonicalize(&found).unwrap_or_else(|e| panic!("resolve {}: {e}", found.display())) } -/// The LLVM `fork` names, made if nobody on this host has made it, and held in -/// use for as long as the returned value lives. `root` records its key. -pub fn resolve(root: &Path, rust_dir: &Path, fork: &Path) -> Llvm { - choose(root, rust_dir, fork, |fork| build_in_fork(root, fork)) +/// The LLVM `fork` names, in `store`: made if nobody on this host has made it, +/// under the n2 installed under `root`, and held in use for as long as the +/// returned value lives. +pub fn resolve(root: &Path, store: &Path, fork: &Path) -> Llvm { + choose(store, fork, |fork| build_in_fork(root, fork)) } /// [`resolve`] with the build that makes an LLVM passed in, so a test can stand /// in for bootstrap: `build` builds in the fork checkout it is given and returns /// the build directory, holding `/llvm` and `/lld`. -fn choose(root: &Path, rust_dir: &Path, fork: &Path, build: impl Fn(&Path) -> PathBuf) -> Llvm { +fn choose(store: &Path, fork: &Path, build: impl Fn(&Path) -> PathBuf) -> Llvm { let key = key(fork); - let dir = store(rust_dir).join(&key); - let using = keystore::made(root, Keyed::Llvm, &store(rust_dir), &key, || defect(&dir), || place(fork, &key, &dir, &build)); - Llvm { dir, _using: using } -} - -/// [`resolve`], recording nothing for `root`: what a sysroot build reads of the -/// LLVM its compiler links, whose record is that compiler's -/// (`compiler::choose`). -pub fn held(root: &Path, rust_dir: &Path, fork: &Path) -> Llvm { - held_with(root, rust_dir, fork, |fork| build_in_fork(root, fork)) -} - -/// [`held`] with the build passed in, as [`choose`] takes it. -fn held_with(root: &Path, rust_dir: &Path, fork: &Path, build: impl Fn(&Path) -> PathBuf) -> Llvm { - let key = key(fork); - let dir = store(rust_dir).join(&key); - let using = crate::buildlock::keyed_made(root, Keyed::Llvm, &key, || defect(&dir), || place(fork, &key, &dir, &build)); + let dir = Keyed::Llvm.store(store).join(&key); + let using = keystore::made(store, Keyed::Llvm, &key, || defect(&dir), || place(fork, &key, &dir, &build)); Llvm { dir, _using: using } } @@ -437,7 +436,7 @@ fn build_in_fork(root: &Path, fork: &Path) -> PathBuf { let build_dir = fork.join("build/toyos-llvm"); fs::create_dir_all(&build_dir).unwrap_or_else(|e| panic!("create {}: {e}", build_dir.display())); let config = build_dir.join("bootstrap.toml"); - fs::write(&config, config_text(&build_dir, &host, host_tools())) + fs::write(&config, config_text(&build_dir, &host, host_tools(), &stamp(fork))) .unwrap_or_else(|e| panic!("write {}: {e}", config.display())); let config = config.to_str().unwrap_or_else(|| panic!("{} is not UTF-8", config.display())); let args = ["build", "--config", config, "src/llvm-project/llvm", "src/llvm-project/lld"]; @@ -450,9 +449,13 @@ fn build_in_fork(root: &Path, fork: &Path) -> PathBuf { } /// Bootstrap's configuration for an LLVM: the options every compiler build's -/// `[llvm]` names, no host library, and `tools`' compilers, for the host alone. -fn config_text(build_dir: &Path, host: &str, tools: &HostTools) -> String { - let off: Vec = NO_HOST_LIBRARIES.iter().map(|option| format!("{option} = \"OFF\"")).collect(); +/// `[llvm]` names, no host library, `stamp` as the revision and repository +/// LLVM's CMake would otherwise ask git for, and `tools`' compilers, for the +/// host alone. +fn config_text(build_dir: &Path, host: &str, tools: &HostTools, stamp: &Stamp) -> String { + let off = NO_HOST_LIBRARIES.iter().map(|option| format!("{option} = \"OFF\"")); + let stamped = [("LLVM_FORCE_VC_REVISION", &stamp.revision), ("LLVM_FORCE_VC_REPOSITORY", &stamp.repository)]; + let defines: Vec = off.chain(stamped.map(|(option, value)| format!("{option} = \"{value}\""))).collect(); format!( r#"change-id = "ignore" profile = "compiler" @@ -464,7 +467,7 @@ target = ["{host}"] [llvm] {llvm} -build-config = {{ {off} }} +build-config = {{ {defines} }} [target.{host}] cc = "{cc}" @@ -472,7 +475,7 @@ cxx = "{cxx}" "#, build_dir = build_dir.display(), llvm = crate::clang::LLVM_CONFIG, - off = off.join(", "), + defines = defines.join(", "), cc = tools.cc.display(), cxx = tools.cxx.display(), ) @@ -486,7 +489,8 @@ mod tests { use super::*; use crate::buildlock; - use crate::compiler::tests::{estate, git, write, LLVM_B}; + use crate::compiler::tests::{estate, git, write, LLVM_A, LLVM_B, LLVM_URL}; + use crate::keystore::tests::{last_used, LONG_AGO}; /// A scratch directory whose read-only LLVMs are made writable again before /// it goes. @@ -577,12 +581,17 @@ mod tests { /// `estate`, every fork's LLVM checked out at the one commit its gitlink /// records. fn estate_built(scratch: &Path) -> (PathBuf, PathBuf, [PathBuf; 3]) { - let (primary, rust_dir, forks) = estate(scratch); + let (primary, store, forks) = estate(scratch); for worktree in &forks { pin_llvm(&worktree.join("rust"), "A"); } - pin_llvm(&rust_dir, "A"); - (primary, rust_dir, forks) + pin_llvm(&primary.join("rust"), "A"); + (primary, store, forks) + } + + /// What the fixtures' forks name as their LLVM's revision and repository. + fn stamped() -> Stamp { + Stamp { revision: LLVM_A.to_string(), repository: LLVM_URL.to_string() } } /// What `f` panicked with; `expect` if it returned. @@ -598,7 +607,7 @@ mod tests { #[test] fn two_compilers_of_one_llvm_make_it_once() { let scratch = Scratch::new("llvm"); - let (primary, rust_dir, [same, a, b]) = estate_built(&scratch); + let (primary, store, [same, a, b]) = estate_built(&scratch); let makes = Cell::new(0); let once = |fork: &Path| { makes.set(makes.get() + 1); @@ -606,7 +615,7 @@ mod tests { fake_build(fork) }; - let la = choose(&a, &rust_dir, &a.join("rust"), once); + let la = choose(&store, &a.join("rust"), once); assert_eq!(makes.get(), 1); assert_eq!(defect(&la.dir), None); assert_eq!(fs::read_to_string(la.dir.join("bin/lld")).unwrap(), "the lld"); @@ -615,13 +624,13 @@ mod tests { assert!(!la.dir.join("build").exists(), "CMake's tree was kept"); assert!(!a.join("rust/build/toyos-llvm").exists(), "the build directory outlived the placement"); - let before = snapshot(&store(&rust_dir)); - let lb = choose(&b, &rust_dir, &b.join("rust"), once); - let primary_s = choose(&primary, &rust_dir, &rust_dir, once); - let same_s = choose(&same, &rust_dir, &same.join("rust"), once); + let before = snapshot(&Keyed::Llvm.store(&store)); + let lb = choose(&store, &b.join("rust"), once); + let primary_s = choose(&store, &primary.join("rust"), once); + let same_s = choose(&store, &same.join("rust"), once); assert_eq!(makes.get(), 1); assert!(lb.dir == la.dir && primary_s.dir == la.dir && same_s.dir == la.dir, "one LLVM commit named two LLVMs"); - assert_eq!(snapshot(&store(&rust_dir)), before, "an LLVM was written after it was whole"); + assert_eq!(snapshot(&Keyed::Llvm.store(&store)), before, "an LLVM was written after it was whole"); } /// **An LLVM keeps what builds read of the install and nothing else**: the @@ -632,8 +641,8 @@ mod tests { #[test] fn an_llvm_keeps_what_builds_read_and_nothing_else() { let scratch = Scratch::new("llvm-kept"); - let (_primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); - let dir = choose(&a, &rust_dir, &a.join("rust"), fake_build).dir; + let (_primary, store, [_same, a, _b]) = estate_built(&scratch); + let dir = choose(&store, &a.join("rust"), fake_build).dir; let apple = host_triple().ends_with("apple-darwin"); let mut files: Vec = snapshot(&dir) .into_iter() @@ -669,8 +678,8 @@ mod tests { #[test] fn a_placed_llvm_is_read_only() { let scratch = Scratch::new("llvm-read-only"); - let (_primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); - let dir = choose(&a, &rust_dir, &a.join("rust"), fake_build).dir; + let (_primary, store, [_same, a, _b]) = estate_built(&scratch); + let dir = choose(&store, &a.join("rust"), fake_build).dir; let mut entries = vec![dir]; while let Some(entry) = entries.pop() { let meta = fs::symlink_metadata(&entry).unwrap(); @@ -687,13 +696,13 @@ mod tests { #[test] fn an_llvm_that_is_not_whole_is_made_again() { let scratch = Scratch::new("llvm-whole"); - let (_primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); + let (_primary, store, [_same, a, _b]) = estate_built(&scratch); let makes = Cell::new(0); let counted = |fork: &Path| { makes.set(makes.get() + 1); fake_build(fork) }; - let dir = choose(&a, &rust_dir, &a.join("rust"), counted).dir; + let dir = choose(&store, &a.join("rust"), counted).dir; for (lost, made) in [("bin/lld", 2), ("lib", 3), ("src/libcxxabi", 4)] { keystore::writable(&dir); let lost = dir.join(lost); @@ -703,7 +712,7 @@ mod tests { fs::remove_file(&lost).unwrap(); } assert!(defect(&dir).is_some_and(|d| d.contains(&lost.display().to_string())), "{:?}", defect(&dir)); - let again = choose(&a, &rust_dir, &a.join("rust"), counted); + let again = choose(&store, &a.join("rust"), counted); assert_eq!((makes.get(), defect(&again.dir)), (made, None)); } } @@ -715,10 +724,10 @@ mod tests { #[test] fn the_key_moves_with_the_llvm_and_only_with_it() { let scratch = Scratch::new("llvm-key"); - let (_primary, rust_dir, [same, a, _b]) = estate(&scratch); + let (primary, _store, [same, a, _b]) = estate(&scratch); let fork = same.join("rust"); let tools = host_tools(); - let config = config_text(Path::new(KEYED_BUILD_DIR), &host_triple(), tools); + let config = config_text(Path::new(KEYED_BUILD_DIR), &host_triple(), tools, &stamp(&fork)); let base = key(&fork); assert_eq!(key_of(&fork, RECIPE, &config, &tools.identity), base); let elsewhere = if host_triple() == "x86_64-unknown-linux-gnu" { @@ -726,7 +735,7 @@ mod tests { } else { "x86_64-unknown-linux-gnu" }; - let elsewhere = config_text(Path::new(KEYED_BUILD_DIR), elsewhere, tools); + let elsewhere = config_text(Path::new(KEYED_BUILD_DIR), elsewhere, tools, &stamp(&fork)); for (what, other) in [ ("the recipe", key_of(&fork, "another recipe", &config, &tools.identity)), ("the [llvm]", key_of(&fork, RECIPE, &config.replace("X86", "RISCV;X86"), &tools.identity)), @@ -736,7 +745,7 @@ mod tests { assert_ne!(other, base, "{what} did not move the key"); } - assert_eq!(key(&fork), key(&rust_dir), "one LLVM commit named two keys"); + assert_eq!(key(&fork), key(&primary.join("rust")), "one LLVM commit named two keys"); assert_eq!(key(&fork), key(&a.join("rust")), "a compiler/ edit moved the LLVM key"); write(&fork.join("compiler/rustc_target/src/new_target.rs"), "pub fn t() {}\n"); git(&fork, &["add", "-A"]); @@ -771,7 +780,7 @@ mod tests { if host_triple().ends_with("apple-darwin") { assert!(lines.iter().any(|l| l.ends_with(".sdk") && Path::new(l).is_dir()), "no SDK: {}", tools.identity); } - let config = config_text(Path::new(KEYED_BUILD_DIR), "h", tools); + let config = config_text(Path::new(KEYED_BUILD_DIR), "h", tools, &stamped()); let named = format!("[target.h]\ncc = \"{}\"\ncxx = \"{}\"\n", tools.cc.display(), tools.cxx.display()); assert!(config.contains(&named), "{config}"); } @@ -782,7 +791,7 @@ mod tests { #[cfg(target_os = "macos")] fn two_sdk_versions_are_two_keys() { let scratch = Scratch::new("llvm-sdk"); - let (_primary, _rust_dir, [same, _a, _b]) = estate(&scratch); + let (_primary, _store, [same, _a, _b]) = estate(&scratch); let fork = same.join("rust"); let [older, newer] = ["26.0", "27.0"].map(|version| { let tools = tools_with(|question| match question { @@ -790,7 +799,7 @@ mod tests { "--show-sdk-version" => format!("{version}\n"), other => panic!("xcrun was asked {other}"), }); - key_of(&fork, RECIPE, &config_text(Path::new(KEYED_BUILD_DIR), &host_triple(), &tools), &tools.identity) + key_of(&fork, RECIPE, &config_text(Path::new(KEYED_BUILD_DIR), &host_triple(), &tools, &stamp(&fork)), &tools.identity) }); assert_ne!(older, newer, "two SDK versions at one SDK path named one LLVM"); } @@ -799,12 +808,12 @@ mod tests { /// `build-config` bootstrap passes to CMake after its own options. #[test] fn the_llvm_config_turns_every_host_library_off() { - let config = config_text(Path::new(KEYED_BUILD_DIR), "h", host_tools()); + let config = config_text(Path::new(KEYED_BUILD_DIR), "h", host_tools(), &stamped()); let off = "build-config = { LLVM_ENABLE_ZLIB = \"OFF\", LLVM_ENABLE_ZSTD = \"OFF\", \ LLVM_ENABLE_LIBXML2 = \"OFF\", CLANG_ENABLE_LIBXML2 = \"OFF\", LLVM_ENABLE_LIBEDIT = \"OFF\", \ LLVM_ENABLE_LIBPFM = \"OFF\", LLVM_ENABLE_FFI = \"OFF\", LLVM_ENABLE_CURL = \"OFF\", \ LLVM_ENABLE_HTTPLIB = \"OFF\", LLVM_ENABLE_ICU = \"OFF\", LLVM_ENABLE_ICONV = \"OFF\", \ - LLVM_ENABLE_Z3_SOLVER = \"OFF\" }\n"; + LLVM_ENABLE_Z3_SOLVER = \"OFF\", LLVM_FORCE_VC_REVISION = "; assert!(config.contains(&format!("\n{off}")), "{config}"); } @@ -814,7 +823,7 @@ mod tests { #[test] fn a_checkout_behind_its_gitlink_is_built_at_the_gitlink() { let scratch = Scratch::new("llvm-behind"); - let (_primary, rust_dir, [same, a, _b]) = estate_built(&scratch); + let (_primary, store, [same, a, _b]) = estate_built(&scratch); let (fork, at_gitlink) = (a.join("rust"), same.join("rust")); for fork in [&fork, &at_gitlink] { pin_llvm(fork, "B"); @@ -825,8 +834,8 @@ mod tests { check_out_llvm(fork, "B"); fake_build(fork) }; - let llvm = choose(&a, &rust_dir, &fork, updating); - assert_eq!((llvm.dir.clone(), defect(&llvm.dir)), (store(&rust_dir).join(key(&at_gitlink)), None)); + let llvm = choose(&store, &fork, updating); + assert_eq!((llvm.dir.clone(), defect(&llvm.dir)), (Keyed::Llvm.store(&store).join(key(&at_gitlink)), None)); } const FORK: &str = "TOYOS_LLVM_TEST_FORK"; @@ -858,7 +867,7 @@ mod tests { fn the_caller_s_environment_reaches_neither_the_build_nor_the_key() { use std::os::unix::fs::PermissionsExt; let scratch = Scratch::new("llvm-environment"); - let (_primary, _rust_dir, [same, _a, _b]) = estate(&scratch); + let (_primary, _store, [same, _a, _b]) = estate(&scratch); let fork = same.join("rust"); write(&fork.join("library/Cargo.lock"), "# lock\n"); write(&fork.join("x"), "#!/bin/sh\nenv > build/toyos-llvm/environment\n"); @@ -899,7 +908,7 @@ mod tests { #[test] fn a_staged_llvm_gitlink_is_refused() { let scratch = Scratch::new("llvm-staged"); - let (_primary, _rust_dir, [same, _a, _b]) = estate(&scratch); + let (_primary, _store, [same, _a, _b]) = estate(&scratch); let fork = same.join("rust"); git(&fork, &["update-index", "--add", "--cacheinfo", &format!("160000,{LLVM_B},{LLVM}")]); let said = refusal("a staged gitlink named an LLVM", || { @@ -913,13 +922,13 @@ mod tests { #[test] fn an_uncommitted_bootstrap_names_no_llvm() { let scratch = Scratch::new("llvm-dirty-key"); - let (_primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); + let (_primary, store, [_same, a, _b]) = estate_built(&scratch); let fork = a.join("rust"); - drop(choose(&a, &rust_dir, &fork, fake_build)); + drop(choose(&store, &fork, fake_build)); write(&fork.join("src/bootstrap/src/core/build_steps/llvm.rs"), "cfg.define(\"LLVM_ENABLE_ZLIB\", \"ON\");\n"); let never = |_: &Path| -> PathBuf { panic!("an LLVM was built from a bootstrap no commit holds") }; let said = refusal("an uncommitted bootstrap edit resolved to the stored LLVM", || { - choose(&a, &rust_dir, &fork, never); + choose(&store, &fork, never); }); assert!(said.contains("src/bootstrap holds changes no commit does"), "{said}"); let said = refusal("an uncommitted bootstrap edit named an LLVM", || { @@ -936,14 +945,14 @@ mod tests { #[test] fn what_the_key_does_not_name_is_never_stored() { let scratch = Scratch::new("llvm-dirt"); - let (_primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); + let (_primary, store, [_same, a, _b]) = estate_built(&scratch); let fork = a.join("rust"); let step = fork.join("src/bootstrap/src/core/build_steps/llvm.rs"); let never = |_: &Path| -> PathBuf { panic!("an LLVM was built from a bootstrap no commit holds") }; write(&step, "cfg.define(\"LLVM_ENABLE_ZLIB\", \"OFF\");\n"); let said = refusal("an uncommitted bootstrap edit was built", || { - choose(&a, &rust_dir, &fork, never); + choose(&store, &fork, never); }); assert!(said.contains("src/bootstrap holds changes no commit does"), "{said}"); git(&fork, &["add", "-A"]); @@ -954,7 +963,7 @@ mod tests { fake_build(fork) }; let said = refusal("an LLVM built while its bootstrap was edited was stored", || { - choose(&a, &rust_dir, &fork, dirtying); + choose(&store, &fork, dirtying); }); assert!(said.contains("src/bootstrap holds changes no commit does"), "{said}"); git(&fork, &["checkout", "-q", "--", "src/bootstrap"]); @@ -965,7 +974,7 @@ mod tests { }; pin_llvm(&fork, "B"); let said = refusal("an LLVM checkout other than the gitlink's was stored", || { - choose(&a, &rust_dir, &fork, lagging); + choose(&store, &fork, lagging); }); assert!(said.contains("is checked out at") && said.contains("submodule update src/llvm-project"), "{said}"); check_out_llvm(&fork, "B"); @@ -976,11 +985,11 @@ mod tests { fake_build(fork) }; let said = refusal("an LLVM whose sources moved while it was built was stored", || { - choose(&a, &rust_dir, &fork, moving); + choose(&store, &fork, moving); }); assert!(said.contains("moved while LLVM"), "{said}"); - let stored: Vec<_> = fs::read_dir(store(&rust_dir)).unwrap().flatten().map(|e| e.file_name()).collect(); + let stored: Vec<_> = fs::read_dir(Keyed::Llvm.store(&store)).unwrap().flatten().map(|e| e.file_name()).collect(); assert!(stored.iter().all(|n| n.to_string_lossy().ends_with(".partial")), "stored: {stored:?}"); } @@ -990,7 +999,7 @@ mod tests { #[test] fn the_runtimes_sources_are_the_commit_s() { let scratch = Scratch::new("llvm-runtimes"); - let (_primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); + let (_primary, store, [_same, a, _b]) = estate_built(&scratch); let fork = a.join("rust"); let checkout = fork.join(LLVM); write(&checkout.join(".git/info/exclude"), "*.pyc\n"); @@ -1000,7 +1009,7 @@ mod tests { fake_build(fork) }; let said = refusal("an edit to the runtimes' sources was stored", || { - choose(&a, &rust_dir, &fork, editing); + choose(&store, &fork, editing); }); assert!(said.contains("holds changes no commit does"), "{said}"); git(&checkout, &["checkout", "-q", "--", "libcxx"]); @@ -1009,27 +1018,27 @@ mod tests { write(&fork.join(LLVM).join("libcxx/utils/cache.pyc"), "what no commit holds"); fake_build(fork) }; - let dir = choose(&a, &rust_dir, &fork, ignored).dir; + let dir = choose(&store, &fork, ignored).dir; assert_eq!(fs::read_to_string(dir.join("src/libcxx/CMakeLists.txt")).unwrap(), "the libcxx of A"); assert!(checkout.join("libcxx/utils/cache.pyc").is_file()); assert!(!dir.join("src/libcxx/utils").exists(), "a file the checkout ignores was stored"); } const WORKTREE: &str = "TOYOS_LLVM_TEST_WORKTREE"; - const RUST_DIR: &str = "TOYOS_LLVM_TEST_RUST_DIR"; + const STORE: &str = "TOYOS_LLVM_TEST_STORE"; const ROLE: &str = "TOYOS_LLVM_TEST_ROLE"; /// The competing process for the tests below: the LLVM the worktree in /// [`WORKTREE`] names, held in use until released — or, as `make`, held - /// while it is being made. + /// while it is being made; or a sweep of the store stopped halfway. #[test] #[ignore = "the competing process for the tests below; never runs on its own"] fn child_role() { let worktree = PathBuf::from(std::env::var(WORKTREE).unwrap_or_else(|_| panic!("child_role ran without {WORKTREE}; it is not a test"))); - let rust_dir = PathBuf::from(std::env::var(RUST_DIR).unwrap()); + let store = PathBuf::from(std::env::var(STORE).unwrap()); match std::env::var(ROLE).unwrap().as_str() { "use" => { - let _held = choose(&worktree, &rust_dir, &worktree.join("rust"), fake_build); + let _held = choose(&store, &worktree.join("rust"), fake_build); buildlock::tests::hold_until_released(); } "make" => { @@ -1037,14 +1046,23 @@ mod tests { buildlock::tests::hold_until_released(); fake_build(fork) }; - choose(&worktree, &rust_dir, &worktree.join("rust"), held); + choose(&store, &worktree.join("rust"), held); + } + "stop-a-sweep" => { + let halfway = |path: &Path| { + keystore::writable(path); + fs::remove_file(path.join("lib/libLLVMCore.a")).unwrap(); + panic!("stopped"); + }; + let stopped = std::panic::catch_unwind(|| keystore::sweep_by(&store, Keyed::Llvm, halfway)); + assert!(stopped.is_err(), "the stand-in removal was never asked"); } other => panic!("unknown child role {other}"), } } - fn elsewhere(role: &str, worktree: &Path, rust_dir: &Path) -> buildlock::tests::Elsewhere { - let env = [(ROLE, std::ffi::OsStr::new(role)), (WORKTREE, worktree.as_os_str()), (RUST_DIR, rust_dir.as_os_str())]; + fn elsewhere(role: &str, worktree: &Path, store: &Path) -> buildlock::tests::Elsewhere { + let env = [(ROLE, std::ffi::OsStr::new(role)), (WORKTREE, worktree.as_os_str()), (STORE, store.as_os_str())]; buildlock::tests::Elsewhere::hold("llvm::tests::child_role", &env) } @@ -1052,37 +1070,38 @@ mod tests { #[test] fn an_llvm_being_made_elsewhere_is_not_made_again() { let scratch = Scratch::new("llvm-made-elsewhere"); - let (primary, rust_dir, [_same, _a, b]) = estate_built(&scratch); - let maker = elsewhere("make", &b, &rust_dir); + let (_primary, store, [_same, _a, b]) = estate_built(&scratch); + let maker = elsewhere("make", &b, &store); let made = key(&b.join("rust")); - assert!(buildlock::keyed_idle(&primary, Keyed::Llvm, &made).is_none(), "a sweep could take an LLVM being made"); + assert!(buildlock::keyed_idle(&store, Keyed::Llvm, &made).is_none(), "a sweep could take an LLVM being made"); maker.release(); let never = |_: &Path| -> PathBuf { panic!("an LLVM another process made was made here too") }; - assert_eq!(defect(&choose(&b, &rust_dir, &b.join("rust"), never).dir), None); + assert_eq!(defect(&choose(&store, &b.join("rust"), never).dir), None); } - /// **A sweep takes an LLVM only once no worktree names it and nobody uses - /// it**: `a` moves to another LLVM while a process of its own still uses - /// the first, and then `b` names the first until it moves too. + /// **A sweep takes an LLVM only once nothing has used it for the store's + /// keep time and nobody uses it**: `a` moves to another LLVM while a + /// process of its own still uses the first, which placing the second + /// leaves, and which goes once that process has let go. #[test] - fn an_llvm_is_swept_once_nobody_names_or_uses_it() { + fn an_llvm_is_swept_once_nothing_used_it_for_the_keep_time() { let scratch = Scratch::new("llvm-sweep"); - let (primary, rust_dir, [_same, a, b]) = estate_built(&scratch); - let user = elsewhere("use", &a, &rust_dir); - let first = store(&rust_dir).join(key(&a.join("rust"))); + let (_primary, store, [_same, a, _b]) = estate_built(&scratch); + let user = elsewhere("use", &a, &store); + let unused = key(&a.join("rust")); + let first = Keyed::Llvm.store(&store).join(&unused); + last_used(&store, Keyed::Llvm, &unused, LONG_AGO); let fork = a.join("rust"); pin_llvm(&fork, "B"); - let second = choose(&a, &rust_dir, &fork, fake_build); + let second = choose(&store, &fork, fake_build); assert_ne!(second.dir, first); assert!(first.is_dir(), "placing an LLVM swept one still in use"); - keystore::record(&b, Keyed::Llvm, &key(&b.join("rust"))); + // The first one's lock was only ever its user's: the placement's sweep + // found it held, and this one is the first here to take it. user.release(); - let swept = |root: &Path| keystore::sweep(root, Keyed::Llvm, &store(&rust_dir)); - assert_eq!(swept(&primary), Vec::::new(), "the sweep took an LLVM a worktree names"); - keystore::record(&b, Keyed::Llvm, &key(&fork)); - assert_eq!(swept(&primary), [first], "the sweep kept an LLVM nobody names, or took a named one"); + assert_eq!(keystore::sweep(&store, Keyed::Llvm), [first], "the sweep kept an LLVM nothing had used for the keep time"); assert!(second.dir.is_dir()); } @@ -1092,43 +1111,40 @@ mod tests { #[test] fn a_stopped_sweep_leaves_no_llvm_that_passes_for_whole() { let scratch = Scratch::new("llvm-sweep-stopped"); - let (primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); - let dir = choose(&a, &rust_dir, &a.join("rust"), fake_build).dir; - keystore::forget(&a, Keyed::Llvm); - let halfway = |path: &Path| { - keystore::writable(path); - fs::remove_file(path.join("lib/libLLVMCore.a")).unwrap(); - panic!("stopped"); - }; - let stopped = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { - keystore::sweep_by(&primary, Keyed::Llvm, &store(&rust_dir), halfway) - })); - assert!(stopped.is_err(), "the stand-in removal was never asked"); + let (_primary, store, [_same, a, _b]) = estate_built(&scratch); + elsewhere("use", &a, &store).release(); + let unused = key(&a.join("rust")); + let dir = Keyed::Llvm.store(&store).join(&unused); + last_used(&store, Keyed::Llvm, &unused, LONG_AGO); + // In a process of its own: this one never holds the key's lock. + let mut sweep = buildlock::tests::rerun("llvm::tests::child_role"); + let stopped = sweep.env(ROLE, "stop-a-sweep").env(WORKTREE, &a).env(STORE, &store).output().unwrap(); + assert!(stopped.status.success(), "{}{}", String::from_utf8_lossy(&stopped.stdout), String::from_utf8_lossy(&stopped.stderr)); assert!(defect(&dir).is_some(), "a stopped sweep left {} passing for whole", dir.display()); - assert_eq!(keystore::sweep(&primary, Keyed::Llvm, &store(&rust_dir)), [dir.with_extension("swept")]); + let left: Vec<_> = fs::read_dir(Keyed::Llvm.store(&store)).unwrap().map(|e| e.unwrap().file_name()).collect(); + assert_eq!(keystore::sweep(&store, Keyed::Llvm), [dir.with_extension("swept")], "the stopped sweep left {left:?}"); } - /// **An LLVM a worktree resolved stays once nothing uses it**: its record, - /// not its use, is what names it. + /// **What an LLVM says it was built from is the commit and the URL its + /// fork's commit names**: both are in the configuration its key reads and + /// its build is given, so git in the checkout that builds it decides + /// neither. Another URL is another LLVM, once a commit holds it. #[test] - fn a_resolved_llvm_is_named_by_its_worktree() { - let scratch = Scratch::new("llvm-named"); - let (primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); - elsewhere("use", &a, &rust_dir).release(); - let dir = store(&rust_dir).join(key(&a.join("rust"))); - assert_eq!(keystore::sweep(&primary, Keyed::Llvm, &store(&rust_dir)), Vec::::new()); - assert_eq!(defect(&dir), None, "the sweep took an LLVM the worktree that resolved it names"); - } + fn an_llvm_is_stamped_with_what_its_fork_s_commit_names() { + let scratch = Scratch::new("llvm-stamp"); + let (_primary, _store, [same, _a, _b]) = estate(&scratch); + let fork = same.join("rust"); + let config = config_text(Path::new(KEYED_BUILD_DIR), "h", host_tools(), &stamp(&fork)); + let named = format!("LLVM_FORCE_VC_REVISION = \"{LLVM_A}\", LLVM_FORCE_VC_REPOSITORY = \"{LLVM_URL}\" }}\n"); + assert!(config.contains(&named), "{config}"); - /// **An LLVM held for a sysroot build is not recorded**: the record follows - /// the compiler's, so a worktree whose compiler is the primary's names none. - #[test] - fn a_held_llvm_is_not_recorded() { - let scratch = Scratch::new("llvm-held"); - let (_primary, rust_dir, [_same, a, _b]) = estate_built(&scratch); - let llvm = held_with(&a, &rust_dir, &a.join("rust"), fake_build); - assert_eq!(defect(&llvm.dir), None); - assert_eq!(keystore::recorded(&a, Keyed::Llvm), None, "a held LLVM was recorded"); + let before = key(&fork); + let moved = "https://llvm.invalid/another.git"; + write(&fork.join(".gitmodules"), &format!("[submodule \"{LLVM}\"]\n\tpath = {LLVM}\n\turl = {moved}\n")); + assert_eq!(key(&fork), before, "a URL no commit holds moved the key"); + git(&fork, &["commit", "-qam", "another URL"]); + assert_eq!(stamp(&fork).repository, moved); + assert_ne!(key(&fork), before, "another URL kept the key"); } /// **A build directory whose compiler links the host's LLVM keeps none of diff --git a/src/release.rs b/src/release.rs index 69322e0bb77..06dc9c1730c 100644 --- a/src/release.rs +++ b/src/release.rs @@ -1,8 +1,9 @@ //! The toolchain a runner builds with, by the build system's own keys, and the //! release main publishes of it. //! -//! **A toolchain is four stores, each a cache entry of the key the build system -//! files it under** ([`LAYERS`]). `toolchain.yml` restores each by the key +//! **A toolchain is four products of a store, each a cache entry of the key the +//! build system files it under** ([`LAYERS`]). A runner's store is in its +//! checkout ([`store`]). `toolchain.yml` restores each by the key //! [`toolchain`] wrote, builds what none restored ([`bootstrap`]) and saves only //! what it built. GitHub's ref scoping is the provenance: an entry a run on main //! saved is restored on every ref, and any other run saves only into its own @@ -14,7 +15,8 @@ //! job, that is refused before it reads anything; run on a tree main has moved //! past, it puts nothing up. //! -//! A dev host installs none: its build system builds its own from `rust/`. +//! A dev host installs none: its build system finds its own in the host's store, +//! or builds it from `rust/` (`src/keystore.rs`). use std::fs; use std::io::Write; @@ -43,7 +45,7 @@ const GLIBC_FLOOR: (u32, u32) = (2, 39); /// What every request the build system makes says it comes from. const USER_AGENT: &str = "toyos-build (https://github.com/ToyOSOrg/ToyOS)"; -/// The stores a toolchain is, in the order a build makes them, each under the +/// The products a toolchain is, in the order a build makes them, each under the /// name its cache entry and its job's outputs carry. const LAYERS: [(Keyed, &str); 4] = [ (Keyed::Llvm, "llvm"), @@ -53,12 +55,12 @@ const LAYERS: [(Keyed, &str); 4] = [ ]; /// One of [`LAYERS`] of this tree's toolchain: the key the build system files -/// it under, and the paths it is, relative to the checkout. +/// it under, and where it is, relative to the checkout. struct Layer { kind: Keyed, name: &'static str, key: Key, - paths: Vec, + path: PathBuf, } impl Layer { @@ -110,34 +112,33 @@ fn publisher(workflow: Option<&str>, event: Option<&str>, repo: &str) -> Result< } } +/// A runner's store: in its checkout, because a cache entry's paths are +/// relative to the workspace, and the job that restores one in a container has +/// another home. +fn store(root: &Path) -> PathBuf { + root.join("rust/build") +} + /// This tree's toolchain as [`LAYERS`], keyed from its sources alone, before -/// any store is there. +/// any is in the store. fn layers(root: &Path) -> Vec { let rust_dir = root.join("rust"); let llvm = crate::llvm::key(&rust_dir); - let compiler = crate::compiler::primary_key(&rust_dir); + let compiler = crate::compiler::key(&rust_dir); let freestanding = crate::sysroot::freestanding_key(root, &compiler, &rust_dir); let sysroot = crate::sysroot::key(root, &freestanding); LAYERS .iter() .map(|&(kind, name)| { - let (key, paths) = match kind { - Keyed::Llvm => (llvm.clone(), vec![crate::llvm::store(&rust_dir).join(&llvm)]), - Keyed::Compiler => ( - compiler.clone(), - vec![crate::toolchain::stage2(&rust_dir), crate::compiler::primary_record(&rust_dir)], - ), - Keyed::Freestanding => { - (freestanding.clone(), vec![crate::sysroot::freestanding_dir(&rust_dir).join(&freestanding)]) - } - Keyed::Sysroot => (sysroot.clone(), vec![crate::sysroot::sysroots_dir(&rust_dir).join(&sysroot)]), + let key = match kind { + Keyed::Llvm => &llvm, + Keyed::Compiler => &compiler, + Keyed::Freestanding => &freestanding, + Keyed::Sysroot => &sysroot, }; - let paths = paths - .iter() - .map(|path| path.strip_prefix(root).unwrap_or_else(|_| panic!("{} is outside the checkout", path.display()))) - .map(Path::to_path_buf) - .collect(); - Layer { kind, name, key, paths } + let dir = kind.store(&store(root)).join(key); + let path = dir.strip_prefix(root).expect("a runner's store is in its checkout").to_path_buf(); + Layer { kind, name, key: key.clone(), path } }) .collect() } @@ -145,13 +146,12 @@ fn layers(root: &Path) -> Vec { /// Why `layer` is not whole in `root`, as the build that makes it decides, if /// it is not. fn defect(root: &Path, layer: &Layer) -> Option { - let dir = root.join(&layer.paths[0]); + let dir = root.join(&layer.path); match layer.kind { Keyed::Llvm => crate::llvm::defect(&dir), - Keyed::Compiler => match fs::read(root.join(&layer.paths[1])) { - Ok(record) if Key::of(&record) == layer.key => crate::toolchain::toolchain_defect(&dir), - _ => Some(format!("{} records no compiler {}", layer.paths[1].display(), layer.key)), - }, + Keyed::Compiler => { + crate::compiler::unplaced(&dir).or_else(|| crate::toolchain::toolchain_defect(&dir.join("stage2"))) + } Keyed::Freestanding => crate::sysroot::unpublished(&dir), Keyed::Sysroot => crate::sysroot::unfinished(&dir), } @@ -185,15 +185,9 @@ pub fn toolchain(root: &Path) -> Result { Ok(layers.iter().map(|layer| format!("{} {}", layer.name, layer.key)).collect::>().join(", ")) } -/// Each layer's entry as `-key` and its paths, one a line, as -/// `-path`. +/// Each layer's entry as `-key` and its path as `-path`. fn outputs(layers: &[Layer]) -> String { - let mut text = String::new(); - for layer in layers { - let paths: Vec = layer.paths.iter().map(|path| path.display().to_string()).collect(); - text += &format!("{0}-key={1}\n{0}-path< String { pub fn bootstrap(root: &Path) -> Result { let file = step_outputs()?; let layers = layers(root); - let restored: Vec = layers.iter().map(|layer| root.join(&layer.paths[0]).exists()).collect(); + let restored: Vec = layers.iter().map(|layer| root.join(&layer.path).exists()).collect(); whole(&layers, &restored, |layer| defect(root, layer)).map_err(|why| format!("restored, {why}"))?; if !restored[3] { let mut lock = crate::buildlock::shared(root, "the toolchain"); - drop(crate::toolchain::ensure(root, &mut lock, false)); + drop(crate::sysroot::ensure(root, &store(root), &mut lock)); whole(&layers, &[true; 4], |layer| defect(root, layer)).map_err(|why| format!("built, {why}"))?; } tell(&file, &built(&layers, &restored))?; @@ -247,29 +241,27 @@ fn built(layers: &[Layer], restored: &[bool]) -> String { .collect() } -/// Install the sysroot its job restored ([`lay_out`]) as rustup's `toyos`, on -/// a runner. +/// Install the sysroot its job restored ([`lay_out`]), on a runner. /// /// Off a runner this says so and does nothing: the build system owns the dev /// host's toolchain. pub fn install(root: &Path) -> Result { if !on_runner() { - return Ok("not a runner: the build system uses this checkout's own toolchain".into()); + return Ok("not a runner: the build system uses the host's own toolchain".into()); } let key = lay_out(root)?; let stage2 = crate::toolchain::stage2(&root.join("rust")); - run(Command::new("rustup").args(["toolchain", "link", "toyos"]).arg(&stage2))?; run(Command::new(stage2.join("bin/rustc")).arg("-vV"))?; - Ok(format!("installed sysroot {key} as `toyos`")) + Ok(format!("installed sysroot {key}")) } -/// Lay the one sysroot under `rust/build/sysroots`, which its job's restore +/// Lay the one sysroot in the runner's [`store`], which its job's restore /// step put there, out as this checkout's installed toolchain /// (`toolchain::Owner::Installed`): at `stage2`, with the witness it records and /// its [`manifest`]. Refused unless that witness is this tree's. fn lay_out(root: &Path) -> Result { let rust_dir = root.join("rust"); - let store = crate::sysroot::sysroots_dir(&rust_dir); + let store = Keyed::Sysroot.store(&store(root)); let restored = fs::read_dir(&store) .and_then(|entries| entries.map(|entry| entry.map(|entry| entry.path())).collect::>>()) .map_err(|e| format!("{}: {e}", store.display()))?; @@ -835,13 +827,12 @@ mod tests { } } - /// A sysroot store under `root`'s `rust/build/sysroots`, as a job restores - /// one, recording `witness`. + /// A sysroot in `root`'s store, as a job restores one, recording `witness`. fn restored(root: &Path, key: &str, witness: &str) -> PathBuf { - let dir = crate::sysroot::sysroots_dir(&root.join("rust")).join(key); + let dir = Keyed::Sysroot.store(&store(root)).join(key); fs::create_dir_all(dir.join("bin")).unwrap(); fs::write(dir.join("bin/rustc"), "rustc").unwrap(); - fs::write(dir.join("SOURCES"), format!("{key}\nfork /a/runner/s/rust\n{witness}\n")).unwrap(); + fs::write(dir.join("SOURCES"), format!("{key}\n{witness}\n")).unwrap(); dir } @@ -854,7 +845,7 @@ mod tests { let tmp = TempDir::new("release-lay-out"); let root = tmp.join("checkout"); checkout(&root, "pub struct A;\n"); - fs::create_dir_all(crate::sysroot::sysroots_dir(&root.join("rust"))).unwrap(); + fs::create_dir_all(Keyed::Sysroot.store(&store(&root))).unwrap(); assert!(lay_out(&root).unwrap_err().contains("holds 0 entries")); let witness = crate::sysroot::witness(&root); let one = restored(&root, "0123456789abcdef", &witness); @@ -880,24 +871,22 @@ mod tests { assert!(lay_out(&root).unwrap_err().contains("named by no key")); } - fn layer(name: &'static str, key: &str, paths: &[&str]) -> Layer { + fn layer(name: &'static str, key: &str, path: &str) -> Layer { let kind = LAYERS.iter().find(|(_, n)| *n == name).unwrap().0; - Layer { kind, name, key: Key::parse(key).unwrap(), paths: paths.iter().map(PathBuf::from).collect() } + Layer { kind, name, key: Key::parse(key).unwrap(), path: PathBuf::from(path) } } - /// The job's outputs, as GitHub's multiline syntax reads them: each layer's - /// cache entry, and each path of it on a line of its own. + /// The job's outputs: each layer's cache entry, and its path. #[test] - fn a_job_is_told_each_layer_s_entry_and_paths() { + fn a_job_is_told_each_layer_s_entry_and_path() { let layers = [ - layer("llvm", "1111111111111111", &["rust/build/llvm/1111111111111111"]), - layer("compiler", "2222222222222222", &["rust/build/h/stage2", "rust/build/toyos-compiler"]), + layer("llvm", "1111111111111111", "rust/build/llvm/1111111111111111"), + layer("compiler", "2222222222222222", "rust/build/compilers/2222222222222222"), ]; assert_eq!( outputs(&layers), - "llvm-key=toolchain-llvm-1111111111111111\nllvm-path< = - LAYERS.iter().enumerate().map(|(at, (_, name))| layer(name, &at.to_string().repeat(16), &["p"])).collect(); + LAYERS.iter().enumerate().map(|(at, (_, name))| layer(name, &at.to_string().repeat(16), "p")).collect(); let broken = |layer: &Layer| (layer.name == "compiler").then(|| "stage2 carries no clang".to_string()); let refused = whole(&layers, &[true, true, false, false], broken).unwrap_err(); assert!(refused.starts_with("compiler 1111111111111111 is not whole") && refused.contains("no clang"), "{refused}"); @@ -916,30 +905,28 @@ mod tests { assert_eq!(whole(&layers, &[true; 4], |_| None), Ok(())); } - /// **Each layer is the store the build system makes, where it makes it**: - /// the LLVM, the freestanding libraries and the sysroot in their stores by - /// key, and the primary's compiler as its `stage2` and its record, each - /// relative to the checkout, keyed before any is built. + /// **Each layer is the product the build system makes, where a runner's + /// store holds it**: the LLVM, the compiler, the freestanding libraries and + /// the sysroot, each by its key and relative to the checkout, keyed before + /// any is built. #[test] fn the_layers_are_the_stores_the_build_makes() { let scratch = TempDir::new("release-layers"); - let (primary, _rust_dir, _) = crate::compiler::tests::estate(&scratch); + let (primary, _store, _) = crate::compiler::tests::estate(&scratch); checkout(&primary, "pub struct A;\n"); let layers = layers(&primary); - let named: Vec<(&str, Vec)> = - layers.iter().map(|l| (l.name, l.paths.iter().map(|p| p.display().to_string()).collect())).collect(); + let named: Vec<(&str, String)> = layers.iter().map(|l| (l.name, l.path.display().to_string())).collect(); let key = |at: usize| layers[at].key.to_string(); - let host = crate::toolchain::host_triple(); assert_eq!( named, [ - ("llvm", vec![format!("rust/build/llvm/{}", key(0))]), - ("compiler", vec![format!("rust/build/{host}/stage2"), "rust/build/toyos-compiler".to_string()]), - ("freestanding", vec![format!("rust/build/freestanding/{}", key(2))]), - ("sysroot", vec![format!("rust/build/sysroots/{}", key(3))]), + ("llvm", format!("rust/build/llvm/{}", key(0))), + ("compiler", format!("rust/build/compilers/{}", key(1))), + ("freestanding", format!("rust/build/freestanding/{}", key(2))), + ("sysroot", format!("rust/build/sysroots/{}", key(3))), ] ); - assert_eq!(layers[1].key, crate::compiler::primary_key(&primary.join("rust"))); + assert_eq!(layers[1].key, crate::compiler::key(&primary.join("rust"))); } /// **A job saves exactly the layers it built**: none where it restored the @@ -950,7 +937,7 @@ mod tests { let layers: Vec = LAYERS .iter() .enumerate() - .map(|(at, (_, name))| layer(name, &at.to_string().repeat(16), &["p"])) + .map(|(at, (_, name))| layer(name, &at.to_string().repeat(16), "p")) .collect(); let told = |restored: [bool; 4]| built(&layers, &restored); assert_eq!(told([true, true, false, true]), "llvm=kept\ncompiler=kept\nfreestanding=kept\nsysroot=kept\n"); diff --git a/src/sysroot.rs b/src/sysroot.rs index be74d4c4a62..dc2f6a3946e 100644 --- a/src/sysroot.rs +++ b/src/sysroot.rs @@ -6,8 +6,8 @@ //! built from: the trees std and `libtoyos_c.a` compile //! ([`SYSROOT_SOURCES`]) and how libc is built ([`SYSROOT_MANIFESTS`], //! `libc::BUILD`), the std fork's `library/` and `src/bootstrap/` in the -//! checkout that builds it, and the compiler's key. `rust/build/ -//! sysroots//` is a whole toolchain — the compiler's files cloned from its +//! checkout that builds it, and the compiler's key. `sysroots//` in the +//! store (`src/keystore.rs`) is a whole toolchain — the compiler's files cloned from its //! `stage2`, the guest targets' libraries built from this key's sources. A build //! compiles against the directory its own key names, so two worktrees with //! different ABIs or different compilers never refuse or wait for each other, @@ -15,7 +15,7 @@ //! //! **The kernel's and the loader's libraries compile none of those trees**, so //! they are a key of their own ([`freestanding_key`]). They are built once per that key -//! into `rust/build/freestanding//` and cloned into every sysroot naming +//! into the store's `freestanding//` and cloned into every sysroot naming //! it, and a crate built against a sysroot learns which targets' libraries //! moved ([`Identity`]). Each build refuses dep-info that says otherwise. //! @@ -27,22 +27,18 @@ //! its ToyOS backend, `sdk/std`, by a `#[path]` as far up, so each checkout's //! std compiles against its own worktree's ABI and backend with nothing //! rewritten. The build is bootstrap's stage-0 local rebuild: the compiler the -//! checkout names (`src/compiler.rs` — the primary's `stage2`, or one of the -//! worktree's own where its `compiler/` differs) compiles the checkout's -//! `library/` for the guest targets into `/build/toyos-std/`. +//! checkout names (`src/compiler.rs`) compiles the checkout's `library/` for +//! the guest targets into `/build/toyos-std/`. //! -//! Locks, in the one order every acquirer takes them: the compiler key's, if the -//! compiler is a worktree's own; the sysroot key's (`buildlock::keyed_*`), with -//! this worktree's build lock put down; then the freestanding key's, while its -//! libraries are cloned in; then, to build, this worktree's exclusively (its -//! fork build directory is written); then, if the compiler is the primary's, the -//! global one shared, because it is read; then the key of the compiler's LLVM, -//! held in use while the C++ runtime is built from its sources. +//! Locks, in the one order every acquirer takes them: the compiler key's; the +//! sysroot key's (`buildlock::keyed_*`), with this worktree's build lock put +//! down; then the freestanding key's, while its libraries are cloned in; then, +//! to build, this worktree's exclusively (its fork build directory is +//! written); then the key of the compiler's LLVM, held in use while the C++ +//! runtime is built from its sources. //! -//! A sysroot or freestanding libraries no worktree names any more are removed -//! by `keystore::sweep`, which every placement runs: each build records the -//! keys it used in its worktree's `target/`, and a key no registered worktree -//! records, that nobody is making or using, goes. +//! A sysroot or freestanding libraries nothing has used for the store's keep +//! time are removed by `keystore::sweep`, which every placement runs. use std::collections::{BTreeMap, BTreeSet}; use std::fmt; @@ -53,7 +49,7 @@ use std::process::Command; use sha2::{Digest, Sha256}; use crate::arch::Arch; -use crate::buildlock::{self, Guard, Held, Keyed, Scope}; +use crate::buildlock::{self, Guard, Held, Keyed}; use crate::compiler::{self, Compiler}; use crate::identity; use crate::keystore::{self, Key}; @@ -103,16 +99,6 @@ const RECIPE: &str = "bootstrap stage-0 local rebuild, libraries from the stamp, runtimes' sources of the compiler's LLVM, the freestanding libraries cloned \ from their key's; 13"; -/// Every sysroot on this host. -pub fn sysroots_dir(rust_dir: &Path) -> PathBuf { - rust_dir.join("build/sysroots") -} - -/// Every freestanding target's libraries on this host, one directory per key. -pub fn freestanding_dir(rust_dir: &Path) -> PathBuf { - rust_dir.join("build/freestanding") -} - /// Whose sources a guest target's libraries compile. #[derive(Clone, Copy, PartialEq, Eq)] enum Libraries { @@ -141,9 +127,6 @@ impl Libraries { /// A sysroot a build compiles against, held in use for as long as this lives. pub struct Sysroot { dir: PathBuf, - /// Whether its compiler is the primary's, which the ToyOS-hosted rustc is - /// built from. - pub primary_compiler: bool, pub identity: Identity, _using: Option, } @@ -157,7 +140,7 @@ impl Sysroot { pub(crate) fn installed(stage2: PathBuf, release: &str) -> Self { let id = crate::release::named_key(release) .unwrap_or_else(|| panic!("the installed TOOLCHAIN names no sysroot key:\n{release}")); - Self { dir: stage2, primary_compiler: true, identity: Identity::new(id.clone(), &id, &id), _using: None } + Self { dir: stage2, identity: Identity::new(id.clone(), &id, &id), _using: None } } /// Its toolchain directory, which `RUSTUP_TOOLCHAIN` names: lent, never @@ -444,7 +427,7 @@ impl Keys { /// The keys of what `root` builds against with its std fork at `fork`, /// compiled by `compiler`. fn of(root: &Path, compiler: &Compiler, fork: &Path) -> Self { - let freestanding = freestanding_key(root, &compiler.key(), fork); + let freestanding = freestanding_key(root, compiler.key(), fork); let sysroot = key(root, &freestanding); let identity = Identity::new(Key::of(compiler.identity().as_bytes()), &freestanding, &sysroot); Self { freestanding, sysroot, identity } @@ -518,9 +501,8 @@ fn pinned_fork(root: &Path) -> String { /// /// The primary's is its own `rust/`, used where it is not initialised yet, which /// whoever initialises it does at the pin. It is no workspace, so one whose -/// `HEAD` is not the pin is moved there, under the global lock held exclusively, -/// which the primary's toolchain phases hold while they build from it; refused -/// by name if it has local changes, or does not hold the pinned commit. +/// `HEAD` is not the pin is moved there; refused by name if it has local +/// changes, or does not hold the pinned commit. /// /// A linked worktree's `rust/` starts as the empty stub `git worktree add` /// leaves; it is made here, the first time it is needed, as a git worktree of @@ -539,8 +521,9 @@ fn pinned_fork(root: &Path) -> String { /// /// Every build in a worktree asks this at once, so the making and the move are /// each decided and done under the worktree's lock held exclusively -/// ([`Held::act_if`]): one build does it, and none reads a checkout another is -/// still writing. +/// ([`Held::act_if`]), which every build that writes or compiles the checkout +/// holds too: one build does it, and none reads a checkout another is still +/// writing. pub fn fork_checkout(root: &Path, lock: &mut Held) -> PathBuf { let fork = root.join("rust"); let primary = match toolchain::owner(root) { @@ -552,7 +535,6 @@ pub fn fork_checkout(root: &Path, lock: &mut Held) -> PathBuf { let pinned = pinned_fork(root); if let Some(primary) = &primary { lock.act_if( - Scope::Worktree, "make the fork checkout", || (!fork.join(".git").exists()).then_some(()), |()| { @@ -581,7 +563,6 @@ pub fn fork_checkout(root: &Path, lock: &mut Held) -> PathBuf { ); } lock.act_if( - Scope::Worktree, "move the fork checkout to its pin", || { let head = git_out(&fork, &["rev-parse", "HEAD"]).trim().to_string(); @@ -594,7 +575,6 @@ pub fn fork_checkout(root: &Path, lock: &mut Held) -> PathBuf { (head != pinned && !ahead).then_some(head) }, |head| { - let _global = primary.is_none().then(|| buildlock::global_exclusive(root, "move the fork checkout to its pin")); if !holds(&fork, &pinned) { match &primary { Some(primary) => git_run(&fork, &["fetch", path_str(&primary.join("rust")), &pinned]), @@ -671,18 +651,18 @@ fn holds(dir: &Path, commit: &str) -> bool { git_try(dir, &["cat-file", "-e", &format!("{commit}^{{commit}}")]).is_ok() } -/// What a sysroot's [`SOURCES`] says: its key, the fork checkout its std was -/// built in, and the witness of the sources it was built from. -fn sources_text(key: &Key, fork: &Path, witness: &str) -> String { - format!("{key}\nfork {}\n{witness}\n", fork.display()) +/// What a sysroot's [`SOURCES`] says: its key, and the witness of the sources +/// it was built from. +fn sources_text(key: &Key, witness: &str) -> String { + format!("{key}\n{witness}\n") } /// The witness the sysroot at `dir` records it was built from ([`sources_text`]). pub(crate) fn recorded_witness(dir: &Path) -> Result { let path = dir.join(SOURCES); let text = fs::read_to_string(&path).map_err(|e| format!("{}: {e}", path.display()))?; - match text.splitn(3, '\n').collect::>().as_slice() { - [_, fork, witness] if fork.starts_with("fork ") => Ok(witness.trim_end_matches('\n').to_string()), + match text.split_once('\n') { + Some((key, witness)) if Key::parse(key).is_some() && !witness.is_empty() => Ok(witness.trim_end_matches('\n').to_string()), _ => Err(format!("{} records no witness: {text:?}", path.display())), } } @@ -702,25 +682,23 @@ pub(crate) fn unfinished(dir: &Path) -> Option { unpublished(dir).or_else(|| toolchain::toolchain_defect(dir)) } -/// The sysroot `key` names at `dir`, recorded as `root`'s, made by `make` if -/// nobody has made it, and held in use for as long as the returned guard lives. -fn held(root: &Path, key: &Key, dir: &Path, make: impl FnMut()) -> Guard { - let store = dir.parent().expect("a sysroot is a directory of its store"); - keystore::made(root, Keyed::Sysroot, store, key, || unfinished(dir), make) +/// The sysroot `key` names at `dir` in `store`, made by `make` if nobody has +/// made it, and held in use for as long as the returned guard lives. +fn held(store: &Path, key: &Key, dir: &Path, make: impl FnMut()) -> Guard { + keystore::made(store, Keyed::Sysroot, key, || unfinished(dir), make) } -/// The sysroot this worktree's sources name, made if nobody has made it, and -/// held in use for as long as the returned value lives. -pub fn ensure(root: &Path, rust_dir: &Path, lock: &mut Held) -> Sysroot { +/// The sysroot this worktree's sources name, in `store`: made if nobody on the +/// host has made it, and held in use for as long as the returned value lives. +pub fn ensure(root: &Path, store: &Path, lock: &mut Held) -> Sysroot { let fork = fork_checkout(root, lock); - let compiler = compiler::resolve(root, rust_dir, &fork, lock); + let compiler = compiler::resolve(root, store, &fork, lock); let keys = Keys::of(root, &compiler, &fork); - let dir = sysroots_dir(rust_dir).join(&keys.sysroot); - keystore::record(root, Keyed::Freestanding, &keys.freestanding); + let dir = Keyed::Sysroot.store(store).join(&keys.sysroot); let using = - lock.without_shared(|| held(root, &keys.sysroot, &dir, || build(root, rust_dir, &compiler, &fork, &keys, &dir))); - Sysroot { dir, primary_compiler: compiler.primary, identity: keys.identity, _using: Some(using) } + lock.without_shared(|| held(store, &keys.sysroot, &dir, || build(root, store, &compiler, &fork, &keys, &dir))); + Sysroot { dir, identity: keys.identity, _using: Some(using) } } /// Only [`whole`] makes a [`Whole`]. @@ -744,13 +722,7 @@ mod whole_toolchain { /// from one that is not, and no std is built for one. pub(super) fn whole(compiler: &Compiler) -> Whole<'_> { if let Some(defect) = toolchain::toolchain_defect(&compiler.stage2) { - let fix = if compiler.primary { - "\nA bootstrap in the primary checkout was stopped before it finished: \ - `cargo run -- --build-only` there completes it." - } else { - "" - }; - panic!("no sysroot is made from {}, and no std was built for one: {defect}{fix}", compiler.stage2.display()); + panic!("no sysroot is made from {}, and no std was built for one: {defect}", compiler.stage2.display()); } Whole(compiler) } @@ -759,22 +731,18 @@ mod whole_toolchain { /// Make the sysroot `keys` names at `dir`, from `root`'s sources and the std /// fork at `fork`, with `compiler`, and the freestanding libraries `keys` /// names. The caller holds the sysroot key's lock. -fn build(root: &Path, rust_dir: &Path, compiler: &Compiler, fork: &Path, keys: &Keys, dir: &Path) { +fn build(root: &Path, store: &Path, compiler: &Compiler, fork: &Path, keys: &Keys, dir: &Path) { let made_from = whole(compiler); - let store = freestanding_dir(rust_dir).join(&keys.freestanding); + let freestanding = Keyed::Freestanding.store(store).join(&keys.freestanding); let _freestanding = keystore::made( - root, + store, Keyed::Freestanding, - &freestanding_dir(rust_dir), &keys.freestanding, - || unpublished(&store), - || build_freestanding(root, compiler, fork, &keys.freestanding, &store), + || unpublished(&freestanding), + || build_freestanding(root, compiler, fork, &keys.freestanding, &freestanding), ); let what = format!("building sysroot {}", keys.sysroot); let _worktree = buildlock::worktree_exclusive(root, &what); - // Only the primary's compiler is rebuilt in place; one of a worktree's own - // is written once and held in use by `compiler`. - let _compiler = compiler.primary.then(|| buildlock::compiler_shared(root, &what)); eprintln!( "Building sysroot {}: ToyOS's std from {}, the compiler {}, the freestanding libraries {}", keys.sysroot, @@ -792,7 +760,7 @@ fn build(root: &Path, rust_dir: &Path, compiler: &Compiler, fork: &Path, keys: & Libraries::Worktree => place_std(&stamp(&built, triple), &lib), Libraries::Freestanding => { keystore::remove(&lib); - clone_tree(&store.join(triple), &lib); + clone_tree(&freestanding.join(triple), &lib); } } } @@ -802,7 +770,7 @@ fn build(root: &Path, rust_dir: &Path, compiler: &Compiler, fork: &Path, keys: & crate::libc::build_c(root, partial, &libc_target, arch); } let _ = fs::remove_dir_all(&libc_target); - let llvm = crate::llvm::held(root, rust_dir, fork); + let llvm = crate::llvm::resolve(root, store, fork); let ninja = crate::n2::ninja(root); for arch in Arch::ALL { let scratch = dir.with_extension(format!("libcxx-{}", arch.name())); @@ -817,7 +785,7 @@ fn build(root: &Path, rust_dir: &Path, compiler: &Compiler, fork: &Path, keys: & "the sources moved while sysroot {key} was being built (they are now {again}); \ nothing was kept, and the next build makes the one they name" ); - sources_text(key, fork, &witness(root)) + sources_text(key, &witness(root)) }); } @@ -826,7 +794,6 @@ fn build(root: &Path, rust_dir: &Path, compiler: &Compiler, fork: &Path, keys: & fn build_freestanding(root: &Path, compiler: &Compiler, fork: &Path, key: &Key, store: &Path) { let what = format!("building the freestanding libraries {key}"); let _worktree = buildlock::worktree_exclusive(root, &what); - let _compiler = compiler.primary.then(|| buildlock::compiler_shared(root, &what)); eprintln!( "Building the freestanding libraries {key}: {} from {}, the compiler {}", Libraries::Freestanding.targets().join(", "), @@ -838,13 +805,13 @@ fn build_freestanding(root: &Path, compiler: &Compiler, fork: &Path, key: &Key, for target in Libraries::Freestanding.targets() { place_std(&stamp(&built, target), &partial.join(target)); } - let again = freestanding_key(root, &compiler.key(), fork); + let again = freestanding_key(root, compiler.key(), fork); assert!( again == *key, "the sources moved while the freestanding libraries {key} were being built (they are \ now {again}); nothing was kept, and the next build makes the ones they name" ); - format!("{key}\nfork {}\n", fork.display()) + format!("{key}\n") }); } @@ -866,7 +833,7 @@ fn publish(dir: &Path, fill: impl FnOnce(&Path) -> String) { let sources = fill(&partial); fs::write(partial.join(SOURCES), sources) .unwrap_or_else(|e| panic!("write {}: {e}", partial.join(SOURCES).display())); - keystore::remove(dir); + keystore::retire(dir); fs::rename(&partial, dir) .unwrap_or_else(|e| panic!("rename {} -> {}: {e}", partial.display(), dir.display())); } @@ -914,16 +881,17 @@ fn prepare_std_build(build_dir: &Path, host: &str, identity: &str, targets: &[&s } } -/// Empty the std build directory `build_dir` of all but what bootstrap -/// downloaded unless `identity` ([`Compiler::identity`]) is the compiler its -/// `compiled-by` records as having compiled the rest, then record `identity` -/// there. +/// Empty bootstrap's build directory `build_dir` of all but what bootstrap +/// downloaded unless `identity` is what its `compiled-by` records as having +/// built the rest, then record `identity` there: for a std build the compiler +/// ([`Compiler::identity`]), for a compiler build its build and its LLVM +/// (`compiler::place`). /// /// Cargo keys what it reuses on `rustc -vV`, which every ToyOS compiler prints /// alike, so another compiler's rlibs stay fresh and the next crate that does /// recompile is refused against them (`E0463 can't find crate`). The removal /// comes before the record, so an interrupted switch removes again. -fn forget_another_compiler(build_dir: &Path, host: &str, identity: &str) { +pub(crate) fn forget_another_compiler(build_dir: &Path, host: &str, identity: &str) { forget_another_compiler_by(build_dir, host, identity, keystore::remove); } @@ -1142,6 +1110,8 @@ fn git_run(dir: &Path, args: &[&str]) { #[cfg(test)] mod tests { use super::*; + use crate::compiler::tests::compiler as placed_compiler; + use crate::keystore::tests::{last_used, LONG_AGO}; use toyos_tmpdir::TempDir; fn git(dir: &Path, args: &[&str]) -> String { @@ -1162,8 +1132,8 @@ mod tests { fs::write(path, text).unwrap(); } - /// A worktree's trees, a fork checkout and a compiler, laid out the - /// way the key reads them. + /// A worktree's trees, a store and a fork checkout, laid out the way the + /// key reads them. fn keyed(base: &Path) -> (PathBuf, PathBuf, PathBuf) { let root = base.join("root"); for tree in SYSROOT_SOURCES { @@ -1177,10 +1147,7 @@ mod tests { write(&fork.join("src/bootstrap/src/lib.rs"), "fn main() {}\n"); write(&fork.join(".gitignore"), "__pycache__\n.DS_Store\n"); git(&fork, &["init", "-q"]); - let rust_dir = base.join("rust"); - write(&rust_dir.join("build/toyos-compiler"), "tree-1"); - write(&toolchain::stage2(&rust_dir).join("lib/librustc_driver-1.dylib"), "a driver"); - (root, rust_dir, fork) + (root, base.join("store"), fork) } /// **Each key is the identity of what it is built from, and only that**: a @@ -1194,8 +1161,9 @@ mod tests { #[test] fn a_comment_is_the_same_sysroot_and_a_signature_is_another() { let base = TempDir::new("key"); - let (root, rust_dir, fork) = keyed(&base); - let k = || Keys::of(&root, &Compiler::primary(&rust_dir), &fork); + let (root, store, fork) = keyed(&base); + let one = placed_compiler(&store, "tree-1"); + let k = || Keys::of(&root, &one, &fork); let was = k(); assert_eq!((was.sysroot.as_str().len(), was.freestanding.as_str().len()), (16, 16), "{was:?}"); let stamp = was.identity.to_string(); @@ -1283,20 +1251,19 @@ mod tests { same(manifest); } - let compiler = Compiler::primary(&rust_dir).key(); + let compiler = one.key(); let config = keyed_std_config(); - assert_eq!(freestanding_key_of(&root, &compiler, &fork, RECIPE, &config), was.freestanding); - assert_ne!(freestanding_key_of(&root, &compiler, &fork, RECIPE, ""), was.freestanding, + assert_eq!(freestanding_key_of(&root, compiler, &fork, RECIPE, &config), was.freestanding); + assert_ne!(freestanding_key_of(&root, compiler, &fork, RECIPE, ""), was.freestanding, "the freestanding key reads no std configuration"); for (what, moved) in [ - ("the recipe", freestanding_key_of(&root, &compiler, &fork, "another recipe", &config)), - ("std's configuration", freestanding_key_of(&root, &compiler, &fork, RECIPE, &format!("{config}\n[rust]\n"))), + ("the recipe", freestanding_key_of(&root, compiler, &fork, "another recipe", &config)), + ("std's configuration", freestanding_key_of(&root, compiler, &fork, RECIPE, &format!("{config}\n[rust]\n"))), ] { assert_ne!(moved, was.freestanding, "{what} kept the old freestanding libraries"); } - write(&rust_dir.join("build/toyos-compiler"), "tree-2"); - let now = k(); + let now = Keys::of(&root, &placed_compiler(&store, "tree-2"), &fork); assert!(now.sysroot != was.sysroot && now.freestanding != was.freestanding, "another compiler kept a key: {now:?}"); assert_eq!(now.identity.stale(Some(&stamp)), Some(Stale::All), "another compiler kept a crate's host half"); } @@ -1310,7 +1277,8 @@ mod tests { #[test] fn a_submodule_is_the_commit_its_gitlink_records_checked_out_or_not() { let base = TempDir::new("key-submodule"); - let (root, rust_dir, _) = keyed(&base); + let (root, store, _) = keyed(&base); + let one = placed_compiler(&store, "tree-1"); let backtrace = base.join("backtrace-src"); write(&backtrace.join("src/lib.rs"), "pub fn trace() {}\n"); git(&backtrace, &["init", "-q"]); @@ -1326,7 +1294,7 @@ mod tests { let clone = base.join("clone"); git(&base, &["clone", "-q", fork.to_str().unwrap(), clone.to_str().unwrap()]); - let k = || freestanding_key(&root, &Compiler::primary(&rust_dir).key(), &clone); + let k = || freestanding_key(&root, one.key(), &clone); assert!(fs::read_dir(clone.join("library/backtrace")).unwrap().next().is_none(), "the clone checked its submodule out"); let unchecked = k(); git(&clone, &["submodule", "update", "-q", "--init", "library/backtrace"]); @@ -1344,29 +1312,31 @@ mod tests { } /// **A sysroot's recorded witness is the one its build wrote**, read back - /// whole; a `SOURCES` naming no fork records none. + /// whole; a `SOURCES` naming a key alone, as the freestanding libraries' + /// does, records none. #[test] fn a_sysroot_records_the_witness_it_was_built_from() { let dir = TempDir::new("recorded-witness"); let witness = "toyos-abi/src/lib.rs:0011223344556677\ntoyos/Cargo.toml:8899aabbccddeeff"; - fs::write(dir.join(SOURCES), sources_text(&Key::of(b"a sysroot"), Path::new("/a/fork/rust"), witness)).unwrap(); + fs::write(dir.join(SOURCES), sources_text(&Key::of(b"a sysroot"), witness)).unwrap(); assert_eq!(recorded_witness(&dir), Ok(witness.to_string())); fs::write(dir.join(SOURCES), "0123456789abcdef\n").unwrap(); - assert!(recorded_witness(&dir).is_err(), "a SOURCES with no fork line recorded a witness"); + assert!(recorded_witness(&dir).is_err(), "a SOURCES naming a key alone recorded a witness"); } - /// **A crate's compiler is the one that built it, rebuilt in place or - /// not**: the primary's is rebuilt where it stands, so a driver its rebuild - /// left leaves all of a crate stale, and the same driver none of it. + /// **A crate's compiler is the one that built it, made again or not**: a + /// compiler swept and made again under its key leaves another driver, which + /// leaves all of a crate stale, and the same driver none of it. #[test] - fn a_compiler_rebuilt_in_place_leaves_all_of_a_crate_stale() { + fn a_compiler_made_again_under_its_key_leaves_all_of_a_crate_stale() { let base = TempDir::new("identity-compiler"); - let (root, rust_dir, fork) = keyed(&base); - let identity = || Keys::of(&root, &Compiler::primary(&rust_dir), &fork).identity; + let (root, store, fork) = keyed(&base); + let one = placed_compiler(&store, "tree-1"); + let identity = || Keys::of(&root, &one, &fork).identity; let stamp = identity().to_string(); assert_eq!(identity().stale(Some(&stamp)), None, "a crate was stale against the compiler that built it"); - write(&toolchain::stage2(&rust_dir).join("lib/librustc_driver-1.dylib"), "a driver, rebuilt in place"); - assert_eq!(identity().stale(Some(&stamp)), Some(Stale::All), "a crate kept what the compiler made before its rebuild"); + write(&one.stage2.join("lib/librustc_driver-1.dylib"), "a driver, made again"); + assert_eq!(identity().stale(Some(&stamp)), Some(Stale::All), "a crate kept what the compiler made before it was made again"); } /// **What one compiler compiled in a std build directory is never another's**: @@ -1376,7 +1346,8 @@ mod tests { #[test] fn another_compiler_s_std_build_goes_and_the_same_one_s_stays() { let base = TempDir::new("compiled-by"); - let (_root, rust_dir, _fork) = keyed(&base); + let (_root, store, _fork) = keyed(&base); + let (one, other) = (placed_compiler(&store, "tree-1").identity(), placed_compiler(&store, "tree-2").identity()); let build = base.join("toyos-std"); let compiled = [ build.join("bootstrap/debug/deps/libserde-1.rlib"), @@ -1395,21 +1366,19 @@ mod tests { write(file, "built"); } }; - let identity = || Compiler::primary(&rust_dir).identity(); lay(); - forget_another_compiler(&build, "host", &identity()); + forget_another_compiler(&build, "host", &one); for file in &compiled { assert!(!file.exists(), "{} was kept, and no record names a compiler for it", file.display()); } assert!(downloaded.iter().all(|f| f.is_file()), "a download went"); lay(); - forget_another_compiler(&build, "host", &identity()); + forget_another_compiler(&build, "host", &one); assert!(compiled.iter().all(|f| f.is_file()), "the same compiler's build went"); - write(&rust_dir.join("build/toyos-compiler"), "tree-2"); - forget_another_compiler(&build, "host", &identity()); + forget_another_compiler(&build, "host", &other); for file in &compiled { assert!(!file.exists(), "{} was kept for another compiler", file.display()); } @@ -1422,10 +1391,10 @@ mod tests { #[test] fn a_std_build_under_the_same_compiler_keeps_no_llvm() { let base = TempDir::new("std-llvm"); - let (_root, rust_dir, _fork) = keyed(&base); + let (_root, store, _fork) = keyed(&base); let build = base.join("toyos-std"); let host = host_triple(); - let identity = Compiler::primary(&rust_dir).identity(); + let identity = placed_compiler(&store, "tree-1").identity(); prepare_std_build(&build, &host, &identity, &GUEST_TARGETS.map(GuestTarget::triple)); let llvm = [ build.join(&host).join("ci-llvm/lib/libLLVM.dylib"), @@ -1457,28 +1426,27 @@ mod tests { #[test] fn a_switch_that_cannot_remove_records_nothing_and_the_next_one_removes() { let base = TempDir::new("compiled-by-stuck"); - let (_root, rust_dir, _fork) = keyed(&base); + let (_root, store, _fork) = keyed(&base); let build = base.join("toyos-std"); - let identity = || Compiler::primary(&rust_dir).identity(); + let (one, other) = (placed_compiler(&store, "tree-1").identity(), placed_compiler(&store, "tree-2").identity()); fs::create_dir_all(&build).unwrap(); - forget_another_compiler(&build, "host", &identity()); + forget_another_compiler(&build, "host", &one); let deps = build.join("bootstrap/debug/deps"); write(&deps.join("libserde-1.rlib"), "built"); - write(&rust_dir.join("build/toyos-compiler"), "tree-2"); // A removal that fails as `keystore::remove` does, whoever runs it. let stuck = std::panic::catch_unwind(|| { - forget_another_compiler_by(&build, "host", &identity(), |path| panic!("remove {}: refused", path.display())) + forget_another_compiler_by(&build, "host", &other, |path| panic!("remove {}: refused", path.display())) }); let refusal = stuck.expect_err("a build that could not be removed was taken for removed"); let refusal = refusal.downcast_ref::().expect("a formatted panic"); assert_eq!(*refusal, format!("remove {}: refused", build.join("bootstrap").display())); - assert_ne!(fs::read_to_string(build.join("compiled-by")).unwrap(), identity(), + assert_ne!(fs::read_to_string(build.join("compiled-by")).unwrap(), other, "the new compiler was recorded over a build it did not remove"); - forget_another_compiler(&build, "host", &identity()); + forget_another_compiler(&build, "host", &other); assert!(!build.join("bootstrap").exists(), "the next call kept the build the stuck one could not remove"); - assert_eq!(fs::read_to_string(build.join("compiled-by")).unwrap(), identity()); + assert_eq!(fs::read_to_string(build.join("compiled-by")).unwrap(), other); } /// A primary with the fork as its `rust` submodule at `C1`, the fork's `C2` @@ -1727,55 +1695,10 @@ mod tests { assert_eq!(git(&primary, &["rev-parse", "HEAD"]), superproject, "git ran in the superproject"); } - /// **The primary's fork checkout moves only while no toolchain phase builds - /// from it**: another primary process's bootstrap holds the global lock - /// exclusively with the worktree lock put down, and the move queues for it - /// and moves nothing until it is let go. - #[test] - fn the_primary_s_fork_checkout_moves_only_while_no_toolchain_phase_builds_from_it() { - let base = TempDir::new("fork-primary-global"); - let (primary, _linked, c1, c2) = two_pins(&base); - let fork = primary.join("rust"); - git(&primary, &["update-index", "--cacheinfo", &format!("160000,{c2},rust")]); - git(&primary, &["commit", "-qm", "pins C2"]); - let bootstrap = buildlock::tests::global_phase_elsewhere(&primary); - std::thread::scope(|scope| { - let mover = scope.spawn(|| drop(fork_checkout(&primary, &mut buildlock::shared(&primary, "a build")))); - let deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); - while !buildlock::tests::global_queued(&primary) { - assert!(!mover.is_finished(), "the move took no global lock"); - assert!(std::time::Instant::now() < deadline, "the move never queued for the global lock"); - std::thread::sleep(std::time::Duration::from_millis(5)); - } - assert_eq!(git(&fork, &["rev-parse", "HEAD"]), c1, "the checkout moved under a running bootstrap"); - bootstrap.release(); - mover.join().unwrap(); - }); - assert_eq!(git(&fork, &["rev-parse", "HEAD"]), c2); - } - - /// **The primary's bootstrap decides from the `compiler/` its tree pins**: - /// a fork checkout left at a commit naming another `compiler/` reads as a - /// stale compiler, and once it is moved to the pin the compiler the primary - /// built is current again. - #[test] - fn the_primary_s_bootstrap_decides_from_the_pinned_compiler() { - let scratch = TempDir::new("fork-primary-compiler"); - let (primary, rust_dir, [_, a, _]) = crate::compiler::tests::estate(&scratch); - let pinned = git(&rust_dir, &["rev-parse", "HEAD"]); - git(&rust_dir, &["checkout", "-q", "--detach", &git(&a.join("rust"), &["rev-parse", "HEAD"])]); - assert!(!compiler::primary_is_current(&rust_dir), "another compiler/ read as the one built"); - - let mut lock = buildlock::shared(&primary, "a build"); - assert_eq!(fork_checkout(&primary, &mut lock), rust_dir); - assert_eq!(git(&rust_dir, &["rev-parse", "HEAD"]), pinned); - assert!(compiler::primary_is_current(&rust_dir), "the pinned compiler/ read as stale"); - } - - /// The primary's compiler under `base`: `rustc` and `rust-lld`, and the C - /// toolchain `src/clang.rs` provisions beside them if `clang`; no cargo. - fn primary_compiler(base: &Path, clang: bool) -> Compiler { - let compiler = Compiler::primary(&base.join("rust")); + /// A compiler in `store`: `rustc` and `rust-lld`, and the C toolchain + /// `src/clang.rs` provisions beside them if `clang`; no cargo. + fn compiler_without_cargo(store: &Path, clang: bool) -> Compiler { + let compiler = placed_compiler(store, "a compiler"); write(&compiler.stage2.join("bin/rustc"), "rustc"); let lld = toolchain::rust_lld(&compiler.stage2); write(&lld, "lld"); @@ -1794,16 +1717,16 @@ mod tests { *refused.downcast::().expect("a formatted refusal") } - /// **A sysroot is whole, or it is made again**: a `stage2` without cargo — - /// what bootstrap leaves until the primary completes it — is refused by - /// name and nothing is published, and one found with its `SOURCES` and - /// without its cargo is rebuilt rather than trusted, once. Placing one - /// removes a sysroot no worktree names. + /// **A sysroot is whole, or it is made again**: a compiler without cargo is + /// refused by name and nothing is published, and a sysroot found with its + /// `SOURCES` and without its cargo is rebuilt rather than trusted, once. + /// Placing one removes a sysroot nothing has used for the store's keep + /// time. #[test] fn a_sysroot_is_whole_or_it_is_made_again() { let base = TempDir::new("whole"); - git(&base, &["init", "-q"]); - let compiler = primary_compiler(&base, true); + let sysroots = Keyed::Sysroot.store(&base); + let compiler = compiler_without_cargo(&base, true); let made = std::cell::Cell::new(0); // `most` bounds the makes so far, so a make that loops fails rather than hangs. let make = |dir: &Path, most: usize| { @@ -1816,22 +1739,24 @@ mod tests { }; let key = Key::of(b"fresh"); - let fresh = sysroots_dir(&base.join("rust")).join(&key); + let fresh = sysroots.join(&key); let said = refusal(|| drop(held(&base, &key, &fresh, || make(&fresh, 1)))); - assert!(said.contains("is missing cargo") && said.contains("`cargo run -- --build-only`"), "{said}"); + assert!(said.contains("is missing cargo"), "{said}"); assert!(!fresh.exists() && !fresh.with_extension("partial").exists(), "a sysroot was published from a stage2 without cargo"); assert_eq!(made.get(), 1); let key = Key::of(b"found"); - let dir = sysroots_dir(&base.join("rust")).join(&key); + let dir = sysroots.join(&key); clone_tree(&compiler.stage2, &dir); write(&dir.join(SOURCES), "found\n"); toolchain::provision_toolchain_cargo(&compiler.stage2); - let orphan = sysroots_dir(&base.join("rust")).join(Key::of(b"named by no worktree")); + let unused = Key::of(b"unused for the keep time"); + let orphan = sysroots.join(&unused); fs::create_dir_all(&orphan).unwrap(); + last_used(&base, Keyed::Sysroot, &unused, LONG_AGO); let using = held(&base, &key, &dir, || make(&dir, 2)); assert_eq!(made.get(), 2, "a sysroot without its cargo was trusted because it has SOURCES"); - assert!(!orphan.exists(), "placing a sysroot left one no worktree names"); + assert!(!orphan.exists(), "placing a sysroot left one nothing had used for the keep time"); assert_eq!(toolchain::toolchain_defect(&dir), None); assert!(dir.join("lib/rustlib/x86_64-unknown-toyos/lib/libstd.rlib").is_file()); drop(using); @@ -1840,14 +1765,14 @@ mod tests { } /// **A sysroot that cannot be made whole is refused after one make, never - /// made again**: a `stage2` without clang — what a stopped bootstrap leaves — - /// is refused before any std is built for it, with nothing published, and a - /// make that leaves its sysroot not whole is refused by what it lacks. + /// made again**: a compiler without clang is refused before any std is + /// built for it, with nothing published, and a make that leaves its sysroot + /// not whole is refused by what it lacks. #[test] fn a_sysroot_that_cannot_be_made_whole_is_made_once_and_refused() { let base = TempDir::new("no-clang"); - git(&base, &["init", "-q"]); - let compiler = primary_compiler(&base, false); + let sysroots = Keyed::Sysroot.store(&base); + let compiler = compiler_without_cargo(&base, false); toolchain::provision_toolchain_cargo(&compiler.stage2); let made = std::cell::Cell::new(0); let once = || { @@ -1856,7 +1781,7 @@ mod tests { }; let key = Key::of(b"cloned"); - let dir = sysroots_dir(&base.join("rust")).join(&key); + let dir = sysroots.join(&key); let filled = std::cell::Cell::new(false); let said = refusal(|| { drop(held(&base, &key, &dir, || { @@ -1868,14 +1793,14 @@ mod tests { })) }); assert!(said.contains("carries no") && said.contains("/clang"), "{said}"); - assert!(said.contains(&compiler.stage2.display().to_string()) && said.contains("`cargo run -- --build-only`"), "{said}"); + assert!(said.contains(&compiler.stage2.display().to_string()), "{said}"); assert!(!filled.get(), "a std was built for a sysroot of a compiler without clang"); assert!(!dir.exists() && !dir.with_extension("partial").exists(), "a sysroot was published from a stage2 without clang"); assert_eq!(made.get(), 1); made.set(0); let key = Key::of(b"made"); - let dir = sysroots_dir(&base.join("rust")).join(&key); + let dir = sysroots.join(&key); let said = refusal(|| { drop(held(&base, &key, &dir, || { once(); diff --git a/src/toolchain.rs b/src/toolchain.rs index 5751ed7bc81..c6cfb813a0e 100644 --- a/src/toolchain.rs +++ b/src/toolchain.rs @@ -6,19 +6,9 @@ use std::sync::OnceLock; use crate::arch::Arch; use crate::buildlock; -use crate::buildlock::Scope; use crate::sysroot::{self, Sysroot, SYSROOT_SOURCES}; -/// Whether the primary's compiler needs a bootstrap. `invalidate_hosted` -/// separates "the compiler changed" from "its `rustc` does not run": only the -/// first makes the ToyOS-hosted rustc stale, and rebuilding that one costs -/// minutes. -#[derive(Clone, Copy, PartialEq, Debug)] -struct Bootstrap { - invalidate_hosted: bool, -} - -/// Which checkout holds the `rust/` submodule and the toolchain built from it. +/// Which checkout holds the `rust/` submodule. pub enum Owner { Us, /// The primary checkout, named so a refusal can point at it. @@ -34,15 +24,8 @@ pub enum Owner { Installed, } -/// One `rust/` per repository, in the primary checkout, and every worktree -/// compiles against it. -/// -/// Not a policy — an affordance. A second checkout of that submodule is a -/// 913 MiB clone (git gives a linked worktree its own, sharing no objects), and -/// a second `build/` beside it is 47 GiB. `git worktree add` leaves `rust/` an -/// empty stub, and leaving it empty is what keeps `git status` clean: git -/// refuses a symlink where a gitlink belongs, and errors out of every command -/// rather than just that one. +/// One `rust/` submodule per repository, in the primary checkout: a linked +/// worktree's `rust/` is a git worktree of it (`sysroot::fork_checkout`). pub fn owner(root: &Path) -> Owner { let primary = crate::primary_checkout(root); let same = fs::canonicalize(root).map(|r| r == primary).unwrap_or(false); @@ -64,15 +47,6 @@ pub fn owner(root: &Path) -> Owner { Owner::Elsewhere(primary) } -/// The shared rust checkout: source, `build/`, and the sysroot every worktree -/// compiles against. -pub fn rust_dir(root: &Path) -> PathBuf { - match owner(root) { - Owner::Us | Owner::Installed => root.join("rust"), - Owner::Elsewhere(primary) => primary.join("rust"), - } -} - /// Of the sysroot's sources, the crates std links, which its dep-info names /// by the path cargo resolved; `sdk/std` it names through the fork. const STD_SOURCES: [&str; 2] = ["toyos-abi/src", "toyos/src"]; @@ -113,40 +87,12 @@ impl GuestTarget { } } -/// The one ToyOS the hosted rustc (`system.toml`'s `hosted-rustc`) is built to -/// run on. -pub const HOSTED_ARCH: Arch = Arch::X86_64; - -/// The primary's compiler, which every sysroot is cloned from and compiled by. +/// The toolchain directory of a checkout whose toolchain arrived as an artifact +/// ([`Owner::Installed`], `src/release.rs`). pub(crate) fn stage2(rust_dir: &Path) -> PathBuf { rust_dir.join(format!("build/{}/stage2", host_triple())) } -/// The ToyOS-hosted rustc's toolchain directory, beside the primary's compiler. -fn hosted_stage2(rust_dir: &Path) -> PathBuf { - rust_dir.join(format!("build/{}/stage2", HOSTED_ARCH.userland())) -} - -/// Whether the primary builds the hosted rustc: a build whose config ships it -/// `asked`, and `rustc` is not there or `stamp`, which says it is this -/// compiler's, is not. -fn hosted_rustc_owed(asked: bool, stamp: &Path, rustc: &Path) -> bool { - asked && (!stamp.exists() || !rustc.exists()) -} - -/// Remove the hosted rustc a compiler rebuild left stale, and its `stamp`: no -/// build reads one until a config that ships it asks, and that build makes it -/// anew. -fn forget_hosted_rustc(rust_dir: &Path, stamp: &Path) { - let gone = |path: &Path, removed: std::io::Result<()>| match removed { - Err(e) if e.kind() != std::io::ErrorKind::NotFound => panic!("remove {}: {e}", path.display()), - _ => {} - }; - gone(stamp, fs::remove_file(stamp)); - let stale = hosted_stage2(rust_dir); - gone(&stale, fs::remove_dir_all(&stale)); -} - /// Every `toyos-abi`/`toyos` source file a std build under `dep_info` actually /// compiled, read out of cargo's dep-info rather than out of what was asked for. fn std_toyos_sources(dep_info: &Path) -> Vec { @@ -309,11 +255,6 @@ pub(crate) fn rustup_home() -> Option { .or_else(|| std::env::var_os("HOME").map(|h| PathBuf::from(h).join(".rustup"))) } -/// Where the machine-global `toyos` rustup toolchain currently points. -pub(crate) fn rustup_link() -> Option { - fs::read_link(rustup_home()?.join("toolchains/toyos")).ok() -} - /// The binaries rustup proxies out of a linked toolchain's `bin/`. /// /// A name that is not there is a name rustup answers for by falling back to @@ -349,9 +290,9 @@ fn narrated_binaries(bin: &Path) -> Vec<&'static str> { /// "the host cargo" flatly would have taken that away from the dev host and /// called it a cleanup. /// -/// The host's is resolved through `rustc --print sysroot` for the same reason -/// [`link_host_target`] does: it is whatever stable toolchain this machine has, -/// and it is not a path any artifact can know. +/// The host's is resolved through `rustc --print sysroot`: it is whatever +/// stable toolchain this machine has, and it is not a path any artifact can +/// know. fn host_cargo() -> &'static Path { static CARGO: OnceLock = OnceLock::new(); CARGO.get_or_init(|| { @@ -386,8 +327,7 @@ fn cargo_link_stale(stage2: &Path) -> bool { /// /// **A symlink, and what survives the artifact round-trip is this step rather /// than the link.** `src/release.rs` excludes it from the tarball: it names a -/// path only the publishing runner has. `Owner::Installed` makes it, exactly as -/// it makes the host target. +/// path only the publishing runner has. `Owner::Installed` makes it. pub(crate) fn provision_toolchain_cargo(stage2: &Path) { let at = stage2.join("bin/cargo"); let _ = fs::remove_file(&at); @@ -421,7 +361,7 @@ pub(crate) fn toolchain_defect(stage2: &Path) -> Option { if !lld.is_file() { return Some(format!( "the toyos toolchain at {} carries no {}, the linker every guest target names: \ - bootstrap puts it there when `write_config` says `lld = true`, and it did not", + bootstrap puts it there when `compiler::config_text` says `lld = true`, and it did not", stage2.display(), lld.display(), )); @@ -440,112 +380,13 @@ pub(crate) fn assert_toolchain_is_honest(stage2: &Path) { } } -/// Whether the primary's toolchain lacks what bootstrap does not put there: -/// `stage2`'s cargo and clang, or the host target in the hosted rustc's sysroot. -fn incomplete(rust_dir: &Path) -> bool { - let stage2 = stage2(rust_dir); - cargo_link_stale(&stage2) || crate::clang::defect(&stage2).is_some() || host_target_missing(rust_dir) -} - -/// Give the primary's toolchain what [`incomplete`] finds missing, its clang by -/// `provision_clang`. -fn complete(rust_dir: &Path, provision_clang: impl FnOnce(&Path)) { - let stage2 = stage2(rust_dir); - if cargo_link_stale(&stage2) { - provision_toolchain_cargo(&stage2); - } - if crate::clang::defect(&stage2).is_some() { - provision_clang(&stage2); - } - if host_target_missing(rust_dir) { - link_host_target(rust_dir); - } -} - -/// Run `bootstrap` in the primary's `rust/` against the LLVM at `llvm`, then -/// remove the LLVM its build directory built itself (`llvm::retire_in_tree`) -/// and [`complete`] what it reassembled. Called inside the act that holds the -/// global lock exclusively. -/// -/// **In the same hold, because bootstrap recreates `stage2` without its cargo -/// and clang**: a completion under a hold of its own queues behind every sysroot -/// build that takes the lock shared in between, and those last minutes. -fn reassemble(rust_dir: &Path, llvm: &Path, bootstrap: impl FnOnce()) { - bootstrap(); - crate::llvm::retire_in_tree(&rust_dir.join("build")); - complete(rust_dir, |stage2| crate::clang::provision(stage2, llvm)); -} - -/// [`reassemble`] the primary's compiler with `bootstrap`, with nothing recording -/// which compiler `stage2` is until it is whole: a bootstrap that is stopped is -/// run again by the primary, and refused by name in every linked worktree. -fn rebuild_compiler(rust_dir: &Path, llvm: &Path, bootstrap: impl FnOnce()) { - crate::compiler::forget(rust_dir); - reassemble(rust_dir, llvm, bootstrap); - crate::compiler::record(rust_dir); -} - -/// What the primary bootstraps: a new compiler when `stage2` is not the one its -/// fork checkout names, and the same one again when its `rustc` does not run. -/// A `stage2` that runs and has no rustup link, as one a runner restored, is -/// linked, not rebuilt. -fn bootstrap(current: bool, runs: bool) -> Option { - if !current { - Some(Bootstrap { invalidate_hosted: true }) - } else { - (!runs).then_some(Bootstrap { invalidate_hosted: false }) - } -} - -/// Whether the `rustc` in `stage2` runs. -fn runs(stage2: &Path) -> bool { - Command::new(stage2.join("bin/rustc")) - .arg("--version") - .stdout(std::process::Stdio::null()) - .stderr(std::process::Stdio::null()) - .status() - .is_ok_and(|s| s.success()) -} - -/// Ensure the toolchain is up to date, and return the sysroot this checkout's -/// sources name — made if nobody has made it (`src/sysroot.rs`). The primary -/// builds the ToyOS-hosted rustc only for a build whose config ships it -/// (`hosted_rustc`). -/// -/// Every step decides under the caller's shared lock and acts under the -/// exclusive one, so the common answer — nothing to do — costs no -/// serialisation, and two agents cannot both conclude the compiler is stale -/// and both start `x.py build` in the same directory. -/// -/// The steps are ordered, and each invalidates what it makes stale rather than -/// threading a `rebuilt` flag through: a step that decides for itself still -/// decides correctly when the process before it was killed halfway. -/// -/// Only the primary checkout builds the compiler. Every checkout builds the -/// sysroot its own sources name, in its own fork checkout. -/// -/// **The lock only covers builds routed through here.** A `./x.py build` typed -/// by hand in `rust/` takes no lock at all, and can still lose the race this -/// serialises: one builder's bootstrap removes and recreates -/// `stage1-std//dist/deps` while another's `rustc` creates a temp file -/// inside it, and the loser dies compiling `core` with `couldn't create a temp -/// dir: No such file or directory`. -pub fn ensure(root: &Path, lock: &mut buildlock::Held, hosted_rustc: bool) -> Sysroot { - let stamps_dir = root.join("target/stamps"); - fs::create_dir_all(&stamps_dir).ok(); - - let rust_dir = match owner(root) { - Owner::Elsewhere(primary) => { - let rust_dir = primary.join("rust"); - assert!( - stage2(&rust_dir).join("bin/rustc").exists(), - "there is no compiler to build with: {} does not exist.\n\ - The primary checkout builds it — run `cargo run -- --build-only` in {} first.", - stage2(&rust_dir).display(), - primary.display() - ); - return sysroot::ensure(root, &rust_dir, lock); - } +/// The sysroot this checkout's sources name, held in use: the host's store's +/// (`src/keystore.rs`), made if nobody on the host has made it +/// (`src/sysroot.rs`), or, in a checkout whose toolchain arrived as an +/// artifact, that one. +pub fn ensure(root: &Path, lock: &mut buildlock::Held) -> Sysroot { + match owner(root) { + Owner::Us | Owner::Elsewhere(_) => sysroot::ensure(root, &crate::keystore::host(), lock), Owner::Installed => { let rust_dir = root.join("rust"); check_installed_toolchain(root, &rust_dir); @@ -553,83 +394,17 @@ pub fn ensure(root: &Path, lock: &mut buildlock::Held, hosted_rustc: bool) -> Sy let release = fs::read_to_string(&release).unwrap_or_else(|e| { panic!("{}: {e}; an installed toolchain carries the TOOLCHAIN it was installed with", release.display()) }); - return Sysroot::installed(stage2(&rust_dir), &release); + Sysroot::installed(stage2(&rust_dir), &release) } - Owner::Us => sysroot::fork_checkout(root, lock), - }; - let hosted_stamp = stamps_dir.join("hosted-rustc.stamp"); - lock.act_if( - Scope::Global, - "build the rust toolchain", - || bootstrap(crate::compiler::primary_is_current(&rust_dir), runs(&stage2(&rust_dir))), - |kind| { - eprintln!("Building full toolchain (this takes a while on first run)..."); - let llvm = crate::llvm::resolve(root, &rust_dir, &rust_dir); - rebuild_compiler(&rust_dir, &llvm.dir, || full_bootstrap(&rust_dir, &llvm.dir)); - if kind.invalidate_hosted { - forget_hosted_rustc(&rust_dir, &hosted_stamp); - } - }, - ); - - let hosted = hosted_stage2(&rust_dir).join("bin/rustc"); - lock.act_if( - Scope::Global, - "build the ToyOS-hosted rustc", - || hosted_rustc_owed(hosted_rustc, &hosted_stamp, &hosted).then_some(()), - |()| { - let llvm = crate::llvm::resolve(root, &rust_dir, &rust_dir); - reassemble(&rust_dir, &llvm.dir, || build_hosted_rustc(&rust_dir, &llvm.dir)); - assert!(hosted.exists(), "Failed to build hosted rustc"); - fs::write(&hosted_stamp, "").unwrap(); - }, - ); - - let stage2 = stage2(&rust_dir); - lock.act_if( - Scope::Global, - "link the toyos rustup toolchain", - || link_stale(&stage2).then_some(()), - |()| { - let status = Command::new("rustup") - .args(["toolchain", "link", "toyos", stage2.to_str().unwrap()]) - .status() - .unwrap_or_else(|e| panic!("Failed to run rustup: {e}")); - assert!(status.success(), "rustup toolchain link failed"); - }, - ); - - lock.act_if( - Scope::Global, - "complete the toyos toolchain", - || incomplete(&rust_dir).then_some(()), - |()| complete(&rust_dir, |stage2| crate::clang::provision(stage2, &crate::llvm::resolve(root, &rust_dir, &rust_dir).dir)), - ); - assert_toolchain_is_honest(&stage2); - - sysroot::ensure(root, &rust_dir, lock) + } } /// Everything a checkout may do with a toolchain it did not build: check that /// it is the one this tree needs, and say what to do when it is not. fn check_installed_toolchain(root: &Path, rust_dir: &Path) { let stage2 = stage2(rust_dir); - let linked = rustup_link(); - assert!( - linked.as_deref() == Some(stage2.as_path()), - "the rustup toolchain `toyos` points at {}, not at the installed toolchain at {}.\n\ - Link it: rustup toolchain link toyos {}", - linked.map_or_else(|| "nothing".to_string(), |p| p.display().to_string()), - stage2.display(), - stage2.display(), - ); - - // Recreated rather than shipped: both of these point into whatever stable - // toolchain this machine has, which is not a path any artifact can know. - // Its clang is the artifact's own, so this is not `complete`. - if host_target_missing(rust_dir) { - link_host_target(rust_dir); - } + // Recreated rather than shipped: it points into whatever stable toolchain + // this machine has, which is not a path any artifact can know. if cargo_link_stale(&stage2) { provision_toolchain_cargo(&stage2); } @@ -648,31 +423,6 @@ fn check_installed_toolchain(root: &Path, rust_dir: &Path) { ); } -/// Whether the `toyos` rustup toolchain points anywhere other than `stage2`. -/// -/// `rustup toolchain link` unlinks and recreates the symlink rather than -/// replacing it atomically, so every call opens a window in which -/// `~/.rustup/toolchains/toyos` does not resolve. Any concurrent `rustc` proxy -/// invocation landing in that window dies with `'rustc' is not installed for the -/// custom toolchain 'toyos'` — which reads as a broken toolchain rather than as -/// contention, because a probe run a moment later succeeds. -/// -/// This ran unconditionally on every `ensure`, i.e. every build. With five -/// agents building in one tree it cost one of them eleven consecutive -/// `cargo test` invocations over about fifteen minutes, while -/// `RUSTUP_TOOLCHAIN=toyos rustc --version` succeeded 20 out of 20 between the -/// attempts. -/// -/// A mismatched or absent link still re-links, so a moved tree or a fresh clone -/// behaves as before; only the no-op case is skipped. -/// -/// Reached only from the primary checkout, which is what makes the window above -/// a window of one: a linked worktree that re-linked would point the name at a -/// stage2 nobody else has. -fn link_stale(stage2: &Path) -> bool { - rustup_link().is_none_or(|current| current != stage2) -} - /// Run bootstrap in the fork checkout `rust_dir`, streaming its output where it /// was going anyway and keeping a copy, with both the checkout's lockfiles put /// back as they were when this returns, however the build that holds them @@ -693,7 +443,7 @@ pub(crate) fn x_build(rust_dir: &Path, args: &[&str], what: &str) -> (bool, Vec< /// rustc strips a Darwin binary by running `rust-objcopy` (`rustc_codegen_ssa`'s /// `back/link.rs`), the rust workspace strips `lld-wrapper`, and the stage-1 /// sysroot carries no `rust-objcopy` when bootstrap copies none of LLVM's tools -/// ([`LEAN`]): the build finds that LLVM's `llvm-objcopy` by that name on `PATH`. +/// (`compiler::config_text`): the build finds that LLVM's `llvm-objcopy` by that name on `PATH`. pub(crate) fn x_build_compiler(rust_dir: &Path, args: &[&str], what: &str, llvm: &Path) -> (bool, Vec) { if !host_triple().ends_with("apple-darwin") { return x_build(rust_dir, args, what); @@ -800,193 +550,6 @@ pub(crate) fn refuse_on_compile_error(log: &[String], what: &str) { panic!("{what} did not compile:\n{}", log[at..end].join("\n")); } -/// What an `x build` failure the artifact check is willing to tolerate actually -/// said, so that "expected" is a claim the reader can check. -fn tolerated_failure(log: &[String], what: &str) { - let errors: Vec<&str> = - log.iter().map(String::as_str).filter(|l| l.trim_start().starts_with("error")).collect(); - eprintln!( - "Note: {what} exited non-zero and the artifacts it must produce are all there, so this \ - is the ToyOS rustdoc link failure. It reported:\n{}", - if errors.is_empty() { - " (no line beginning `error`)".to_string() - } else { - errors.join("\n") - } - ); -} - -/// What the primary's compiler build builds: rustc and the host's libraries, -/// which build scripts and proc macros link. No rustdoc: no build runs one. -const COMPILER_BUILD: [&str; 7] = ["build", "--stage", "2", "--warnings", "warn", "compiler/rustc", "library"]; - -fn full_bootstrap(rust_dir: &Path, llvm: &Path) { - // Ensure library/backtrace is checked out — std depends on it. - // Other rust submodules (llvm, docs, cargo) are handled by bootstrap on demand. - crate::ensure_submodule(rust_dir, "library/backtrace"); - - let host = host_triple(); - write_config(rust_dir, &host, false, llvm); - - // Clean cached std for all ToyOS targets so bootstrap picks up compiler changes - // (e.g. target spec changes like default_uwtable that affect codegen). - for target in GUEST_TARGETS { - let stage1_std = rust_dir.join(format!("build/{host}/stage1-std/{}", target.triple())); - if stage1_std.exists() { - fs::remove_dir_all(&stage1_std).ok(); - } - } - - let (ok, log) = x_build_compiler(rust_dir, &COMPILER_BUILD, "the toolchain", llvm); - refuse_on_compile_error(&log, "the toolchain"); - assert!(ok, "the toolchain build failed, and nothing in its output was a compile error"); -} - -fn build_hosted_rustc(rust_dir: &Path, llvm: &Path) { - eprintln!("Building ToyOS-hosted rustc..."); - let host = host_triple(); - write_config(rust_dir, &host, true, llvm); - - let (ok, log) = - x_build_compiler(rust_dir, &["build", "--stage", "2", "--warnings", "warn"], "the hosted rustc", llvm); - refuse_on_compile_error(&log, "the hosted rustc"); - - // rustdoc for ToyOS may fail to link; rustc and librustc_driver may not. - let toyos_stage2 = rust_dir.join(format!("build/{}/stage2", HOSTED_ARCH.userland())); - assert!( - toyos_stage2.join("bin/rustc").exists(), - "the hosted rustc build failed and {} is not there.\n\ - Nothing in its output was a compile error, so this is a link or a bootstrap failure.", - toyos_stage2.join("bin/rustc").display() - ); - assert!( - fs::read_dir(toyos_stage2.join("lib")) - .map(|d| d.filter_map(|e| e.ok()) - .any(|e| e.file_name().to_string_lossy().starts_with("librustc_driver"))) - .unwrap_or(false), - "the hosted rustc build failed: librustc_driver*.so is not in {}", - toyos_stage2.join("lib").display() - ); - if !ok { - tolerated_failure(&log, "the hosted rustc build"); - } - - // That build reassembled the host's `stage2` without `rust-lld` - // (`write_config` says why), so the host-only build runs once more to put - // it back: everything it would compile is already built. - write_config(rust_dir, &host, false, llvm); - let (ok, log) = x_build_compiler(rust_dir, &COMPILER_BUILD, "the toolchain, reassembled", llvm); - refuse_on_compile_error(&log, "the toolchain, reassembled"); - assert!( - rust_lld(&stage2(rust_dir)).is_file(), - "the toolchain's reassembly after the hosted rustc left no {}", - rust_lld(&stage2(rust_dir)).display() - ); - if !ok { - tolerated_failure(&log, "the toolchain's reassembly"); - } -} - -/// `bootstrap.toml` for the host-only toolchain, every compiler's -/// (`compiler::config_text`), or with the ToyOS-hosted rustc. -/// -/// `lld = true` is what puts `rust-lld` in every stage's sysroot, where rustc -/// finds the linker every guest target names. The hosted rustc's build cannot -/// have it: bootstrap would then build LLD for the ToyOS host from C++, which -/// nothing here can compile yet. Every assemble removes the host's -/// `stage2` first, so [`build_hosted_rustc`] reassembles it under the host-only -/// config after. -/// -/// `clang::LLVM_CONFIG` is the `[llvm]` both builds share, and both link the LLVM -/// at `llvm` (`src/llvm.rs`), whose LLD every guest target names by path. -/// -/// The host's `default-linker-linux-override` is pinned off because bootstrap -/// otherwise ties it to `lld` for `x86_64-unknown-linux-gnu`, and a host rustc -/// whose build environment flips with the config is rebuilt by each of those -/// two builds. -/// -/// The host-only toolchain builds no guest target's libraries: every sysroot -/// builds its own (`src/sysroot.rs`). -fn write_config(rust_dir: &Path, host: &str, with_hosted_rustc: bool, llvm: &Path) { - let config = if with_hosted_rustc { - let targets = std::iter::once(host) - .chain(GUEST_TARGETS.map(GuestTarget::triple)) - .map(|t| format!("\"{t}\"")) - .collect::>() - .join(", "); - format!( - r#"change-id = "ignore" -profile = "compiler" - -[build] -host = ["{host}", "{hosted}"] -target = [{targets}] - -[llvm] -{llvm} - -[rust] -incremental = true -lld = false -{LEAN} - -[target.{host}] -{HOST_LINKER_PIN} -{external} - -{userland}"#, - hosted = HOSTED_ARCH.userland(), - llvm = crate::clang::LLVM_CONFIG, - external = crate::llvm::host_lines(llvm), - userland = hosted_targets(llvm), - ) - } else { - crate::compiler::config_text(&rust_dir.join("build"), host, llvm) - }; - fs::write(rust_dir.join("bootstrap.toml"), config).unwrap(); -} - -/// The `[target]` sections of ToyOS userland in the hosted rustc's build: each -/// links with the LLD of the LLVM at `llvm`, and the one the hosted rustc runs -/// on builds it. -fn hosted_targets(llvm: &Path) -> String { - Arch::ALL - .iter() - .map(|arch| { - let linker = format!("linker = \"{}\"", llvm.join("bin/lld").display()); - let hosted = if *arch == HOSTED_ARCH { - // Cranelift because no LLVM is built for a ToyOS host yet, and - // only for that reason: the hosted rustc carries LLVM once clang - // and libc++ run on ToyOS, and Cranelift is not where the - // compiler that builds ToyOS goes. - // - // The archiver is that LLVM's too: the compiler's crates carry - // C built for this target (blake3's assembly), and a host `ar` - // that indexes only its own object format, as macOS's does, - // leaves those ELF members out of the index lld pulls from. - format!( - "\nar = \"{}\"\ncodegen-backends = [\"cranelift\"]", - llvm.join("bin/llvm-ar").display(), - ) - } else { - String::new() - }; - format!("[target.{}]\n{linker}{hosted}\nrpath = false\n\n", arch.userland()) - }) - .collect() -} - -/// What the host rustc links its own binaries with, held to one answer in every -/// `bootstrap.toml` that builds a host compiler: [`write_config`] says why. -pub(crate) const HOST_LINKER_PIN: &str = "default-linker-linux-override = \"off\""; - -/// The `[rust]` options every `bootstrap.toml` that builds a compiler shares -/// beyond its profile's: no LLVM tool copied into the compiler's sysroot, since -/// `clang::provision` puts there the ones a build runs; no debuginfo in rustc, -/// which no build reads; and no codegen test, for which bootstrap demands -/// LLVM's `FileCheck` beside `llvm-config` (`src/bootstrap/src/core/sanity.rs`). -pub(crate) const LEAN: &str = "llvm-tools = false\ndebuginfo-level-rustc = 0\ncodegen-tests = false"; - /// The linker every guest target names, as the toolchain at `toolchain` carries /// it: `lib/rustlib//bin/rust-lld`, where rustc itself looks for it. pub fn rust_lld(toolchain: &Path) -> PathBuf { @@ -1016,9 +579,7 @@ pub fn host_triple() -> String { /// The stable host toolchain's sysroot, as `rustc --print sysroot` reports it. /// -/// Both [`host_cargo`] and [`link_host_target`] resolve their answer through it -/// for the one reason [`host_cargo`]'s doc gives: it is whatever stable -/// toolchain this machine has, and that is not a path any artifact can name. +/// [`host_cargo`] resolves its answer through it, for the reason its doc gives. fn host_sysroot() -> PathBuf { let output = Command::new("rustc") .args(["--print", "sysroot"]) @@ -1028,36 +589,6 @@ fn host_sysroot() -> PathBuf { PathBuf::from(sysroot.trim()) } -/// Whether the ToyOS sysroot is missing the host target proc-macros compile against. -fn host_target_missing(rust_dir: &Path) -> bool { - let toyos_sysroot = rust_dir.join(format!("build/{}/stage2/lib/rustlib", HOSTED_ARCH.userland())); - toyos_sysroot.exists() && !toyos_sysroot.join(host_triple()).exists() -} - -fn link_host_target(rust_dir: &Path) { - let host = host_triple(); - let host_target_dir = rust_dir - .join(format!("build/{}/stage2/lib/rustlib", HOSTED_ARCH.userland())) - .join(&host); - - let source = host_sysroot().join("lib/rustlib").join(&host); - assert!( - source.exists(), - "Host target {} not found in stable toolchain at {}", - host, - source.display() - ); - - std::os::unix::fs::symlink(&source, &host_target_dir).unwrap_or_else(|e| { - panic!( - "Failed to symlink {} -> {}: {}", - host_target_dir.display(), - source.display(), - e - ) - }); -} - #[cfg(test)] mod tests { use super::*; @@ -1197,35 +728,6 @@ mod tests { assert!(said.contains("clang") && !said.contains("rust-lld,"), "the refusal names clang alone: {said}"); } - /// Every build links the host's LLVM, copies none of its tools and leaves - /// rustc without debuginfo; the host-only one builds no guest target, and - /// in the hosted rustc's each guest links through that LLVM's LLD, named - /// by path. - #[test] - fn every_build_links_the_host_s_llvm_and_names_its_lld_by_path() { - let rust_dir = TempDir::new("lld-config"); - let llvm = rust_dir.join("build/llvm/k"); - let lld = format!("linker = \"{}\"", llvm.join("bin/lld").display()); - let lean = "\nllvm-tools = false\ndebuginfo-level-rustc = 0\ncodegen-tests = false\n"; - for (hosted, lld_flag) in [(true, "false"), (false, "true")] { - write_config(&rust_dir, "h", hosted, &llvm); - let config = fs::read_to_string(rust_dir.join("bootstrap.toml")).unwrap(); - assert!(config.contains(&format!("\n[rust]\nincremental = true\nlld = {lld_flag}{lean}")), "{config}"); - assert_eq!(config.contains(&lld), hosted, "{config}"); - assert!(!config.contains("\"rust-lld\""), "{config}"); - assert_eq!(config.contains("\ntarget = [\"h\"]\n"), !hosted, "{config}"); - let host = format!( - "[target.h]\ndefault-linker-linux-override = \"off\"\nllvm-config = \"{}/bin/llvm-config\"\nllvm-has-rust-patches = true\n", - llvm.display() - ); - assert!(config.contains(&host), "{config}"); - if !hosted { - let keyed = crate::compiler::config_text(&rust_dir.join("build"), "h", &llvm); - assert_eq!(config, keyed, "the primary's compiler is built under a configuration its key does not read"); - } - } - } - /// **Bootstrap never sees GitHub Actions' variables**, whatever its /// caller's environment: it takes `HEAD^1`'s artifacts when it does. `./x` /// here is this test binary, running [`a_fake_bootstrap_that_reads_ci`]. @@ -1254,186 +756,6 @@ mod tests { println!("GITHUB_ACTIONS {}, CI {}", read("GITHUB_ACTIONS"), read("CI")); } - /// **A bootstrap leaves the primary nothing that waits on another - /// worktree's sysroot build**: the act that reassembles `stage2` completes it - /// before its exclusive hold ends, so the step after it — run while a - /// sysroot build holds the lock shared — decides it has nothing to do, and - /// takes no lock. - #[test] - fn a_bootstrap_leaves_nothing_to_wait_on_a_sysroot_build_for() { - let rust_dir = TempDir::new("no-wait"); - let stage2 = stage2(&rust_dir); - let llvm = store_llvm(&rust_dir); - reassemble(&rust_dir, &llvm, || bootstrapped(&rust_dir)); - assert!( - !incomplete(&rust_dir), - "a bootstrap let its exclusive hold go with a global step left, which the primary's \ - build then queues for behind every sysroot build" - ); - assert_eq!(toolchain_defect(&stage2), None, "a bootstrap let its exclusive hold go with stage2 not whole"); - } - - /// The LLVM `clang::provision` reads, in `rust_dir`'s store. - fn store_llvm(rust_dir: &Path) -> PathBuf { - let llvm = rust_dir.join("build/llvm/k"); - let files = [ - ("bin/clang", "clang"), - ("bin/llvm-ar", "llvm-ar"), - ("bin/llvm-objcopy", "llvm-objcopy"), - ("lib/clang/22/include/stddef.h", "stddef"), - ]; - for (file, text) in files { - fs::create_dir_all(llvm.join(file).parent().unwrap()).unwrap(); - fs::write(llvm.join(file), text).unwrap(); - } - llvm - } - - /// What bootstrap leaves in `rust_dir`: `stage2` made again, with `rustc` - /// and `rust-lld` and none of cargo, clang or an LLVM tool; and the hosted - /// rustc's sysroot without the host target. - fn bootstrapped(rust_dir: &Path) { - let stage2 = stage2(rust_dir); - let _ = fs::remove_dir_all(&stage2); - for file in [stage2.join("bin/rustc"), rust_lld(&stage2)] { - fs::create_dir_all(file.parent().unwrap()).unwrap(); - fs::write(file, b"").unwrap(); - } - let hosted = rust_dir.join(format!("build/{}/stage2/lib/rustlib", HOSTED_ARCH.userland())); - fs::create_dir_all(&hosted).unwrap(); - } - - /// What a build directory holds of an LLVM of its own: bootstrap's LLVM and - /// LLD. - fn in_tree_llvm(rust_dir: &Path) -> [PathBuf; 2] { - let host = rust_dir.join("build").join(host_triple()); - let own = [host.join("llvm"), host.join("lld")]; - for dir in &own { - fs::create_dir_all(dir.join("bin")).unwrap(); - fs::write(dir.join("bin/tool"), "a tool").unwrap(); - } - own - } - - /// **A primary whose compiler is built against the store keeps no LLVM of - /// its own**: the rebuild that links the store removes the one its build - /// directory built. - #[test] - fn a_rebuilt_compiler_leaves_no_llvm_of_its_own() { - let scratch = TempDir::new("in-tree-llvm"); - let (_primary, rust_dir, _) = crate::compiler::tests::estate(&scratch); - let own = in_tree_llvm(&rust_dir); - let llvm = store_llvm(&rust_dir); - rebuild_compiler(&rust_dir, &llvm, || bootstrapped(&rust_dir)); - for dir in own { - assert!(!dir.exists() && !dir.with_extension("swept").exists(), "{} outlived the rebuild", dir.display()); - } - assert_eq!(toolchain_defect(&stage2(&rust_dir)), None); - } - - /// **Landing the store moves an existing primary onto it**: a primary whose - /// record was written before its compiler linked the host's LLVM is not - /// current, so its next build bootstraps, and that rebuild removes the LLVM - /// and LLD its build directory built. Its LLVM checkout sitting at a commit - /// its gitlink does not name changes neither answer. - #[test] - fn a_primary_recorded_before_the_store_is_rebuilt_onto_it() { - let scratch = TempDir::new("store-migration"); - let (_primary, rust_dir, _) = crate::compiler::tests::estate(&scratch); - crate::compiler::tests::llvm_checkout(&rust_dir); - assert!(crate::compiler::primary_is_current(&rust_dir)); - crate::compiler::tests::record_before_the_store(&rust_dir); - let own = in_tree_llvm(&rust_dir); - let kind = bootstrap(crate::compiler::primary_is_current(&rust_dir), true); - assert!(kind.is_some_and(|k| k.invalidate_hosted), "a primary recorded before the store was taken for current"); - rebuild_compiler(&rust_dir, &store_llvm(&rust_dir), || bootstrapped(&rust_dir)); - assert!(own.iter().all(|dir| !dir.exists()), "the rebuild kept the LLVM its build directory built"); - assert!(crate::compiler::primary_is_current(&rust_dir), "the rebuild recorded a compiler that is not current"); - } - - /// **A stopped bootstrap is run again**: nothing records which compiler - /// `stage2` is while one runs, so the primary's next build is not told the - /// old one is current; and the LLVM the old one linked stays. - #[test] - fn a_stopped_bootstrap_leaves_no_record() { - let rust_dir = TempDir::new("stopped"); - let record = crate::compiler::primary_record(&rust_dir); - fs::create_dir_all(record.parent().unwrap()).unwrap(); - fs::write(&record, "the compiler before").unwrap(); - let own = in_tree_llvm(&rust_dir); - let stopped = std::panic::catch_unwind(|| rebuild_compiler(&rust_dir, Path::new("no-llvm"), || panic!("stopped"))); - assert!(stopped.is_err()); - assert!(!record.exists(), "a stopped bootstrap left the record of the compiler before it"); - assert!(own.iter().all(|dir| dir.join("bin/tool").is_file()), "a stopped bootstrap took the LLVM its compiler linked"); - } - - /// **The hosted rustc is built only for a build whose config ships it**, - /// and then only when the one this compiler built is not there. - #[test] - fn the_hosted_rustc_is_built_only_when_a_build_ships_it() { - let rust_dir = TempDir::new("hosted-owed"); - let stamp = rust_dir.join("stamps/hosted-rustc.stamp"); - let rustc = hosted_stage2(&rust_dir).join("bin/rustc"); - assert!(!hosted_rustc_owed(false, &stamp, &rustc), "a build that ships no hosted rustc built one"); - assert!(hosted_rustc_owed(true, &stamp, &rustc)); - for file in [&stamp, &rustc] { - fs::create_dir_all(file.parent().unwrap()).unwrap(); - fs::write(file, "").unwrap(); - } - assert!(!hosted_rustc_owed(true, &stamp, &rustc), "a hosted rustc this compiler built was built again"); - fs::remove_file(&stamp).unwrap(); - assert!(hosted_rustc_owed(true, &stamp, &rustc), "a hosted rustc of another compiler was taken"); - } - - /// **A compiler rebuild leaves no hosted rustc of the compiler it - /// replaced**, nor its stamp; with neither there, that is no error. - #[test] - fn a_rebuilt_compiler_leaves_no_hosted_rustc_of_the_one_before() { - let rust_dir = TempDir::new("hosted-stale"); - let stamp = rust_dir.join("stamps/hosted-rustc.stamp"); - let rustc = hosted_stage2(&rust_dir).join("bin/rustc"); - for file in [&stamp, &rustc] { - fs::create_dir_all(file.parent().unwrap()).unwrap(); - fs::write(file, "").unwrap(); - } - forget_hosted_rustc(&rust_dir, &stamp); - assert!(!stamp.exists() && !hosted_stage2(&rust_dir).exists(), "the old compiler's hosted rustc stayed"); - forget_hosted_rustc(&rust_dir, &stamp); - } - - /// **The primary bootstraps a new compiler exactly when its `stage2` is not - /// current, and otherwise only when rustup has none.** - #[test] - fn the_primary_bootstraps_when_stale_or_missing() { - let new = Some(Bootstrap { invalidate_hosted: true }); - let again = Some(Bootstrap { invalidate_hosted: false }); - for (current, runs, want) in [ - (true, true, None), - (true, false, again), - (false, true, new), - (false, false, new), - ] { - assert_eq!(bootstrap(current, runs), want, "current {current}, runs {runs}"); - } - } - - /// **What decides a rebuild of a current `stage2` is whether its `rustc` - /// runs**, not whether rustup names it: one a runner restored has no - /// rustup link and is linked, not built again. - #[test] - fn a_compiler_s_rustc_runs_or_it_is_built_again() { - let stage2 = TempDir::new("runs"); - let rustc = stage2.join("bin/rustc"); - assert!(!runs(&stage2), "a stage2 with no rustc ran"); - fs::create_dir_all(rustc.parent().unwrap()).unwrap(); - // This test binary refuses `--version`; the host's rustc answers it. - for (what, ran) in [(std::env::current_exe().unwrap(), false), (host_sysroot().join("bin/rustc"), true)] { - let _ = fs::remove_file(&rustc); - std::os::unix::fs::symlink(&what, &rustc).unwrap(); - assert_eq!(runs(&stage2), ran, "{}", what.display()); - } - } - /// The negative control is the defect itself: this is verbatim what cargo /// wrote for a worktree build before the override existed. #[test] diff --git a/src/userlandhost.rs b/src/userlandhost.rs index 8309d8848b8..1b4b250bab6 100644 --- a/src/userlandhost.rs +++ b/src/userlandhost.rs @@ -187,7 +187,7 @@ pub struct Program { /// declares about the hosts. pub fn programs(root: &Path) -> Result, String> { let mut found = Vec::new(); - for dir in crate::build::shipped(root)?.programs { + for dir in crate::build::shipped(root).programs { let at = rel(root, &dir); let text = std::fs::read_to_string(dir.join("Cargo.toml")) .map_err(|e| format!("{at}/Cargo.toml: {e}"))?; @@ -516,7 +516,7 @@ mod tests { fn every_program_the_images_ship_declares_what_it_is_to_a_host() { let root = repo_root(); let programs = programs(&root).expect("every declaration reads"); - let shipped = crate::build::shipped(&root).expect("the modes' configs").programs; + let shipped = crate::build::shipped(&root).programs; let dirs: BTreeSet = programs.iter().map(|p| root.join(&p.dir)).collect(); assert_eq!(dirs, shipped); assert!(programs.iter().any(|p| p.host == Host::App(Vec::new())), "no app: {programs:?}"); diff --git a/system.toml b/system.toml index 9a6e902ac4a..b5430d2c924 100644 --- a/system.toml +++ b/system.toml @@ -4,8 +4,6 @@ # it in every boot is a listening port nobody asked for. # `src/build.rs`'s `no_shipped_boot_config_starts_sshserver` is the gate. -hosted-rustc = false - # `assets/soundfont.sf2` is in here and is doom's music: GeneralUser GS cut down # to the 37 melodic programs and 23 percussion keys `assets/DOOM1.WAD`'s own MUS # headers select, by `src/soundfont.rs`. `NOTICE` carries its licence, the diff --git a/tests/common/compile.rs b/tests/common/compile.rs index d18e0d5273a..b573d22212b 100644 --- a/tests/common/compile.rs +++ b/tests/common/compile.rs @@ -23,7 +23,7 @@ pub fn c_sysroot() -> CSysroot { static SYSROOT: OnceLock = OnceLock::new(); let sysroot = SYSROOT.get_or_init(|| { let mut lock = toyos_build::buildlock::shared(&repo_root(), "the C sysroot"); - toyos_build::toolchain::ensure(&repo_root(), &mut lock, false) + toyos_build::toolchain::ensure(&repo_root(), &mut lock) }); CSysroot::of(sysroot.dir(), super::qemu::SUITE_ARCH) }