Repository navigation
The local APIC runs in xAPIC mode where CPUID offers no x2APIC - #841
Conversation
The kernel set IA32_APIC_BASE.EXTD unconditionally, which is #GP on a CPU whose CPUID.01H:ECX[21] is clear; on the BSP that fault lands before the IDT loads, so such a machine triple-faults with nothing on any channel. An AMD laptop without x2APIC is one. IA32_APIC_BASE joins control_regs' declaration: written whole, the registers at FEE00000H (a firmware that moved them is refused), x2APIC where CPUID offers it and xAPIC where it does not, decided by the BSP and asserted on every AP. apic.rs reaches every register through one Reg type keyed by its xAPIC MMIO offset, whose x2APIC MSR is 0x800 + offset/16; the xAPIC page is mapped uncacheable by the BSP. The ICR is the one register the modes lay out differently: x2APIC writes it once behind the fence its non-serializing WRMSR needs, xAPIC writes the 8-bit destination half and then the command half with interrupts closed. An APIC id an 8-bit destination cannot carry (or 0xFF, the broadcast) is refused by name in the ICR, the compatibility-format MSI and the I/O APIC entry; msi_message used to shift any id into the address unchecked. The two asm stubs that wrote x2APIC MSRs directly (the timer's re-arm and EOI, the TLB IPI's EOI) now call into apic.rs. VT-d's EIME follows the local APICs' mode as well as ECAP.EIM: an xAPIC compares the 8-bit destination field. Checked against the Intel SDM Vol. 3A (325384-093US) chapter 13 and the AMD APM Vol. 2 (24593 rev. 3.45) chapter 16, cited at the register sites. The new guest test desktop_without_x2apic boots tests/panelcase on Profile::MetalNoX2apic (TCG, the one CPU declaration with -x2apic) to the compositor holding the panel on four CPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
A kernel that declared xAPIC on a CPU that offers x2APIC would still boot every guest of the suite, since each runs either mode; the T14, whose CPUs offer x2APIC, is where that choice is visible. The row's judge and desktop_without_x2apic now read the same helper, apic_mode_held: every CPU's apic_base line, the machine's report and the BSP's LAPIC line name the mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
Measurement before building: the base image, -x2apic, serial — part 1/1, whole, from |
|
Measurement before building: the base image, -x2apic, QEMU -d int,cpu_reset — part 1/5, whole, from |
|
Measurement before building: the base image, -x2apic, QEMU -d int,cpu_reset — part 2/5, whole, from |
|
Measurement before building: the base image, -x2apic, QEMU -d int,cpu_reset — part 3/5, whole, from |
|
Measurement before building: the base image, -x2apic, QEMU -d int,cpu_reset — part 4/5, whole, from |
|
Measurement before building: the base image, -x2apic, QEMU -d int,cpu_reset — part 5/5, whole, from |
|
Measurement before building: the base image, +x2apic, serial — part 1/1, whole, from |
|
Negative control: the patch reverted onto the base (git apply -R) — part 1/1, whole, from |
|
Negative control: the script — part 1/1, whole, from |
|
Negative control: desktop_without_x2apic on the base's kernel/, EXIT=1 — part 1/1, whole, from |
|
Guest suite at 6c8192e, EXIT=0 — part 1/1, whole, from |
|
T14 request staged at 6c8192e (log, then request.txt), EXIT=2 — part 1/1, whole, from |
|
Host gate at 6c8192e: cargo run -- --ci host, EXIT=0 — part 1/9, whole, from |
|
Host gate at 6c8192e: cargo run -- --ci host, EXIT=0 — part 2/9, whole, from |
|
Host gate at 6c8192e: cargo run -- --ci host, EXIT=0 — part 3/9, whole, from |
|
Host gate at 6c8192e: cargo run -- --ci host, EXIT=0 — part 4/9, whole, from |
|
Host gate at 6c8192e: cargo run -- --ci host, EXIT=0 — part 5/9, whole, from |
|
Host gate at 6c8192e: cargo run -- --ci host, EXIT=0 — part 6/9, whole, from |
|
Host gate at 6c8192e: cargo run -- --ci host, EXIT=0 — part 7/9, whole, from |
|
Host gate at 6c8192e: cargo run -- --ci host, EXIT=0 — part 8/9, whole, from |
|
Host gate at 6c8192e: cargo run -- --ci host, EXIT=0 — part 9/9, whole, from |
|
Review of #841 at 6c8192e, round 1. Net lines ( The two open risks
BLOCKER
NOTE
SEND BACK |
…ion is apic's `apic::init` ran before `percpu::init_bsp` loaded the IDT, and under xAPIC it maps the register page: a heap push, perhaps a page-table page, a TLB flush and a log line, any of which faulting triple-faults with nothing on any channel — the failure this branch exists to remove. The BSP's per-CPU block now takes its id from `cpu::hardware_id()`, as every AP's does, and the APIC is declared after it. VT-d's remapping entry carried its own copy of the eight-bit bound (`NARROW_DESTINATIONS = 0xFF`). Its refusal and its `MSG_DEST` const assert now ask `apic::narrow_destination`, now a `const fn`. The x2APIC opt-out issue is made true of the tree: the mode follows CPUID in `control_regs::init_apic`, and the untested `x2apic &&` gate on `EIME` in `vtd::remappable` is recorded there, its exit reaching it on the T14. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
build-x86.log at e1d7370 — part 1/1, whole log split on line boundaries |
|
build-aarch64.log at e1d7370 — part 1/1, whole log split on line boundaries |
|
ci-host.log at e1d7370 — part 1/9, whole log split on line boundaries |
|
ci-host.log at e1d7370 — part 2/9, whole log split on line boundaries |
|
host 2/9, at head 9cc0637 (merge of origin/main 8b2969d): |
|
host 3/9, at head 9cc0637 (merge of origin/main 8b2969d): |
|
host 4/9, at head 9cc0637 (merge of origin/main 8b2969d): |
|
host 5/9, at head 9cc0637 (merge of origin/main 8b2969d): |
|
host 6/9, at head 9cc0637 (merge of origin/main 8b2969d): |
|
host 7/9, at head 9cc0637 (merge of origin/main 8b2969d): |
|
host 8/9, at head 9cc0637 (merge of origin/main 8b2969d): |
|
host 9/9, at head 9cc0637 (merge of origin/main 8b2969d): |
|
suite 1/1, at head 9cc0637 (merge of origin/main 8b2969d): |
The one conflict, in `boot::interrupts`, is textual: main keeps the FACS `power::init_reset` returns for `acpi_mode::init`, and this branch brings `percpu::init_bsp` ahead of `apic::init` on the CPU's hardware id. Both are kept; the order of calls is either side's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
build-x86 1/1 — at head 7c3200f (merge of origin/main 1924919) |
|
build-aarch64 1/1 — at head 7c3200f (merge of origin/main 1924919) |
|
host 1/9 — at head 7c3200f (merge of origin/main 1924919) |
|
host 2/9 — at head 7c3200f (merge of origin/main 1924919) |
|
host 3/9 — at head 7c3200f (merge of origin/main 1924919) |
|
host 4/9 — at head 7c3200f (merge of origin/main 1924919) |
|
host 5/9 — at head 7c3200f (merge of origin/main 1924919) |
|
host 6/9 — at head 7c3200f (merge of origin/main 1924919) |
|
host 7/9 — at head 7c3200f (merge of origin/main 1924919) |
|
host 8/9 — at head 7c3200f (merge of origin/main 1924919) |
|
host 9/9 — at head 7c3200f (merge of origin/main 1924919) |
|
suite 1/1 — at head 7c3200f (merge of origin/main 1924919) |
, #851 and #853, into consent tests/common/qemu.rs: Profile carries both Desktop and MetalNoX2apic (#841). userland/compositor/src/session.rs: the crate import takes main's list, without CURSOR_PX (#814), plus the branch's PROMPT_POLL_HANDLES. Cargo.lock is main's, re-resolved: grants, and filepicker's toyos-manifest and the build's toyos-desktop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
The kernel set
IA32_APIC_BASE.EXTDon every CPU whatever CPUID said. On a CPU whoseCPUID.01H:ECX[21]is clear that write is#GP, and on the BSP it comes before the IDT loads, so the machine triple-faults with nothing on any channel. An AMD laptop without x2APIC is such a machine. This branch drives the local APIC in xAPIC (MMIO) mode where CPUID offers no x2APIC, and keeps x2APIC wherever it exists.What changed, per decision
IA32_APIC_BASEjoinscontrol_regs.init_apicwrites it whole: the registers atFEE00000H,EN,EXTDexactly whereCPUID.01H:ECX[21]is set, and the CPU's ownBSPbit. It reads the register back and asserts it. The BSP publishes the mode, and every AP that computes a different one panics by name. A firmware that moved the registers is refused rather than followed. The read-modify-write inapic::enable_x2apicis gone.control_regs: N of N cpus hold …now ends inapic=<mode>, and each CPU logscontrol_regs: cpuN apic_base=… <mode>.apic::Regis the xAPIC MMIO offset, and its x2APIC MSR is0x800 + offset/16(SDM §13.12.1.2, APM §16.11.1). Itsreadandwritebranch on the declared mode. The BSP maps the xAPIC page uncacheable withmap_mmio; APs reach it through the shared kernel tables. The ID is all 32 bits in x2APIC and bits 31:24 in xAPIC.map_mmiopushes onto the heap, can allocate a page-table page, flushes the TLB and logs, and any fault there beforelidttriple-faults with nothing on any channel, which is the failure this branch removes. Soarch::boot::interruptsnow runspercpu::init_bsp(cpu::hardware_id())and thenapic::init(). The BSP's per-CPU block takes its id the way every AP's echo does.smp::boot_apsstill asserts that the BSP'sapic::id()equalscpu::hardware_id(). Nothing ininit_bspreaches an APIC register, and interrupts are still closed. A register reached before the declaration panics by name (control_regs::apic_mode).WRMSRbehind theMFENCE; LFENCEits non-serializing write needs. xAPIC: the destination half, then the command half that sends, with interrupts closed between the two writes. That is Linux'sdefault_send_IPI_single_phys, without its Delivery Status poll: APM §16.5 says software may writeICRLagain without polling DS, and SDM §13.6.1 asks for no wait. xAPIC MMIO is serializing on a UC page (SDM §13.12.3's note), so it needs no fence.apic::narrow_destinationrefuses an id above0xFE, because0xFFis the broadcast. It is aconst fnand the tree's one statement of the 8-bit bound. It is used by the xAPIC ICR (panic naming the id), by the I/O APIC's direct entry (RouteError::DestTooWide, as before), and by VT-d's remapping entry withoutEIME(Refused::DestinationTooWide, as before). The compatibility-format MSI (msi_message, which since A claimed function's message is its claim slot's LPI through the GICv3 ITS, no generic signature carries an x86 vector, and the kernel copies and reports through no user leaf of a device's registers #846 takes only a vector and targetsMSG_DEST) shiftsMSG_DEST_NARROW, a const that isnarrow_destination(MSG_DEST)and fails the build where it refuses. VT-d's own copy,NARROW_DESTINATIONS = 0xFF, is gone, and so is its const assert onMSG_DEST, whichMSG_DEST_NARROWnow states once.EIMEholds only whatnarrow_destinationproduced.vtd::interrupt::entrytakes aDestination:Extended(u32)underEIME,Narrow(u8)without it.allocatepassesnarrow_destination'sOkvalue, andclaimpassesMSG_DEST_NARROW, nowpub(crate). Before this,allocatechecked the bound and then shifted the rawu32, andclaimchecked nothing.MSG_DESTtoapic.rs, and the auto-merge brought back the I/O APIC's>= 0xFFand an unboundedMSG_DEST << 12inmsi_message. The resolution keeps A claimed function's message is its claim slot's LPI through the GICv3 ITS, no generic signature carries an x86 vector, and the kernel copies and reports through no user leaf of a device's registers #846's structure (vtd::interrupt::is_armedandpininioapic::route,vtd::interrupt::msimaking the compatibility message) and folds both ontonarrow_destination.pio.rspassesMSG_DESTtoioapic::route, which narrows it.apic::rearm_last,apic::ring0_fireandapic::eoi. The timer's Ring 0 branch now pushes the full caller-saved set before its first call instead of after it. It still pushes ten registers, so[rbp + 80]is still the interruptedrip.EIMEfollows the local APICs' mode. It is set only where every unit hasECAP.EIMand the APICs are x2APIC. An xAPIC compares the 8-bit destination field (VT-d Section 9.9). No tier reaches thex2apic &&term, and deleting it passes every test. QEMU'sintel-iommutakeseim=ononly with KVM's split irqchip, and the harness'sIOMMU_DEFAULTleaveseimoff. The T14 runs x2APIC, and the AMD laptop has AMD-Vi, not VT-d. This is recorded in the newissues/vtd-keeps-eime-off-under-an-xapic-and-no-test-reaches-it.md, owned by the IOMMU track, whose exit is a T14 row in xAPIC on its VT-d unit that reds with the term deleted.issues/x2apic-is-enabled-without-reading-firmwares-opt-out.md, so it is asmainhas it. It still citesenable_x2apic, which this branch deletes, until whoever takes it up rewrites it.tests/common/qemu.rs, is two newProfilevariants side by side: main'sMetalAmdViand this branch'sMetalNoX2apic, each with its doc comment, itsArch::X86_64arm and itsshapearm. Nothing else changed.arch::boot::interrupts, is textual: main binds the FACSpower::init_resetnow returns (let facs = …, carried toacpi_mode::initthroughPlatform), and this branch movespercpu::init_bsp(cpu::hardware_id())ahead ofapic::init. Both are kept; the calls run in the same order either side has. Nothing else changed.apic.rs(§11.x/§12.x of earlier editions) were renumbered to 093US where the lines were touched.High-risk checks (devices, concurrency primitives)
-cpu qemu64,+rdrand,+smap,+fsgsbase,-x2apic,+smep: QEMU's-d intshowsv=0dat theWRMSRwithEAX=0xfee00d00(base | EN | EXTD), thenv=08, thenTriple fault. The serial ends atACPI: reset register …, the line beforeapic::init. The same image with+x2apicboots toSMP: 4 of 4. Logs: manual-off-serial, manual-off-qemulog 1/5, manual-off-qemulog 2/5, manual-off-qemulog 3/5, manual-off-qemulog 4/5, manual-off-qemulog 5/5, manual-on-serial.negative-control.patch(patch (script: script)) applied withgit apply -R,kernel/verified identical toorigin/main(60dc7fc), at head 6c8192e, the test run, the patch re-applied,git status --porcelain --ignore-submodules=noneempty. Result:desktop_without_x2apicred, EXIT=1:QEMU died before Boot: complete (. Log: neg.boot:testcases boot:latencycase, judged with--metal-readback: EXIT=0,251 passed, 0 failed, 3 boot(s), withcontrol_regs: cpu0 apic_base=0xfee00d00 X2apic.--metal-readbackgives EXIT=0,251 passed, 0 failed, 3 boot(s). That includesioapic_topology,claim_reuses_its_remapping_entry,irq_census_conservationandsmp_roster_and_tsc_trail(holdingX2apic).Destinationchange rewrites howallocateandclaimcompose an entry, whichclaim_reuses_its_remapping_entryandirq_census_conservationread on the T14's VT-d unit, so the same two boots were run again.testcaseswas restaged after the first flash was cut short.--metal-readbackgives EXIT=0,251 passed, 0 failed, 3 boot(s), withclaim_reuses_its_remapping_entry,irq_census_conservationandsmp_roster_and_tsc_trailamong them. Not rerun after the merges with origin/main 8b2969d and 1924919: the first changes this branch's side only intests/common/qemu.rs, a QEMU harness file no metal row reads; the second only bindsinit_reset's new return value on a line beside the APIC bring-up, whose calls and their order are this branch's as the reading ran them.apic_base=… Xapicline,LAPIC: Xapic enabled (ID 0),LAPIC timer: 998040 ticks/10ms,SMP: 16 of 16 MADT cpus online,16 of 16 cpus hold … apic=Xapic,Boot: complete (1950ms), and userland running to a power-button stop, with 78 of the compositor's two-second statistics lines from 12.148 s to 166.851 s. The log has notlb: shootdowns=line: the stop says the census after logkeeper's last write, so/logcannot carry it. The Delivery Status no-poll is therefore unread on that machine, and is recorded in the newtoolingissueissues/the-xapic-icr-send-polls-no-delivery-status-and-no-machine-has-read-it.md, owned by the orchestrator, with the laptop's shootdown line as its exit.The new guest test, and why QEMU
desktop_without_x2apic(aMACHINE_TESTSentry) bootstests/panelcase(logkeeper + compositor) on a newProfile::MetalNoX2apic. That is Metal's shape on four CPUs, under TCG whatever the host. Its CPU is the one declaration insrc/arch.rswith+x2apicreplaced by-x2apic, refused by name if that declaration ever stops offering it. The test waits forcompositor: readyand for the panel's fill to leave the boot fill. It then assertsmust_be_clean, every CPU'sapic_base … Xapicline,4 of 4 cpus hold … apic=Xapic,LAPIC: Xapic enabled (, andsmp_roster_and_tsc_trail(all four up, every AP's TSC bracketed).Why it needs QEMU: xAPIC mode exists only on a CPU without x2APIC. No type or host build drives a local APIC. The T14's CPUs offer x2APIC, so no metal row there reaches xAPIC, and the AMD laptop is not on the bench to rerun per change. TCG is forced. Under TCG a guest's CPUID is exactly the
-cpuline on any host, CI's KVM runner included. Whether KVM's split irqchip, which the q35 profiles' VT-d unit needs, accepts a CPU without x2APIC was not measured: this host has no KVM.smp_roster_and_tsc_trail's T14 row now also holds every CPU toX2apicthrough the same helper,apic_mode_held. A kernel that declared xAPIC on a CPU offering x2APIC would boot every QEMU guest of the suite, since each mode works there. Only the T14 sees that choice, and this row is the check for it.Gates, at head 7c3200f (the merge of origin/main 1924919; base 1924919)
cargo run -- --ci hostcargo run -- --build-onlycargo run -- --build-only --arch aarch64desktop_without_x2apicand main'samdvi_firmware_leftamong them)cargo test --test toyos-buildcargo test --test toyos-build -- --metal --metal-readback <dir> boot:testcases boot:latencycaseThe first four ran on the committed tree at 7c3200f: the host gate alongside the two image builds and the suite, which ran one after another. Round 1's negative control and logs, at 6c8192e, are linked above.
Net lines (
git diff --shortstat origin/main...HEAD): 16 files, +495 −170. Production (kernel/) +367 −164, tests (tests/) +81 −6,issues/+47 (two new files). The production growth is the xAPIC register path, theIA32_APIC_BASEdeclaration, theDestinationan entry takes, and their citations.What I am unsure of
cpu::hardware_id()reads CPUID leaf 0x1F/0xB first. On an xAPIC part that leaf may or may not exist. SDM §13.12.8.1 says its EDX[7:0] equals the initial APIC id whenever it does, and the roster asserts the two agree on every CPU, so a disagreement on the laptop panics by name rather than misrouting. The BSP's per-CPU id now comes from this leaf too, andsmp::boot_apsasserts it equals the APIC's own ID register.cargo runprofiles (src/qemu.rs) have no no-x2APIC option. Only the harness has one.🤖 Generated with Claude Code
https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C