# Testing wolfTrust separates host behavior, architecture-accurate emulation, and physical-hardware evidence. A result from one environment must not be reported as a result from another. ## Validation layers | Environment | What it validates | What it does not validate | | --- | --- | --- | | Native host | State machines, manifests, IPC ownership, copied transfers, services, storage, crypto integration, recovery decisions, and negative inputs | Cortex-M exception return, CMSE, SAU, MPU, GTZC, or physical flash behavior | | M33MU | Cortex-M33 instruction flow, TrustZone transitions, CMSE gateway calls, MPU faults, guest scheduling, authenticated boot, and target service interactions | STM32H563 peripherals, real option bytes, WRP, ST-Link, or silicon timing | | STM32H563 | The actual NUCLEO-H563ZI boot chain, Secure/Non-secure attribution, faults, flash, UART, and selected end-to-end behavior | Other devices, other provisioning states, peer-flash confidentiality, or adversarial peripheral and Non-secure NVIC ownership | ## Host tests Run the complete native set: ```sh make test ``` It ends with `make test-provisioning`: the production lock gates of both provisioning backends, run against stub vendor tools (see [Provisioning](Provisioning.md#testing)). The suite list is generated by: ```sh make -s -C tests/host print-suites ``` Current suites cover domain and manifest validation, lifecycle, guest verification, rollback decisions, IPC and FF-M behavior, SPM policy, gateway vectors, Secure Partition layout and recovery, crypto-engine relay and key isolation, vault and storage services, attestation and COSE integration, firmware update, runtime remeasurement, linked Secure layout, VNET, public PSA headers, boot-handoff record consumption, and negative paths. The manifest_ports suite generates every `port/*/manifest*.json` with that port's build arguments, for the default profile (update partition left out) and the `full` profile, and runs each through the boot-time validator, the port binding, and FF-M activation, since default images validate the manifest only at build time. `tests/host/manifest/test_generator.py` checks that each manifest the boot-time checks reject also fails generation. The monitor suite checks bounded guest recovery, idle backoff, and terminal lifecycle decisions against the production scheduler with simulated architecture operations. The attestation IAK suite runs wolfHSM NVM with both the default 8-byte and STM32H5 16-byte flash programming units. Additional host checks: ```sh make test-compilers make test-sanitize make test-valgrind ``` Valgrind requires the tool on the host. `make test-compilers` reruns all suites with the selected `CC` (default `cc`); it does not select multiple compilers. Run it once per compiler, for example `make test-compilers CC=clang`. Sanitizer support depends on the local toolchain. ## LTO validation Every Secure link generates `wolftrust.map` and runs the linked-image layout check. Compare optimized and diagnostic builds with separate output trees: ```sh make BUILD_DIR=build-lto size-report make BUILD_DIR=build-no-lto WT_LTO=0 size-report ``` The default M33MU and hardware commands exercise the LTO image. A release-size result is valid only when the applicable target scenarios pass without a fault marker. The cross-compilation workflow rebuilds one output tree from explicit `WT_LTO=0` to the default `WT_LTO=1`, verifies the build stamp and object formats change, checks that isolation-critical objects remain non-LTO, and requires the optimized ELF to be smaller. It also injects a forbidden heap symbol twice to prove a rejected ELF, flat binary, map, and CMSE import library are deleted instead of being reused by the next Make invocation. The per-PR core/port split workflow builds the `CONFIG_VNET=y` Secure image. The M33MU smoke tier builds and runs the `WT_CONFORMANCE=1` `confboot` layout on every PR; the VNET layout runs with the full matrix (`ci:h5` label, push to main, nightly), so both optional isolation-band configurations stay under the linked-image check in CI. ## PSA FF conformance ```sh make test-conformance ``` This target fetches the pinned Arm PSA architecture tests. When M33MU is available it runs the target FF-M IPC suite. Without M33MU it runs only the host-side client and policy subset and prints explicit warnings. Do not treat that fallback as target conformance. The dedicated guest configurations also exercise the PSA Crypto, Storage, and Initial Attestation validation applications. Those are target scenarios, not part of a plain `make test` result. ## M33MU scenarios The baseline target command is: ```sh make test-target ``` The engine defaults to `native`. Set `WT_ENGINE` to exercise the same target path with either backend: ```sh WT_ENGINE=native make test-target WT_ENGINE=hsm make test-target ``` It runs the port's smoke tier from its `port.json` (listed in [Ports](Ports.md)); `WT_TIER=full` runs every scenario. Detection accepts `m33mu` on `PATH` or a path in `M33MU`; a port whose emulator carries a model patch (`M33MU_PATCH` in its profile) builds its pinned emulator instead. If the emulator is unavailable, the target reports a skip rather than a pass. Each tier runs on both build profiles, the lean default first, then `full`, with their logs under `logs/default` and `logs/full`; `WT_PROFILE=default` or `WT_PROFILE=full` runs one. The default profile runs the image an unqualified `make` builds: its one-guest `positive`, the level 3 scenarios, the native wire key ops, and the SP fault, panic, attestation, launch, rollback, re-measurement, manifest, SP budget and peripheral negatives, plus the silicon-revision negative on the STM32H563 (see [Build profiles](Macros.md#build-profiles)). Each port's `port.json` lists its lean set under `m33mu.default`; the two-guest, VNET, conformance and update scenarios run only on `full`. Additional focused runs use: ```sh WT_ENGINE=native tests/target/run_target.sh stm32h563 positive WT_ENGINE=hsm tests/target/run_target.sh stm32h563 positive ``` One runner serves every port: `tests/target/run_target.sh ` builds the chain the port's profile (`tests/target/ports//m33mu.sh`) describes and hands the boot log to the judge in `tests/target/lib/scenarios.py`, which holds every scenario's assertions as data and decides which apply to the port from its guest kinds and the capabilities `port//port.json` declares absent (each absence names its reason and is reported as N/A). `tests/scripts/test_scenario_asserts.py` mutates every check in `make test` and requires the judge to refuse it. `scenarios.py --list` is the authoritative scenario list. It includes authenticated-boot failure, rollback, runtime remeasurement, Secure Partition recovery, key and vault isolation, storage recovery, attestation negatives, firmware update, manifest rejection, GTZC behavior, and VNET paths. `restartidle` runs the recovery check on the one-guest default image. It requires every allowed relaunch and the final terminal verdict (BKPT 0x7d, no guest left). The full tier runs it with both crypto engines on every port. It also runs `tests/target/test_systick_idle.sh`, which executes the production Armv8-M timer code with modeled Non-secure timer registers and real Secure SysTick interrupts. The check covers timer configuration and pending-tick preservation across idle, repeated idle, resume, and interrupt ordering. It also covers the guest trap-frame filter (a Secure tick never reads through a guest stack pointer outside that guest's RAM), MSP_NS capture for a PSP_NS frame, the Secure priority band, per-guest Non-secure mask save and restore, and the restore dropping a Secure FF-M entry mask a restarted guest left. M33MU has no `AIRCR.PRIS`, so the claim that Non-secure masks never stop the Secure tick prints SKIP there and is proven by the STM32H563 `nsmaskneg` hardware scenario. M33MU also drops `FAULTMASK_NS` writes, so the fixture's `FAULTMASK_NS` checks print SKIP there. The fixture checks which abandoned guest IRQs are queued for retirement; the retiring exception return itself runs only in the MIMXRT700 `handlerirqneg` hardware scenario. Five scenarios cover processor-state isolation, and the emulator proves less than their names suggest: - `fpneg` proves containment only. A floating-point instruction in the SERVICE_HSM partition takes the NOCP UsageFault and does not escalate. M33MU ends the run when it raises NOCP, so partition restart and guest survival are not shown here; the STM32H563 `fpneg` run checks them. - `sealneg` and `sealhaltneg` prove only the SPM's software check. In `sealneg` a partition overwrites its own stack-top seal, that partition alone faults at its next resume, and the guests keep running. In `sealhaltneg` a partition's seal is overwritten before its first dispatch (the same check runs on every dispatch) and the platform halts before that partition runs. No architectural unstack fault is exercised, on the emulator or by the STM32H563 `sealneg` run; M33MU models neither the seal nor the function return integrity check. - `sealbootneg` damages one main-stack seal word in the reset path; the boot halts on the production panic before any partition or guest runs. - `sealpivotneg` issues a partition's blocking wait with the stack pointer parked on its stack top, so the exception frame lands on the seal words; that partition alone faults and restarts, and the guests keep running. `bandneg1` through `bandneg6` prove the vault, attestation, and crypto partitions cannot reach each other's data band: | Scenario | Prober | Band touched | | --- | --- | --- | | `bandneg1` | crypto | vault | | `bandneg2` | crypto | attestation | | `bandneg3` | attestation | vault | | `bandneg4` | attestation | crypto | | `bandneg5` | vault | attestation | | `bandneg6` | vault | crypto | The prober reads the band, is restarted, writes the band, and is restarted again. Both accesses must fault on the prober's own stack, and the full positive lifecycle must still complete. The crypto and attestation probers also confirm the keystore services they do not own are refused. `restartneg1`, `restartneg2`, and `restartneg3` prove a restarted crypto, attestation, or vault partition starts from its band's link-time image. The partition changes initialized and zero-initialized state in its own band and faults; its restarted instance faults again if either value survived. `manifestneg` removes a required feature, `manifestneg2` declares isolation level 2, and `manifestneg3` composes a partition table that reaches another partition's band. Each must halt the boot before anything is scheduled. The first two corrupt the manifest after the build validated it, so they build with `WT_MANIFEST_RUNTIME_CHECK=1`; `manifestneg3` runs on the default image. Each numbered probe variant is its own matrix row; CI packs each family into one job. Three more cover the SPM's own fault handling: - `mspovfneg` pushes on the Secure main stack in the reset path until `MSPLIM_S` raises STKOF. M33MU escalates the entry-time STKOF to HardFault and ends the run there without executing the handler, so the emulator proves only the limit; the STM32H563 run reads the SPM fault latch and checks that no guest ran. - `xnneg` makes the privileged SVC gate call a thunk copied into SPM `.bss` while a partition thread domain is installed; the execute-never cover faults the fetch. M33MU pends that synchronous fault instead of escalating it past the active SVC, so the emulator proves only the denied fetch; the STM32H563 run checks the SPM fault latch and the halt. - `svcneg` has the ITS partition issue the scheduler's internal guest-return SVC; the partition alone is panicked and restarted, and the lifecycle completes. `busfaultneg` (the SERVICE_HSM partition reads an MPU-permitted window past the end of physical SRAM) and `nsbusfaultneg` (guest0 turns off its own MPU and reads an unmapped Non-secure peripheral hole; the monitor restarts it to its limit while guest1 runs) run only on the STM32H563: M33MU turns an unmapped data access into a MemManage fault and never vectors a data BusFault, so these scenarios have no emulator row until the pinned emulator models it. VNET has convenience targets: ```sh make test-vnet make test-vnet-target ``` `test-vnet` is host-only. `test-vnet-target` launches two authenticated wolfIP guests under M33MU. The wolfIP guests poll once per guest millisecond and idle between ticks so empty copied IPC calls leave time for the guest clock to advance. The `vnet` and `vnetneg` M33MU scenarios allow 180 wall-clock seconds for the initial ARP delay in guest time and the copied IPC round trip on shared runners. Both require the mediated ping reply and clean breakpoint exit; `vnetneg` also requires both isolation faults and recovery. ### Engine matrix The full CI scenario list adds `engine: [native, hsm]` as a matrix dimension. Every scenario runs with both engines except the engine-specific ones listed in `ENGINE_ONLY` in `tests/target/lib/scenario_matrix.py`: the wolfHSM-only scenarios run under hsm only, and `nativewire` and `nativewireasm`, which drive the native wire, under native only. The generator is the source of truth for the exact list: `python3 tests/target/lib/scenario_matrix.py --port stm32h563 --tier full --json`. `hsmattackneg` drives the raw wolfHSM protocol from a compromised-guest probe. It checks that a forged wolfHSM client ID cannot select the attestation key and that a wolfHSM NVM-group packet cannot reach the rollback store. The native engine does not link the wolfHSM client wire, server, or message handlers, so that exact attack surface does not exist there. Native key and namespace behavior remains covered by the common positive, cross-domain, keystore, storage, attestation, and Crypto-validation rows. Validation of the engine split completed under both engines with: - the applicable M33MU scenario matrix; - the Arm FF-M IPC suite at 85 passed, 4 heap-dependent tests skipped, and 0 failed, test for test as recorded in [`tests/target/ffm_ipc_results.txt`](../tests/target/ffm_ipc_results.txt); - the current dev_apis Crypto schedule at 64 passed, 13 skipped, and 0 failed (77 scheduled tests; c047 is configuration-skipped in addition to the upstream schedule); and - the STM32H563 positive, restart, cross-domain, and conformance hardware suite. The engine dimension changes crypto dispatch, not what M33MU proves. Emulator results still do not establish STM32 attribution or physical flash behavior. ### Scenario catalogue Every port runs the same scenario names; its `port.json` lists the ones it runs and exempts the rest with a reason. Beyond the families above: - `positive` boots the whole chain: wolfBoot verifies the Secure image, and both guests launch, reach the SPM through the SG veneers, and finish. - `guestisoneg` (the former `gtzcneg`, `ahbscneg` and `sauneg`, which still select it): guest 0 stores into guest 1's RAM window, the port's isolation refuses the store, the monitor relaunches guest 0 through its restart budget and quarantines it, and guest 1 keeps running. - `restart` makes guest 0 read Secure RAM on every launch until it is quarantined. `authneg` flips one byte of guest 0's image after its digest was pinned, so launch verification refuses guest 0 while guest 1 runs. `handlerneg` parks guest 0 in a Non-secure handler that never returns; the Secure tick still preempts it and the monitor restarts it. - `rollbackneg`, `remeasureneg`, `manifestneg` and `spbudgetneg` end on the verdict breakpoint their probe emits: a downgraded boot is refused, an image tampered after launch verification is caught by the re-measure before dispatch, a corrupted manifest halts boot, and restart-budget exhaustion escalates to the fail-closed platform recovery. - `crossdomain` and `keystoreneg`: an unprivileged storage partition's read of SPM-private RAM, or of the shared keystore band, faults at the port's band address and the guests ride it out. `spfaultneg` and `panicneg`: a partition that faults, or that the SPM must panic, is restarted in place once and serves the guests that follow. - `periphneg` has guest 0 switch off its own MPU and read the SPM's entropy peripheral through the Non-secure alias; a port with a DMA probe also runs Non-secure DMA copies out of Secure memory. Nothing may be read or changed. - The PSA test guest (`tests/firmware/psa-guest/`) runs `bothpsa` (the client lifecycle from both guests in one boot: SHA-256 KAT, ITS and PS, key-ops with a cross-key refusal, RNG, AES-CTR, and an attestation token verified against the IAK and wolfBoot's measurement), `bothiso` (a forged handle, an oversized vector, a vector in the peer window and an unknown SID refused), `attestneg`, and `fwustage` (a candidate staged through SERVICE_FWU, armed, and restored READY). - The remaining level 3 negatives (`deputyneg`, `hsmpinneg`, `hsmfaultneg`, `periphspneg`) run the PSA guest in both windows, and guest 0 ends the run on its breakpoint once its lifecycle is done; `manifestneg2`, `manifestneg3` and `sealhaltneg` end on their verdict breakpoint, and `mspovfneg` and `xnneg` at the SPM fault. - `confboot` hosts Arm's unmodified FF-M IPC suite against the conformance image and expects 85 passed, 4 skipped, 0 failed. `devstorage`, `devcrypto`, `devattest` and `devattestqcbor` run the dev_apis suites. `vaultrecover` proves a foreign vault pool self-heals on a development device, and `vaultrecoversec` that SECURED refuses it gracefully. - `wrpfence` and `wrpoff` build with `WT_GUEST_FLASH_WRP=1`: behind a first stage that arms the guest flash fence both guests launch, and behind one that does not, neither may enter its domain. A console boot ends on the emulator's wall-clock budget once the guests idle (exit status 127); a verdict or traced boot stops on its breakpoint with status 0; any other status fails the run. A port whose emulator or first stage needs a carried patch builds its own pinned copy, and its guide lists what each patch models. Emulator results are recorded separately from the board's. ## Hardware One runner serves every board: `tests/target/run_hardware.sh ` builds the chain the emulator boots, programs it through the port's profile (`tests/target/ports//hw.sh`), boots the board once, captures its console, reads the Secure image's latches and the guests' mailboxes over SWD, and hands that evidence to the judge in `tests/target/lib/scenarios.py` with `--env hardware`. The hardware assertions are the same declared blocks the emulator uses, resolved for the port's guest kinds; a block that needs a capability the port's registry declares absent is reported N/A with the reason, never skipped silently. Each run keeps its evidence under `logs/-hw/-/` (`console.log`, `swd.txt`, `readback-*.bin`, `program.log`, `result.json`), outside `build/`, and one run at a time owns the board through `logs/-hw.lock`. ```sh make test-hardware TARGET= # the eight every board runs make test-hardware TARGET= WT_HW_TIER=qualification tests/target/run_hardware.sh tests/target/run_hardware.sh build # toolchain host tests/target/run_hardware.sh flash # probe host, same checkout ``` The build half copies the first stage and the guest images under `build/hw/` and records them in `build/hw-stamp.sh` relative to the checkout, so the flash half needs nothing outside it. The eight scenarios every board runs on silicon are the contract a port meets (`scenarios.py --hardware-minimum`): `positive`, `guestisoneg`, `periphneg`, `restart`, `psa`, `wrpoff`, `crossdomain` and `authneg`. The qualification tier adds the port's merge-gating extras from `port//port.json` (`hardware.qualification`), and `hardware.optin` names the board-only scenarios a caller selects by name. The suite skips, never passes, without the board. Every profile keeps three rules: the probe is bound by serial or unique id for every tool call (`WT_HW_PROBE` picks one of several), the preflight only reads, and no option byte, fuse or life-cycle state is written by the runner (`tests/target/provisioning/provisioning_ctrl.sh` owns those, gated). `tests/target/test_hw_runner.sh` checks the hook contract, those rules, the evidence paths and the board lock for every port under `make test`. ### Bare-metal guest mailbox The bare-metal guest records its progress in a mailbox at the base of its own Non-secure RAM window and echoes it on the console; the runner reads it over SWD. Each guide gives the port's addresses and console. | Mailbox word | Offset | Pass value | | --- | ---: | ---: | | signature | `+0x00` | `0x47543030` | | step | `+0x04` | `0x00000005` | | `psa_framework_version()` | `+0x08` | `0x00000100` | | `psa_connect(SERVICE_HSM)` handle | `+0x10` | > 0, distinct per guest | | status | `+0x14` | `0x600D600D` | | console `VERID` as the guest reads it | `+0x18` | non-zero | | isolation probe latch (`guestisoneg`) | `+0x1C` | `1` faulted or `2` blocked; `3` leaked | | isolation probe read-back (`guestisoneg`) | `+0x20` | not the stored sentinel | `0x600D600D` in both mailboxes proves the chain booted and the Secure runtime served both Non-secure clients through the veneers; `0xBAD00000` records a failed check at the `step` reached. Hardware test commands write flash and reset the board. Review the board guide before running them. ## CI coverage The workflows under `.github/workflows/` separately run: - host unit tests; - compiler variants, sanitizers, and Valgrind; - Cortex-M33 cross-compilation of both crypto engines; - dependency integration; - the core/port split guard and the docs guard (no internal-ledger or home-directory references in the published docs); - fuzz targets; - selected and nightly M33MU scenarios for every registered port, each under its own pinned emulator image ([Ports](Ports.md)). ### Trigger routing The host, compiler, sanitizer, Valgrind, cross-compilation, integration, and core/port split checks run on every pull request, including drafts. The fuzz target also runs on pull requests as a 60-second libFuzzer smoke pass; the nightly schedule and manual dispatch run the 600-second soak instead. The `M33MU` workflow (wolfBoot plus both guest lifecycles, then a `select` job that picks each port's tier and one matrix job that runs exactly those scenario groups) is tiered. Every pull request runs the smoke tier on both crypto engines for each port whose CI label is optional; a port marked label only in [Ports](Ports.md) runs on a pull request only with its own label, `ci:all`, or `ci:m33mu`. The full matrix runs on a push to `main`, on the nightly schedule, on manual dispatch (with a `port` input), and on a pull request that carries the port's label, `ci:all`, or `ci:m33mu`. The scenario groups per port and tier are in its `port.json`; `make test-target TARGET=` runs the same smoke tier locally and `WT_TIER=full` every scenario. Every tier runs on both build profiles, and the level 3 bar (`l3_gaps()` in the matrix selftest) must hold on `default` and on `full`. ### Running M33MU off a pull request To run a port's full matrix against a branch without a PR, dispatch the workflow: ```sh gh workflow run m33mu.yml --ref -f port=mimxrt700 ``` To run a single scenario locally, use `tests/target/run_target.sh ` (for example `positive` or `crossdomain`). The GitHub-hosted workflows do not establish a physical-board result. Hardware output must come from the hardware runner attached to a board. ## Interpreting failures - A host pass plus target failure usually points to architecture glue, image assembly, linker placement, or hardware policy rather than a neutral state machine. - A missing target prints `SKIP`. It is not a successful run. - Preserve the first failing marker and the generated log before rebuilding; target runners place detailed output under `logs/` or the configured scenario log. - Confirm that Secure, guest, manifest, flash, and emulator addresses were built from the same configuration.