Skip to content

Latest commit

 

History

History
481 lines (400 loc) · 23.8 KB

Testing.md

File metadata and controls

481 lines (400 loc) · 23.8 KB

Testing

wolfTrust separates host behavior, architecture-accurate emulation, and physical-hardware evidence. A result from one environment must not be reported as a result from another.

Validation layers

Environment What it validates What it does not validate
Native host State machines, manifests, IPC ownership, copied transfers, services, storage, crypto integration, recovery decisions, and negative inputs Cortex-M exception return, CMSE, SAU, MPU, GTZC, or physical flash behavior
M33MU Cortex-M33 instruction flow, TrustZone transitions, CMSE gateway calls, MPU faults, guest scheduling, authenticated boot, and target service interactions STM32H563 peripherals, real option bytes, WRP, ST-Link, or silicon timing
STM32H563 The actual NUCLEO-H563ZI boot chain, Secure/Non-secure attribution, faults, flash, UART, and selected end-to-end behavior Other devices, other provisioning states, peer-flash confidentiality, or adversarial peripheral and Non-secure NVIC ownership

Host tests

Run the complete native set:

make test

It ends with make test-provisioning: the production lock gates of both provisioning backends, run against stub vendor tools (see Provisioning).

The suite list is generated by:

make -s -C tests/host print-suites

Current suites cover domain and manifest validation, lifecycle, guest verification, rollback decisions, IPC and FF-M behavior, SPM policy, gateway vectors, Secure Partition layout and recovery, crypto-engine relay and key isolation, vault and storage services, attestation and COSE integration, firmware update, runtime remeasurement, linked Secure layout, VNET, public PSA headers, boot-handoff record consumption, and negative paths. The manifest_ports suite generates every port/*/manifest*.json with that port's build arguments, for the default profile (update partition left out) and the full profile, and runs each through the boot-time validator, the port binding, and FF-M activation, since default images validate the manifest only at build time. tests/host/manifest/test_generator.py checks that each manifest the boot-time checks reject also fails generation. The monitor suite checks bounded guest recovery, idle backoff, and terminal lifecycle decisions against the production scheduler with simulated architecture operations. The attestation IAK suite runs wolfHSM NVM with both the default 8-byte and STM32H5 16-byte flash programming units.

Additional host checks:

make test-compilers
make test-sanitize
make test-valgrind

Valgrind requires the tool on the host. make test-compilers reruns all suites with the selected CC (default cc); it does not select multiple compilers. Run it once per compiler, for example make test-compilers CC=clang. Sanitizer support depends on the local toolchain.

LTO validation

Every Secure link generates wolftrust.map and runs the linked-image layout check. Compare optimized and diagnostic builds with separate output trees:

make BUILD_DIR=build-lto size-report
make BUILD_DIR=build-no-lto WT_LTO=0 size-report

The default M33MU and hardware commands exercise the LTO image. A release-size result is valid only when the applicable target scenarios pass without a fault marker. The cross-compilation workflow rebuilds one output tree from explicit WT_LTO=0 to the default WT_LTO=1, verifies the build stamp and object formats change, checks that isolation-critical objects remain non-LTO, and requires the optimized ELF to be smaller. It also injects a forbidden heap symbol twice to prove a rejected ELF, flat binary, map, and CMSE import library are deleted instead of being reused by the next Make invocation.

The per-PR core/port split workflow builds the CONFIG_VNET=y Secure image. The M33MU smoke tier builds and runs the WT_CONFORMANCE=1 confboot layout on every PR; the VNET layout runs with the full matrix (ci:h5 label, push to main, nightly), so both optional isolation-band configurations stay under the linked-image check in CI.

PSA FF conformance

make test-conformance

This target fetches the pinned Arm PSA architecture tests. When M33MU is available it runs the target FF-M IPC suite. Without M33MU it runs only the host-side client and policy subset and prints explicit warnings. Do not treat that fallback as target conformance.

The dedicated guest configurations also exercise the PSA Crypto, Storage, and Initial Attestation validation applications. Those are target scenarios, not part of a plain make test result.

M33MU scenarios

The baseline target command is:

make test-target

The engine defaults to native. Set WT_ENGINE to exercise the same target path with either backend:

WT_ENGINE=native make test-target
WT_ENGINE=hsm make test-target

It runs the port's smoke tier from its port.json (listed in Ports); WT_TIER=full runs every scenario. Detection accepts m33mu on PATH or a path in M33MU; a port whose emulator carries a model patch (M33MU_PATCH in its profile) builds its pinned emulator instead. If the emulator is unavailable, the target reports a skip rather than a pass. Each tier runs on both build profiles, the lean default first, then full, with their logs under logs/default and logs/full; WT_PROFILE=default or WT_PROFILE=full runs one. The default profile runs the image an unqualified make builds: its one-guest positive, the level 3 scenarios, the native wire key ops, and the SP fault, panic, attestation, launch, rollback, re-measurement, manifest, SP budget and peripheral negatives, plus the silicon-revision negative on the STM32H563 (see Build profiles). Each port's port.json lists its lean set under m33mu.default; the two-guest, VNET, conformance and update scenarios run only on full.

Additional focused runs use:

WT_ENGINE=native tests/target/run_target.sh stm32h563 positive
WT_ENGINE=hsm tests/target/run_target.sh stm32h563 positive

One runner serves every port: tests/target/run_target.sh <port> <scenario> builds the chain the port's profile (tests/target/ports/<port>/m33mu.sh) describes and hands the boot log to the judge in tests/target/lib/scenarios.py, which holds every scenario's assertions as data and decides which apply to the port from its guest kinds and the capabilities port/<port>/port.json declares absent (each absence names its reason and is reported as N/A). tests/scripts/test_scenario_asserts.py mutates every check in make test and requires the judge to refuse it. scenarios.py --list is the authoritative scenario list. It includes authenticated-boot failure, rollback, runtime remeasurement, Secure Partition recovery, key and vault isolation, storage recovery, attestation negatives, firmware update, manifest rejection, GTZC behavior, and VNET paths.

restartidle runs the recovery check on the one-guest default image. It requires every allowed relaunch and the final terminal verdict (BKPT 0x7d, no guest left). The full tier runs it with both crypto engines on every port. It also runs tests/target/test_systick_idle.sh, which executes the production Armv8-M timer code with modeled Non-secure timer registers and real Secure SysTick interrupts. The check covers timer configuration and pending-tick preservation across idle, repeated idle, resume, and interrupt ordering. It also covers the guest trap-frame filter (a Secure tick never reads through a guest stack pointer outside that guest's RAM), MSP_NS capture for a PSP_NS frame, the Secure priority band, per-guest Non-secure mask save and restore, and the restore dropping a Secure FF-M entry mask a restarted guest left. M33MU has no AIRCR.PRIS, so the claim that Non-secure masks never stop the Secure tick prints SKIP there and is proven by the STM32H563 nsmaskneg hardware scenario. M33MU also drops FAULTMASK_NS writes, so the fixture's FAULTMASK_NS checks print SKIP there. The fixture checks which abandoned guest IRQs are queued for retirement; the retiring exception return itself runs only in the MIMXRT700 handlerirqneg hardware scenario.

Five scenarios cover processor-state isolation, and the emulator proves less than their names suggest:

  • fpneg proves containment only. A floating-point instruction in the SERVICE_HSM partition takes the NOCP UsageFault and does not escalate. M33MU ends the run when it raises NOCP, so partition restart and guest survival are not shown here; the STM32H563 fpneg run checks them.
  • sealneg and sealhaltneg prove only the SPM's software check. In sealneg a partition overwrites its own stack-top seal, that partition alone faults at its next resume, and the guests keep running. In sealhaltneg a partition's seal is overwritten before its first dispatch (the same check runs on every dispatch) and the platform halts before that partition runs. No architectural unstack fault is exercised, on the emulator or by the STM32H563 sealneg run; M33MU models neither the seal nor the function return integrity check.
  • sealbootneg damages one main-stack seal word in the reset path; the boot halts on the production panic before any partition or guest runs.
  • sealpivotneg issues a partition's blocking wait with the stack pointer parked on its stack top, so the exception frame lands on the seal words; that partition alone faults and restarts, and the guests keep running.

bandneg1 through bandneg6 prove the vault, attestation, and crypto partitions cannot reach each other's data band:

Scenario Prober Band touched
bandneg1 crypto vault
bandneg2 crypto attestation
bandneg3 attestation vault
bandneg4 attestation crypto
bandneg5 vault attestation
bandneg6 vault crypto

The prober reads the band, is restarted, writes the band, and is restarted again. Both accesses must fault on the prober's own stack, and the full positive lifecycle must still complete. The crypto and attestation probers also confirm the keystore services they do not own are refused.

restartneg1, restartneg2, and restartneg3 prove a restarted crypto, attestation, or vault partition starts from its band's link-time image. The partition changes initialized and zero-initialized state in its own band and faults; its restarted instance faults again if either value survived.

manifestneg removes a required feature, manifestneg2 declares isolation level 2, and manifestneg3 composes a partition table that reaches another partition's band. Each must halt the boot before anything is scheduled. The first two corrupt the manifest after the build validated it, so they build with WT_MANIFEST_RUNTIME_CHECK=1; manifestneg3 runs on the default image.

Each numbered probe variant is its own matrix row; CI packs each family into one job.

Three more cover the SPM's own fault handling:

  • mspovfneg pushes on the Secure main stack in the reset path until MSPLIM_S raises STKOF. M33MU escalates the entry-time STKOF to HardFault and ends the run there without executing the handler, so the emulator proves only the limit; the STM32H563 run reads the SPM fault latch and checks that no guest ran.
  • xnneg makes the privileged SVC gate call a thunk copied into SPM .bss while a partition thread domain is installed; the execute-never cover faults the fetch. M33MU pends that synchronous fault instead of escalating it past the active SVC, so the emulator proves only the denied fetch; the STM32H563 run checks the SPM fault latch and the halt.
  • svcneg has the ITS partition issue the scheduler's internal guest-return SVC; the partition alone is panicked and restarted, and the lifecycle completes.

busfaultneg (the SERVICE_HSM partition reads an MPU-permitted window past the end of physical SRAM) and nsbusfaultneg (guest0 turns off its own MPU and reads an unmapped Non-secure peripheral hole; the monitor restarts it to its limit while guest1 runs) run only on the STM32H563: M33MU turns an unmapped data access into a MemManage fault and never vectors a data BusFault, so these scenarios have no emulator row until the pinned emulator models it.

VNET has convenience targets:

make test-vnet
make test-vnet-target

test-vnet is host-only. test-vnet-target launches two authenticated wolfIP guests under M33MU.

The wolfIP guests poll once per guest millisecond and idle between ticks so empty copied IPC calls leave time for the guest clock to advance. The vnet and vnetneg M33MU scenarios allow 180 wall-clock seconds for the initial ARP delay in guest time and the copied IPC round trip on shared runners. Both require the mediated ping reply and clean breakpoint exit; vnetneg also requires both isolation faults and recovery.

Engine matrix

The full CI scenario list adds engine: [native, hsm] as a matrix dimension. Every scenario runs with both engines except the engine-specific ones listed in ENGINE_ONLY in tests/target/lib/scenario_matrix.py: the wolfHSM-only scenarios run under hsm only, and nativewire and nativewireasm, which drive the native wire, under native only. The generator is the source of truth for the exact list: python3 tests/target/lib/scenario_matrix.py --port stm32h563 --tier full --json.

hsmattackneg drives the raw wolfHSM protocol from a compromised-guest probe. It checks that a forged wolfHSM client ID cannot select the attestation key and that a wolfHSM NVM-group packet cannot reach the rollback store. The native engine does not link the wolfHSM client wire, server, or message handlers, so that exact attack surface does not exist there. Native key and namespace behavior remains covered by the common positive, cross-domain, keystore, storage, attestation, and Crypto-validation rows.

Validation of the engine split completed under both engines with:

  • the applicable M33MU scenario matrix;
  • the Arm FF-M IPC suite at 85 passed, 4 heap-dependent tests skipped, and 0 failed, test for test as recorded in tests/target/ffm_ipc_results.txt;
  • the current dev_apis Crypto schedule at 64 passed, 13 skipped, and 0 failed (77 scheduled tests; c047 is configuration-skipped in addition to the upstream schedule); and
  • the STM32H563 positive, restart, cross-domain, and conformance hardware suite.

The engine dimension changes crypto dispatch, not what M33MU proves. Emulator results still do not establish STM32 attribution or physical flash behavior.

Scenario catalogue

Every port runs the same scenario names; its port.json lists the ones it runs and exempts the rest with a reason. Beyond the families above:

  • positive boots the whole chain: wolfBoot verifies the Secure image, and both guests launch, reach the SPM through the SG veneers, and finish.
  • guestisoneg (the former gtzcneg, ahbscneg and sauneg, which still select it): guest 0 stores into guest 1's RAM window, the port's isolation refuses the store, the monitor relaunches guest 0 through its restart budget and quarantines it, and guest 1 keeps running.
  • restart makes guest 0 read Secure RAM on every launch until it is quarantined. authneg flips one byte of guest 0's image after its digest was pinned, so launch verification refuses guest 0 while guest 1 runs. handlerneg parks guest 0 in a Non-secure handler that never returns; the Secure tick still preempts it and the monitor restarts it.
  • rollbackneg, remeasureneg, manifestneg and spbudgetneg end on the verdict breakpoint their probe emits: a downgraded boot is refused, an image tampered after launch verification is caught by the re-measure before dispatch, a corrupted manifest halts boot, and restart-budget exhaustion escalates to the fail-closed platform recovery.
  • crossdomain and keystoreneg: an unprivileged storage partition's read of SPM-private RAM, or of the shared keystore band, faults at the port's band address and the guests ride it out. spfaultneg and panicneg: a partition that faults, or that the SPM must panic, is restarted in place once and serves the guests that follow.
  • periphneg has guest 0 switch off its own MPU and read the SPM's entropy peripheral through the Non-secure alias; a port with a DMA probe also runs Non-secure DMA copies out of Secure memory. Nothing may be read or changed.
  • The PSA test guest (tests/firmware/psa-guest/) runs bothpsa (the client lifecycle from both guests in one boot: SHA-256 KAT, ITS and PS, key-ops with a cross-key refusal, RNG, AES-CTR, and an attestation token verified against the IAK and wolfBoot's measurement), bothiso (a forged handle, an oversized vector, a vector in the peer window and an unknown SID refused), attestneg, and fwustage (a candidate staged through SERVICE_FWU, armed, and restored READY).
  • The remaining level 3 negatives (deputyneg, hsmpinneg, hsmfaultneg, periphspneg) run the PSA guest in both windows, and guest 0 ends the run on its breakpoint once its lifecycle is done; manifestneg2, manifestneg3 and sealhaltneg end on their verdict breakpoint, and mspovfneg and xnneg at the SPM fault.
  • confboot hosts Arm's unmodified FF-M IPC suite against the conformance image and expects 85 passed, 4 skipped, 0 failed. devstorage, devcrypto, devattest and devattestqcbor run the dev_apis suites. vaultrecover proves a foreign vault pool self-heals on a development device, and vaultrecoversec that SECURED refuses it gracefully.
  • wrpfence and wrpoff build with WT_GUEST_FLASH_WRP=1: behind a first stage that arms the guest flash fence both guests launch, and behind one that does not, neither may enter its domain.

A console boot ends on the emulator's wall-clock budget once the guests idle (exit status 127); a verdict or traced boot stops on its breakpoint with status 0; any other status fails the run. A port whose emulator or first stage needs a carried patch builds its own pinned copy, and its guide lists what each patch models. Emulator results are recorded separately from the board's.

Hardware

One runner serves every board: tests/target/run_hardware.sh <port> <scenario> builds the chain the emulator boots, programs it through the port's profile (tests/target/ports/<port>/hw.sh), boots the board once, captures its console, reads the Secure image's latches and the guests' mailboxes over SWD, and hands that evidence to the judge in tests/target/lib/scenarios.py with --env hardware. The hardware assertions are the same declared blocks the emulator uses, resolved for the port's guest kinds; a block that needs a capability the port's registry declares absent is reported N/A with the reason, never skipped silently. Each run keeps its evidence under logs/<port>-hw/<scenario>-<engine>/ (console.log, swd.txt, readback-*.bin, program.log, result.json), outside build/, and one run at a time owns the board through logs/<port>-hw.lock.

make test-hardware TARGET=<port>                       # the eight every board runs
make test-hardware TARGET=<port> WT_HW_TIER=qualification
tests/target/run_hardware.sh <port> <scenario>
tests/target/run_hardware.sh <port> <scenario> build   # toolchain host
tests/target/run_hardware.sh <port> <scenario> flash   # probe host, same checkout

The build half copies the first stage and the guest images under build/hw/ and records them in build/hw-stamp.sh relative to the checkout, so the flash half needs nothing outside it.

The eight scenarios every board runs on silicon are the contract a port meets (scenarios.py --hardware-minimum): positive, guestisoneg, periphneg, restart, psa, wrpoff, crossdomain and authneg. The qualification tier adds the port's merge-gating extras from port/<port>/port.json (hardware.qualification), and hardware.optin names the board-only scenarios a caller selects by name. The suite skips, never passes, without the board.

Every profile keeps three rules: the probe is bound by serial or unique id for every tool call (WT_HW_PROBE picks one of several), the preflight only reads, and no option byte, fuse or life-cycle state is written by the runner (tests/target/provisioning/provisioning_ctrl.sh owns those, gated). tests/target/test_hw_runner.sh checks the hook contract, those rules, the evidence paths and the board lock for every port under make test.

Bare-metal guest mailbox

The bare-metal guest records its progress in a mailbox at the base of its own Non-secure RAM window and echoes it on the console; the runner reads it over SWD. Each guide gives the port's addresses and console.

Mailbox word Offset Pass value
signature +0x00 0x47543030
step +0x04 0x00000005
psa_framework_version() +0x08 0x00000100
psa_connect(SERVICE_HSM) handle +0x10 > 0, distinct per guest
status +0x14 0x600D600D
console VERID as the guest reads it +0x18 non-zero
isolation probe latch (guestisoneg) +0x1C 1 faulted or 2 blocked; 3 leaked
isolation probe read-back (guestisoneg) +0x20 not the stored sentinel

0x600D600D in both mailboxes proves the chain booted and the Secure runtime served both Non-secure clients through the veneers; 0xBAD00000 records a failed check at the step reached.

Hardware test commands write flash and reset the board. Review the board guide before running them.

CI coverage

The workflows under .github/workflows/ separately run:

  • host unit tests;
  • compiler variants, sanitizers, and Valgrind;
  • Cortex-M33 cross-compilation of both crypto engines;
  • dependency integration;
  • the core/port split guard and the docs guard (no internal-ledger or home-directory references in the published docs);
  • fuzz targets;
  • selected and nightly M33MU scenarios for every registered port, each under its own pinned emulator image (Ports).

Trigger routing

The host, compiler, sanitizer, Valgrind, cross-compilation, integration, and core/port split checks run on every pull request, including drafts. The fuzz target also runs on pull requests as a 60-second libFuzzer smoke pass; the nightly schedule and manual dispatch run the 600-second soak instead.

The M33MU workflow (wolfBoot plus both guest lifecycles, then a select job that picks each port's tier and one matrix job that runs exactly those scenario groups) is tiered. Every pull request runs the smoke tier on both crypto engines for each port whose CI label is optional; a port marked label only in Ports runs on a pull request only with its own label, ci:all, or ci:m33mu. The full matrix runs on a push to main, on the nightly schedule, on manual dispatch (with a port input), and on a pull request that carries the port's label, ci:all, or ci:m33mu. The scenario groups per port and tier are in its port.json; make test-target TARGET=<port> runs the same smoke tier locally and WT_TIER=full every scenario. Every tier runs on both build profiles, and the level 3 bar (l3_gaps() in the matrix selftest) must hold on default and on full.

Running M33MU off a pull request

To run a port's full matrix against a branch without a PR, dispatch the workflow:

gh workflow run m33mu.yml --ref <branch> -f port=mimxrt700

To run a single scenario locally, use tests/target/run_target.sh <port> <key> (for example positive or crossdomain).

The GitHub-hosted workflows do not establish a physical-board result. Hardware output must come from the hardware runner attached to a board.

Interpreting failures

  • A host pass plus target failure usually points to architecture glue, image assembly, linker placement, or hardware policy rather than a neutral state machine.
  • A missing target prints SKIP. It is not a successful run.
  • Preserve the first failing marker and the generated log before rebuilding; target runners place detailed output under logs/ or the configured scenario log.
  • Confirm that Secure, guest, manifest, flash, and emulator addresses were built from the same configuration.