xipfs: compaction command, test suite and NXFLAT execute-in-place demo by casaroli · Pull Request #3665 · apache/nuttx-apps

casaroli · 2026-07-25T15:08:44Z

Summary

The applications that go with XIPFS, the writable execute-in-place file
system added in the companion PR:

Companion PR: apache/nuttx#19536

This one depends on that one and cannot be built without it: every Kconfig
option here depends on CONFIG_FS_XIPFS.

Three additions:

system/xipfs — a command that compacts a volume and reports how its
erase blocks are laid out. Because a XIPFS file is a single contiguous
extent, an allocation can fail for lack of a contiguous run while plenty of
free space remains, and compaction is never automatic: the file system
returns -ENOSPC and the caller decides. This command is that decision made
by hand. -n surveys without moving anything and prints the per-file table,
a block map, and the share of free space a single allocation cannot reach —
which is the number a caller facing -ENOSPC actually wants. The pass is
asked for through a descriptor for the mountpoint directory, so no file is
held open while it runs and one pass reaches every extent.
testing/fs/xipfs — the test suite, 14 sections selectable by name
because the power-loss sweeps run for minutes while everything else runs in
seconds. Beyond the routine VFS paths it aims at the properties that are
easy to get wrong and quiet when they are: that a pin taken by a task that
then dies without unmapping is still released, and that failing the Nth
flash operation and remounting always yields a consistent volume with every
committed file intact. The sweeps cover create, unlink, mkdir/rmdir and
defragmentation, each with the failing operation left torn as well as
cleanly refused.
examples/nxflatxip — the end-to-end demonstration: write an NXFLAT
module into the volume at run time the way a download would, confirm the
file system hands out a real flash pointer for it, and run two instances
concurrently. Both report the same text address, inside the flash window,
while their stacks differ. Two subcommands cover the file system rather than
the module: bench times a bulk read out of mapped flash before and after a
write, and defrag fragments the volume until an allocation genuinely
cannot be satisfied, compacts, retries, then verifies every relocated file
byte-for-byte and again after a remount.

A note on the demo module

The module is built in-tree exactly the way examples/nxflat builds its test
programs, so it needs the same two host tools, mknxflat and ldnxflat.

It deliberately has no static data and no string constants, and reports
through a callback into the firmware instead of formatting its own output.
That is not stylistic. The ldnxflat in circulation resolves the GOT entries
for .bss objects into the import table — so the module overwrites the
addresses of the functions it imports and branches into nothing on its next
call out — and drops the addend already in the instruction when resolving the
PC-relative references to .rodata, so every string pointer lands in the
middle of the code. Both were diagnosed on hardware; the comment at the top of
module/xipmod.c records them. The callback also exercises the other
direction of the interface: firmware code entered with the module's data base
still live in its PIC register.

Impact

New applications, all off by default, and all depending on
CONFIG_FS_XIPFS, so nothing changes for existing configurations.
examples/nxflatxip additionally depends on NXFLAT and LIBC_EXECFUNCS,
and its build needs mknxflat/ldnxflat and a board Make.defs that names
them — the same requirement examples/nxflat already has.
testing/fs/xipfs writes and erases its volume continuously, the power-loss
sections most of all. The Kconfig help and the documentation both say to
point it at a volume whose contents do not matter.
No existing file is modified.

Testing

Host: macOS 15 (arm64), arm-none-eabi-gcc 14.2.rel1.

Targets:

sim:xipfs — XIPFS on rammtd.
pimoroni-pico-2-plus:xipfs — XIPFS on real QSPI NOR via /dev/rpflash.

Every hardware run was bracketed by a uname -a check of the git revision and
board name, before the first command and after the last, so the logs cannot be
from a different build.

Full suite on real flash

[identity ok before] rev 0e9e189444, board pimoroni-pico-2-plus
-- basic VFS --                      -- directories --
-- write-once enforcement --         -- defragmentation --
-- strict mapping semantics --       -- power loss during create (clean) --
-- XIP mapping --                    -- power loss during create (torn) --
-- concurrent mappings --            -- power loss during unlink --
-- cross-task pin refcounting --     -- power loss during mkdir and rmdir --
-- pin release on task exit --       -- power loss during defrag --
    swept 110 injection points, covering every operation of the whole pass
==== 90 passed, 0 failed ====
[identity ok after] rev 0e9e189444, board pimoroni-pico-2-plus

Same suite on sim:xipfs: ==== 90 passed, 0 failed ====.

`nxflatxip` on hardware

=== NXFLAT module executed in place from xipfs ===
staged 268 bytes to /mnt/xipfs/xipmod
extent: block 97 x1, size 268, flash addr 0x10161000
running two concurrent instances...
[while running] pin count on the shared text = 2
  instance seed 1: text 0x10161070, stack 0x20006980, sum 2080 -- data private and intact
  instance seed 2: text 0x10161070, stack 0x20007588, sum 4160 -- data private and intact
[after exit]    pin count on the shared text = 2 (released when this task exits)
=== done ===

`nxflatxip defrag` on hardware

[1] writing 55 files...
[2] deleting every other file to fragment it...
[3] requesting a 209-block (836 KB) contiguous file
    result: FAILED -ENOSPC (free space is not contiguous)
[4] compacting...
    elapsed          : 1230 ms
    extents relocated: 27
    blocks reclaimed : 27
    largest free run : 913408 bytes (223 blocks)
    stop reason      : 0 (done)
[5] retrying, sized to the reported largest run (223 blocks)
    result: OK -- allocation now fits
[6] verifying every relocated file byte-for-byte
    all survivors intact
[7] remounting to prove it is durable
    all files intact after remount
=== defrag complete ===

`nxflatxip bench` on hardware

XIP read benchmark (256 KB)
  before any flash write : 30 ms
  after  a flash write   : 20 ms

`xipfs` command on hardware

nsh> xipfs -n
/mnt/xipfs: 250 blocks of 4096 bytes (1000 KB)
  file                             start  blocks    bytes  pin
  bin/greet                            0       1        6    0
  bin/sub/deep                         1       1        5    0
     0 |##................................................|
  '.' free   '#' in use   'P' pinned by a live mapping
  used 2, free 248, largest free run 248 blocks (992 KB)
  1 free run, fragmentation 0%
  nothing to gain: all free space is already contiguous

tools/checkpatch.sh -c -u -m -g passes on every commit in the branch.

casaroli · 2026-07-26T08:30:13Z

The failing sim:alsa CI job is unrelated to this PR's XIPFS/NXFLAT example, test, and command changes.

The Linux (sim-01) log fails while linking the bundled LAME encoder, with unresolved x86 runtime-dispatch vector symbols including quantize_lines_xrpow_avx2, quantize_lines_xrpow_sse2, and count_bit_esc_sse2. This is a pre-existing CMake source-integration issue: the LAME CMake target omits the vector translation units enabled by its generated configuration.

The fixes are tracked separately:

Root-cause Apps fix: audioutils/lame: Add CMake vector sources. #3666
Companion NuttX codec-header integration fix: sim/alsa: Use bundled codec headers. nuttx#19541

The two fixes were validated together with the CI-equivalent sim:alsa CMake/Ninja build. The Apps PR should merge first; subsequent NuttX PR CI runs will then use the corrected Apps master.

xiaoxiang781216 · 2026-07-26T14:32:31Z

+  if (fd >= 0)
+    {
+      n = read(fd, got, len);
+      ok = (n == (ssize_t)len) && (memcmp(got, want, len) == 0);


remove the cast

xiaoxiang781216 · 2026-07-26T14:33:02Z

+
+  for (i = 0; i < len; i++)
+    {
+      buf[i] = (uint8_t)(seed + (i * 7));


remove the cast

casaroli

@xiaoxiang781216 i addressed most of the concerns, most casts are gone with the use of <inttypes.h>, some of them are necessary (signed/unsigned comparison, larger-width arithmethic), and some of them are explicit truncation, I think we should keep them to avoid modern compiler warnings.

casaroli · 2026-07-27T00:27:46Z

+      width = XIPFS_NAME_MAX;
+      for (f = 0; f < s->nfiles; f++)
+        {
+          int len = (int)strlen(s->files[f].name);


i think we should keep this to make the conversion explicit

casaroli · 2026-07-27T00:30:55Z

+    }
+
+  nread = read(fd, g_buffer, len + 1);
+  if (nread == (ssize_t)len && memcmp(g_buffer, expect, len) == 0)


I think this is necessary

casaroli · 2026-07-27T00:43:20Z

+  fill_pattern(g_buffer, len, seed);
+
+  nwritten = write(fd, g_buffer, len);
+  if (nwritten != (ssize_t)len)


this is necessary to avoid warning

xipfs has a defragmentation ioctl and no way to reach it from a shell. The only caller was a demo in examples/nxflatxip, which built its own block map to show what compaction had done. xipfs [-n] [-t <ms>] [<mountpoint>] -n surveys and reports without moving anything: where each file sits, a map of the volume, and how much of the free space a single allocation can reach. That last number is the one a caller facing -ENOSPC actually wants; at 0% the largest possible file already fits however scattered the map looks. The compaction is asked for through a descriptor for the mountpoint directory, so no file inside the volume is open while it runs and a single pass can reach every extent. The walk descends into the directories xipfs synthesises from names, and reports each file by its path relative to the mount, so a volume that uses them is described in full rather than down to its first level. Block totals come from statfs rather than XIPFSIOC_EXTENTINFO, because that one does name a file and an empty volume has none, yet its geometry is still worth reporting. Assisted-by: Claude Code:claude-opus-4-8 Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>

Twelve sections, selected by name on the command line because the power loss sweeps run for minutes while everything else runs in seconds. The routine part covers the VFS paths, the write-once rules (reopen for write, append, seek during write, truncate of a written file are all refused) and both mmap variants: that a plain mapping lands inside the media window with no heap growth, that MAP_XIP_STRICT fails with ENXIO rather than copying, and that N mappings of one file produce N pins on one extent. The rest is aimed at the two properties that are easy to get wrong and quiet when they are: Pin release. A pin taken by one task and a task that dies with a mapping still live both have to end with the extent movable again -- the second without the task ever calling munmap, since a module that faults never will. Both are checked by asking the defragmenter to move the extent afterwards. Power-loss atomicity. With CONFIG_FS_XIPFS_FAULT_INJECT the suite fails the Nth flash operation, remounts, and checks the volume is consistent and every file that had been committed is byte-for-byte intact. It sweeps N across create, unlink and defragmentation, and repeats the sweep with the failing write torn -- a partial program rather than a clean refusal -- which is what a real power loss mid-program leaves behind. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>

The point of a writable execute-in-place file system is that a module can arrive after the firmware was built and still run without being copied into RAM. This demonstrates exactly that, end to end: it writes an NXFLAT module into xipfs the way a download would, declaring the size up front so the extent is exact, checks the file system can hand out a real flash pointer for it, and runs two instances concurrently. Both instances report the address of their own text and of their stack. The text address is the same in both and is inside the flash window -- it is the address the file occupies on the media -- while the stacks differ. While they run, the extent carries one pin per instance, which is what stops the defragmenter relocating code that is executing. Two subcommands cover the file system rather than the module. bench times a bulk read out of the mapped flash before and after a write, which on the RP2350 is the cost of whichever path the driver used to restore XIP. defrag fills the volume, deletes every other file until an allocation genuinely cannot be satisfied, compacts, retries the same allocation, then verifies every relocated file byte for byte and again after a remount, printing a block map at each step. The module has neither static data nor string constants, and reports through a callback into the firmware instead of formatting its own output. That is not stylistic: the ldnxflat in circulation resolves the GOT entries for .bss objects into the import table, and drops the addend when it resolves the PC-relative references to .rodata, so a module using either quietly computes wrong addresses. The comment in module/xipmod.c records both. The callback also exercises the other direction of the interface, firmware code entered with the module's data base still live in r10. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>

casaroli mentioned this pull request Jul 25, 2026

fs/xipfs: writable execute-in-place file system for downloadable modules apache/nuttx#19536

Open

github-actions Bot added Size: XL Area: System Area: Examples labels Jul 25, 2026

acassis previously approved these changes Jul 25, 2026

View reviewed changes

casaroli dismissed acassis’s stale review via f490d5b July 25, 2026 16:40

casaroli force-pushed the fs-xipfs branch from 1cc8cc4 to f490d5b Compare July 25, 2026 16:40

xiaoxiang781216 reviewed Jul 26, 2026

View reviewed changes

casaroli force-pushed the fs-xipfs branch from f490d5b to c4bf93b Compare July 27, 2026 00:25

casaroli commented Jul 27, 2026

View reviewed changes

casaroli added 3 commits July 27, 2026 02:51

casaroli force-pushed the fs-xipfs branch from c4bf93b to 7ee5eca Compare July 27, 2026 00:53

Uh oh!

Conversation

casaroli commented Jul 25, 2026

Summary

A note on the demo module

Impact

Testing

Full suite on real flash

nxflatxip on hardware

nxflatxip defrag on hardware

nxflatxip bench on hardware

xipfs command on hardware

Uh oh!

casaroli commented Jul 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xiaoxiang781216 Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xiaoxiang781216 Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

casaroli left a comment

Choose a reason for hiding this comment

Uh oh!

casaroli Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

casaroli Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

casaroli Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

`nxflatxip` on hardware

`nxflatxip defrag` on hardware

`nxflatxip bench` on hardware

`xipfs` command on hardware