SquashFS: extended file inodes (files over 4 GB, sparse, hard links), sparse blocks, compressor options - #78
Merged
Olof-Lagerkvist merged 5 commits intoSep 9, 2026
Conversation
The reader knew only the basic regular file inode (type 2). mksquashfs writes the extended form (type 9) for any file larger than 4 GB, for sparse files and for files with more than one hard link, so a disk image or a hard-linked file inside an image could not be opened. ExtendedFileInode reads the 56-byte layout (64-bit start block and size, sparse bytes, link count, xattr index); the block list follows it as before. RegularInode.StartBlock is a long now. A block stored with length 0 is sparse and reads as zeros; FileContentBuffer handled it as a block to decompress. While there, the offset within a block was taken from the read's start position instead of the current one, so a read spanning blocks copied from the wrong place after the first block. Gzip images made with a compression level (and lzo and zstd images with options) carry a compressor options record; the reader refused them, though decompression does not depend on the options. They are read into ZLibCompressionOptions, LzoCompressionOptions and ZStdCompressionOptions. Test: an image made by mksquashfs 4.7 (gzip level 6, 128 KB blocks) with a sparse file, a hard-linked file, a fragment-only file and a three-block file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
0xorial
force-pushed
the
squashfs-extended-file-inode
branch
from
September 9, 2026 15:28
e10fd5b to
40ae40d
Compare
Member
|
Great work! Thanks! It looks generally good, but I would feel more confident about it if you could add some more test cases for the specific changes. For example, one with an explicit single |
…r 4 GiB Two tests asked for in review. SingleReadAcrossStoredBlockBoundary reads text.bin (128 KB blocks) with one Read() call starting in one stored block and ending in the next, and in the fragment that holds the tail: the case the offset fix is for. SparseFileLargerThan4GiB opens huge.bin, 4 GiB + 4 bytes with "mid" at 2 GiB + 5 and "end!" after the 4 GiB mark, which only an extended inode can describe; nearly every block is sparse, so the image is 4 KB and the reads cost nothing. The images are built by make-fixtures.sh with fixed times and ownership, so they can be regenerated byte for byte; the script lists their SHA-256. extended-inodes.sqsh is regenerated the same way. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SquashFixtureRecipeTest builds the two source trees in code, runs mksquashfs with the fixed times and ownership, and asserts the images equal the embedded ones byte for byte, so the binaries in the repository are verified by the suite wherever squashfs-tools is installed (Linux and macOS); without the tool, and on Windows, it passes without checking. Replaces make-fixtures.sh. Written against the .NET Framework targets too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Member
|
Thanks! The issue now seems to be related to the new C# code for creating the squashfs image. It looks like it expects another version of mksquashfs than what squashfs-tools contains in the CI Ubuntu environment. So, I would suggest either reverting back to the sh script based logic or change the new C# code to accept a different version of squashfs generator. |
…ase only Another release of squashfs-tools (4.6.1 on the Ubuntu runner) lays an image out differently, so the rebuilt images cannot equal the embedded ones byte for byte there. Byte equality is now demanded only when the installed mksquashfs is the release that made the fixtures; with any other release the rebuilt image is compared with the embedded one through the reader, file by file: names, sizes and bytes, the 4 GiB file by sampled ranges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ones as the fallback The reader tests now run on images built at test time by whatever squashfs-tools is installed (4.5 or later), once per run, from the recipe in SquashFixtures; the images embedded in the assembly are opened only where the tool is absent. The comparison of rebuilt images against the embedded ones is gone: with a trusted mksquashfs on the machine there is nothing to prove about the binaries, and without it there is nothing to compare. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Olof-Lagerkvist
marked this pull request as ready for review
September 9, 2026 23:12
Olof-Lagerkvist
merged commit Sep 9, 2026
bedb3c2
into
LTRData:LTRData.DiscUtils-initial
3 checks passed
Member
|
Merged! Thanks a lot for your contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The SquashFS reader could not open a file that mksquashfs stores in an extended file inode (type 9): any file larger than 4 GB, a sparse file, or a file with more than one hard link. Such files came out as unreadable. This adds the inode, reads sparse blocks as zeros, and accepts images whose superblock carries compressor options (gzip made with a compression level, lzo, zstd), which the reader refused although decompression does not depend on them.
Changes
ExtendedFileInode: the 56-byte layout (64-bit start block and size, sparse bytes, link count, xattr index), block list after it as for the basic form.InodegetsReadHeader/WriteHeaderfor the common 16 bytes;RegularInode.StartBlockbecomeslong.FileContentBuffer: a block stored with length 0 is sparse and reads as zeros. Also fixes a pre-existing bug: the offset within a block was computed from the read's start position (pos) instead of the current position, so a read spanning more than one block copied from the wrong offset after the first block.CompressionOptions: gzip, lzo and zstd options are read (ZLibCompressionOptionswith level, window size and strategies;LzoCompressionOptionswith algorithm and level;ZStdCompressionOptionswith level) instead of throwing "Unsupported compression options".Test
SquashFileSystemReaderTest.ExtendedFileInodesSparseBlocksAndCompressorOptionsopensextended-inodes.sqsh(4 KB, made by mksquashfs 4.7.5 with-comp gzip -Xcompression-level 6 -b 128K -no-xattrs): a sparse 2 MB file ending in "end", a hard-linked file under two names, a fragment-only file, and a file spanning three blocks, and checks their content, including a read across a block boundary.The reader was also exercised on a 95 GB VHDX inside a 43 GB image (1.1 million files walked through DiscUtils.Ntfs on top of it), with the image's hash matching the loose file.
Draft: opened for review of the approach first; happy to adjust naming or split the offset fix into its own PR.