Skip to content

[memory] layout file: save a built forest and load it validated, owned or mapped - #69

Merged
victor-ab merged 3 commits into
victor-ab/lightgbm-textfrom
victor-ab/compiled-layout
Nov 7, 2026
Merged

victor-ab merged 3 commits into
victor-ab/lightgbm-textfrom
victor-ab/compiled-layout

Conversation

@victor-ab

@victor-ab victor-ab commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Opt-in and experimental, behind the layout-file feature.

Motivation

Even with the rest of the stack, loading a large categorical model rebuilds its layout from the Treelite file every time: on the stack's private model, about 1 s and a heap peak over three times the loaded model, in every process. A native compiled model loads in a fraction of that and keeps its code in file-backed pages that processes share. This PR measures the same for TreeWalker: save the built forest once, then load it validated, either into owned arrays or from a file mapping.

Design

Behind an opt-in layout-file feature; the default build and representation are unchanged. Files hold production layouts only: in research builds the three functions return LoadError::Unsupported, and research rebuilds from the source model.

  1. layout file: add opt-in format and layout saving. Forest::save_layout(path) writes a .twl file: a versioned header (magic TWLAYOUT, schema 1, crate version, build ID, architecture, endianness, pointer width, link stride, configuration fingerprint, threshold type), then 16-byte-aligned, length-prefixed arrays in native bytes, and explicit scalar fields for the metadata: no Rust layouts, pointers or caches are saved. It stores the exception words and either width of relative slots (from [memory] layout: exceptions take 6 bytes, with 16-bit feature-relative slots and a 32-bit fallback #67); Fixed leaf values travel in their nodes (from [memory] layout: hold fixed leaf values in their leaf nodes #74), and Wide plans keep their leaf table. Array counts are bounded by the bytes left in the file, at each element type's minimum encoded size. The build ID hashes build.rs and every file under src/, so a file loads only into a build of the same sources, whatever the crate version.
  2. layout file: validate owned layouts before exposing the kernel. Forest::load_layout(path), safe: streams into owned arrays and validates everything the unchecked kernel relies on (configuration and precision, the dummy, node fields and links, Fixed leaves' encoding and exact-sum bounds and Wide leaves' table indices, binary-tree topology and coverage, recomputed scratch needs, prefixes, categorical domains, exception slots within their feature, in the width a build would choose, and matching words, monotone offsets, predicate coverage, and the varying predicates' order by feature and kind, which their mask ranges rely on); features own disjoint slots and tables and trees disjoint nodes, nodes' mask fields match their predicates, and the mask plan equals one rebuilt from the validated bins and predicates (the rebuild capped at the stored plan's size). Checks that would only guard the outputs are left out, as a writer can produce a valid file with any predictions and the checksum covers accidents: the two heavy-side encodings' agreement, the no-category code having no exceptions, and threshold and codebook order. Validation takes O(R log R) time in the file's records (sorts, and a binary search per split); its scratch holds the rebuilt plan beside the stored one, and ownership and traversal tables. Predictors rebuild their chunks and scratch. Errors are LoadError::Io, LoadError::Layout and LoadError::Limit. A checksum guards against corruption in addition to, never instead of, validation.
  3. layout file: map validated arrays from immutable files. On 64-bit Unix, unsafe fn Forest::map_layout(path) borrows the nodes (with their Fixed leaf values), exception words and slots from a read-only private mapping, with the same validation; the forest and its clones and predictors keep the mapping. Array access is a pointer and length either way, with no owned-or-mapped branch in the kernel. The caller must ensure the file is neither written nor truncated until every forest, clone and predictor from it drops: validation can't protect a live mapping from later writes.

docs/layout-file.md has the format, limits and the unsafe contract.

Evidence

On the private model, Apple M4 Pro, against a normal load of the Treelite file with this stack (from_bytes, the file read included; heap peaks against from_reader's, the lower):

load time heap peak heap after loading
normal load about 1 s (100%) 290.3 MiB (100%) 91.4 MiB (100%)
load_layout (owned) 5% 31% 100%
map_layout (mapped) 5% 0.3% 0.8%, the rest file-backed

Load times include validation (three interleaved rounds of three loads; owned and mapped loads take 0.05 to 0.06 s, normal ones 0.96 to 1.06 s). With the model's configuration, validation's scratch is small: the owned load peaks at the loaded model's heap. It isn't small for every model: with every feature varying, the peak is 1.84 times the loaded heap owned and 2.51 times mapped, mostly the rebuilt mask plan and the ownership tables; on a public generated forest with 1,024 two-sided categorical predicates, 1.37 and 1.64 times. Saving takes 0.05 s; the file is 90.8 MiB, about the loaded model's size. Request latency on a mapped forest equals the normal forest's within noise: at 1, 32 and 512 candidates, the p50 differences of four paired rounds (A B B A) span zero.

  • Outputs are bitwise equal for the normal, owned and mapped forests, on synthetic rows and on the generated requests.
  • Tests: round trips on all the repository's import fixtures (Treelite and native LightGBM) and on generated edge forests (many numeric features, 1,024 two-sided categorical predicates, singleton and nested), for both slot widths, owned and mapped; corruption tests for each validation category, with the checksum recomputed so validation must catch them; aliased tables and trees rejected before quadratic work; a mask plan whose rebuild outgrows the stored one rejected at the stored size. The public API's tests run from experiments/layout-file, a crate outside the workspace that enables only layout-file, so they test the production build, with body corruptions whose checksum is recomputed and a file from other sources rejected; CI and the fast checks run it.
  • cargo test --release --features research,layout-file and the production crate's tests pass. Clippy for the library alone, with and without layout-file (every commit with it), the workspace, and research; fmt; Miri on the codec and owned-storage unsafe paths.

Decided in review

  • Ship behind the opt-in layout-file feature while the format is v1; make it the default once it has a build identity (now in the header) and a compatibility policy, and keep the owned loader for users who load into their own memory.
  • Mapping on 64-bit Unix only, and the unsafe fn contract, which fits a deploy that writes a new file and renames it into place. Durable publication waits until a consumer needs it.

@victor-ab
victor-ab added this pull request to stack #70 November 6, 2026 13:20
@victor-ab victor-ab changed the title victor ab/compiled layout [memory] compiled: save a built forest and load it validated, owned or mapped Nov 6, 2026
@victor-ab
victor-ab marked this pull request as draft November 6, 2026 13:20
@victor-ab
victor-ab force-pushed the victor-ab/compiled-layout branch 2 times, most recently from d60ed60 to 1711364 Compare November 6, 2026 15:22
@victor-ab
victor-ab removed this pull request from stack #70 November 6, 2026 23:40
@victor-ab
victor-ab force-pushed the victor-ab/compiled-layout branch from 1711364 to 6985924 Compare November 6, 2026 23:40
@victor-ab
victor-ab changed the base branch from victor-ab/leaf-i64 to victor-ab/lightgbm-text November 6, 2026 23:40
@victor-ab
victor-ab added this pull request to stack #76 November 6, 2026 23:41
@victor-ab victor-ab changed the title [memory] compiled: save a built forest and load it validated, owned or mapped [memory] layout file: save a built forest and load it validated, owned or mapped Nov 6, 2026
@victor-ab
victor-ab force-pushed the victor-ab/compiled-layout branch 3 times, most recently from 083cd0b to 11e3cde Compare November 7, 2026 00:24
@victor-ab
victor-ab force-pushed the victor-ab/compiled-layout branch from 11e3cde to 2462f33 Compare November 7, 2026 01:15
@victor-ab
victor-ab marked this pull request as ready for review November 7, 2026 01:15
@victor-ab
victor-ab requested a review from ukaratay November 7, 2026 01:18
@victor-ab
victor-ab force-pushed the victor-ab/compiled-layout branch from 2462f33 to d277f56 Compare November 7, 2026 02:19

@ukaratay ukaratay left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is to support the library usage inside edge nodes with many processes. That makes map_layout the part that matters: one shared copy per node instead of 91 MiB of heap per process, and a load in 5% of the time. So yes to shipping it, behind the opt-in layout-file feature while the format is v1. Once it has a build identity and a compatibility policy, I'd make it default and drop the feature. It's still a library, so the owned streaming loader stays for users who load into their own memory.

Wrong in the description:

  • "Validation work is linear in the file's records." It sorts (validate.rs:35) and binary-searches once per split (layout_file.rs:276, :338), so O(R log R). That's fine as a bound, but say so.
  • "Validation allocates nothing beyond the arrays." It builds a second MaskPlan while the stored one is alive (validate.rs:358-360), plus scratch for ownership, indexes and queries. Measure the peak on a predicate-heavy fixture and state that number.

Missing:

  • Tests of the production build. Research changes the header, the codec tail and the checks, and CI only lints the production layout-file build. The six integration tests pass from an external crate that depends on the library with only layout-file. Put that crate in CI, with corruptions whose checksum is recomputed.
  • A build identity in the header. This stack changes the layout while staying at 3.2.2, so a file from another commit passes the header check. "Bump the schema" is a rule nothing enforces.

What I'd cut:

  • The checks that only guard output correctness: sides.rs with its seeded hashes and merge budget, the threshold order, code zero. A hostile writer can already produce a valid file with any predictions it wants, and the checksum covers accidents. Your doc says the two-sided agreement is "semantic, never an index the kernel trusts". Keep every check that guards an unchecked access; the safe load_layout needs those.
  • The research schema. Files become production-only, research rebuilds from the source model, and the tests move into the production crate above. This is the one cut that removes an option, so push back if research needs mapped layouts.

On your other questions: mapping on 64-bit Unix only is fine. unsafe fn is the right shape, because MAP_PRIVATE can't stop writes or truncation after validation. The contract fits a deploy that writes a new file and renames it into place. Durable publication can wait until a consumer needs it.

@victor-ab
victor-ab force-pushed the victor-ab/compiled-layout branch from d277f56 to 07c27f9 Compare November 7, 2026 18:29
@victor-ab

Copy link
Copy Markdown
Collaborator Author

Done, in the same three commits:

  • Description: validation is O(R log R), from the sorts and the per-split binary search. The "allocates nothing" claim was wrong. Measured with dhat (forest only): with the private model's configuration, the owned load peaks at its loaded heap (1.00×) and the mapped one at 1.05× its heap. With every feature varying, the ratios are 1.84× and 2.51×, and on a public generated forest with 1,024 two-sided categorical predicates, 1.37× and 1.64×. The docs state these.
  • Production tests: experiments/layout-file is a crate outside the workspace that depends on the library with only layout-file; inside the workspace, cargo test --workspace would unify research back on. It holds the integration tests, plus body corruptions with the checksum recomputed and a file built from other sources, through both load_layout and map_layout. CI's fast job and checks.sh fast run it.
  • Build identity: with layout-file on, build.rs hashes build.rs and every file under src/ into the header, by sorted path and content, with the file checksum's hash. Cargo.toml is left out, as publishing rewrites it. Any source change makes older files fail with LoadError::Layout, whatever the version.
  • Cut: sides.rs with its hashes and budget, the code-zero check, threshold order and codebook order; none of them guards an unchecked access. Kept: the varying predicates' (feature, kind) order, which build_feature_ranges' partition relies on, now with a regression test.
  • Research schema: cut, as nothing in research needs mapped layouts. In research builds, all three functions return LoadError::Unsupported rather than a forest without its research data. The in-crate white-box tests use internal entry points on the production format.

@victor-ab
victor-ab requested a review from ukaratay November 7, 2026 18:30

@ukaratay ukaratay left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All five done, and the production crate runs in CI (7 tests on this head). The build ID drops files on any source edit, comments included. That's the right direction while the format is v1.

Save a built forest's prediction layout behind the opt-in layout-file
feature, so a later process can load it without decoding the source
model or rebuilding its categorical chains. The default build and its
representation are unchanged.

Forest::save_layout writes a .twl file: a header with the TWLAYOUT
magic, schema version 1, the crate version, a build ID, the target's
architecture, endianness, pointer width and link stride, a
configuration fingerprint and the threshold type; then 16-byte-aligned,
length-prefixed arrays in native bytes and metadata encoded field by
field, never Rust layouts, pointers or caches; then a checksum.
Exception slots keep their feature-relative width, Fixed leaf values
travel in their nodes, and Wide plans keep their one leaf vector.

With layout-file on, build.rs hashes build.rs and every file under src/,
by sorted path and content, into the build ID, so a file loads only into
a build of the sources that wrote it, whatever the crate version. Files
hold production layouts only: in research builds the layout-file
functions return Unsupported, and research rebuilds from the source
model.

The schemas are in layout_file::codec, except those that need a type's
private fields, which sit beside the type. Array counts are bounded by
the bytes left in the file at each element's minimum encoded size.
LoadError::Layout reports an incompatible or corrupt file. CI lints the
production layout-file build and runs its tests; docs/layout-file.md
describes the format.
Forest::load_layout reads a layout file into owned arrays and, before
returning, validates everything the unchecked kernel relies on: the
configuration and precision, node fields and links, Fixed leaves'
encoding and exact-sum bounds, Wide leaves' table indices, binary-tree
topology and coverage, recomputed scratch needs, prefixes, categorical
domains, exception slots within their feature and in the width a build
would choose, predicate coverage, and the varying predicates' order by
feature and kind, which their mask ranges rely on. Features must own
disjoint slots and tables and trees disjoint nodes, nodes' mask fields
must match their predicates, and the mask plan must equal one rebuilt
from the validated bins and predicates, the rebuild capped at the stored
plan's size. Validation takes O(R log R) time in the file's records; its
scratch holds the rebuilt plan beside the stored one, and ownership
and traversal tables. Predictors rebuild their chunks and scratch.

Checks that would only guard the outputs are left out, as a writer can
produce a valid file with any predictions and the checksum covers
accidents: the two heavy-side encodings' agreement, the no-category
code having no exceptions, and the order of thresholds and codebooks.

Tests round-trip every import fixture and generated edge forests, and
corrupt validated fields with the checksum recomputed, so that
validation, not the checksum, must reject them. The public API's tests
live in experiments/layout-file, a crate outside the workspace that
enables only layout-file, so they test the production build; CI and the
fast checks run it.
On 64-bit Unix, the unsafe Forest::map_layout borrows the nodes, with
their Fixed leaf values, and the exception words and slots from a
read-only private mapping, with the same validation as load_layout. The
other metadata and a Wide plan's leaf values stay owned. The forest, its
clones and its predictors keep the mapping alive. An array is a pointer
and a length either way, so the kernel has no owned-or-mapped branch;
without layout-file the arrays remain plain vectors.

The caller must keep the file from being written or truncated until
every forest, clone and predictor from it drops: validation cannot
protect a live mapping from later writes. The round-trip and corruption
tests run mapped as well as owned, and docs/layout-file.md gives
validation's heap peak for both.
@victor-ab
victor-ab force-pushed the victor-ab/compiled-layout branch from 07c27f9 to 84f0207 Compare November 7, 2026 20:49
@victor-ab
victor-ab merged commit e50042f into main Nov 7, 2026
6 of 8 checks passed
@victor-ab
victor-ab deleted the victor-ab/compiled-layout branch November 15, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants