Repository navigation
[memory] layout file: save a built forest and load it validated, owned or mapped - #69
Conversation
d60ed60 to
1711364
Compare
1711364 to
6985924
Compare
083cd0b to
11e3cde
Compare
11e3cde to
2462f33
Compare
2462f33 to
d277f56
Compare
ukaratay
left a comment
There was a problem hiding this comment.
This change is to support the library usage inside edge nodes with many processes. That makes map_layout the part that matters: one shared copy per node instead of 91 MiB of heap per process, and a load in 5% of the time. So yes to shipping it, behind the opt-in layout-file feature while the format is v1. Once it has a build identity and a compatibility policy, I'd make it default and drop the feature. It's still a library, so the owned streaming loader stays for users who load into their own memory.
Wrong in the description:
- "Validation work is linear in the file's records." It sorts (
validate.rs:35) and binary-searches once per split (layout_file.rs:276,:338), so O(R log R). That's fine as a bound, but say so. - "Validation allocates nothing beyond the arrays." It builds a second
MaskPlanwhile the stored one is alive (validate.rs:358-360), plus scratch for ownership, indexes and queries. Measure the peak on a predicate-heavy fixture and state that number.
Missing:
- Tests of the production build. Research changes the header, the codec tail and the checks, and CI only lints the production
layout-filebuild. The six integration tests pass from an external crate that depends on the library with onlylayout-file. Put that crate in CI, with corruptions whose checksum is recomputed. - A build identity in the header. This stack changes the layout while staying at 3.2.2, so a file from another commit passes the header check. "Bump the schema" is a rule nothing enforces.
What I'd cut:
- The checks that only guard output correctness:
sides.rswith its seeded hashes and merge budget, the threshold order, code zero. A hostile writer can already produce a valid file with any predictions it wants, and the checksum covers accidents. Your doc says the two-sided agreement is "semantic, never an index the kernel trusts". Keep every check that guards an unchecked access; the safeload_layoutneeds those. - The research schema. Files become production-only, research rebuilds from the source model, and the tests move into the production crate above. This is the one cut that removes an option, so push back if research needs mapped layouts.
On your other questions: mapping on 64-bit Unix only is fine. unsafe fn is the right shape, because MAP_PRIVATE can't stop writes or truncation after validation. The contract fits a deploy that writes a new file and renames it into place. Durable publication can wait until a consumer needs it.
d277f56 to
07c27f9
Compare
|
Done, in the same three commits:
|
ukaratay
left a comment
There was a problem hiding this comment.
All five done, and the production crate runs in CI (7 tests on this head). The build ID drops files on any source edit, comments included. That's the right direction while the format is v1.
Save a built forest's prediction layout behind the opt-in layout-file feature, so a later process can load it without decoding the source model or rebuilding its categorical chains. The default build and its representation are unchanged. Forest::save_layout writes a .twl file: a header with the TWLAYOUT magic, schema version 1, the crate version, a build ID, the target's architecture, endianness, pointer width and link stride, a configuration fingerprint and the threshold type; then 16-byte-aligned, length-prefixed arrays in native bytes and metadata encoded field by field, never Rust layouts, pointers or caches; then a checksum. Exception slots keep their feature-relative width, Fixed leaf values travel in their nodes, and Wide plans keep their one leaf vector. With layout-file on, build.rs hashes build.rs and every file under src/, by sorted path and content, into the build ID, so a file loads only into a build of the sources that wrote it, whatever the crate version. Files hold production layouts only: in research builds the layout-file functions return Unsupported, and research rebuilds from the source model. The schemas are in layout_file::codec, except those that need a type's private fields, which sit beside the type. Array counts are bounded by the bytes left in the file at each element's minimum encoded size. LoadError::Layout reports an incompatible or corrupt file. CI lints the production layout-file build and runs its tests; docs/layout-file.md describes the format.
Forest::load_layout reads a layout file into owned arrays and, before returning, validates everything the unchecked kernel relies on: the configuration and precision, node fields and links, Fixed leaves' encoding and exact-sum bounds, Wide leaves' table indices, binary-tree topology and coverage, recomputed scratch needs, prefixes, categorical domains, exception slots within their feature and in the width a build would choose, predicate coverage, and the varying predicates' order by feature and kind, which their mask ranges rely on. Features must own disjoint slots and tables and trees disjoint nodes, nodes' mask fields must match their predicates, and the mask plan must equal one rebuilt from the validated bins and predicates, the rebuild capped at the stored plan's size. Validation takes O(R log R) time in the file's records; its scratch holds the rebuilt plan beside the stored one, and ownership and traversal tables. Predictors rebuild their chunks and scratch. Checks that would only guard the outputs are left out, as a writer can produce a valid file with any predictions and the checksum covers accidents: the two heavy-side encodings' agreement, the no-category code having no exceptions, and the order of thresholds and codebooks. Tests round-trip every import fixture and generated edge forests, and corrupt validated fields with the checksum recomputed, so that validation, not the checksum, must reject them. The public API's tests live in experiments/layout-file, a crate outside the workspace that enables only layout-file, so they test the production build; CI and the fast checks run it.
On 64-bit Unix, the unsafe Forest::map_layout borrows the nodes, with their Fixed leaf values, and the exception words and slots from a read-only private mapping, with the same validation as load_layout. The other metadata and a Wide plan's leaf values stay owned. The forest, its clones and its predictors keep the mapping alive. An array is a pointer and a length either way, so the kernel has no owned-or-mapped branch; without layout-file the arrays remain plain vectors. The caller must keep the file from being written or truncated until every forest, clone and predictor from it drops: validation cannot protect a live mapping from later writes. The round-trip and corruption tests run mapped as well as owned, and docs/layout-file.md gives validation's heap peak for both.
07c27f9 to
84f0207
Compare
Opt-in and experimental, behind the
layout-filefeature.Motivation
Even with the rest of the stack, loading a large categorical model rebuilds its layout from the Treelite file every time: on the stack's private model, about 1 s and a heap peak over three times the loaded model, in every process. A native compiled model loads in a fraction of that and keeps its code in file-backed pages that processes share. This PR measures the same for TreeWalker: save the built forest once, then load it validated, either into owned arrays or from a file mapping.
Design
Behind an opt-in
layout-filefeature; the default build and representation are unchanged. Files hold production layouts only: in research builds the three functions returnLoadError::Unsupported, and research rebuilds from the source model.layout file: add opt-in format and layout saving.Forest::save_layout(path)writes a.twlfile: a versioned header (magicTWLAYOUT, schema 1, crate version, build ID, architecture, endianness, pointer width, link stride, configuration fingerprint, threshold type), then 16-byte-aligned, length-prefixed arrays in native bytes, and explicit scalar fields for the metadata: no Rust layouts, pointers or caches are saved. It stores the exception words and either width of relative slots (from [memory] layout: exceptions take 6 bytes, with 16-bit feature-relative slots and a 32-bit fallback #67); Fixed leaf values travel in their nodes (from [memory] layout: hold fixed leaf values in their leaf nodes #74), and Wide plans keep their leaf table. Array counts are bounded by the bytes left in the file, at each element type's minimum encoded size. The build ID hashesbuild.rsand every file undersrc/, so a file loads only into a build of the same sources, whatever the crate version.layout file: validate owned layouts before exposing the kernel.Forest::load_layout(path), safe: streams into owned arrays and validates everything the unchecked kernel relies on (configuration and precision, the dummy, node fields and links, Fixed leaves' encoding and exact-sum bounds and Wide leaves' table indices, binary-tree topology and coverage, recomputed scratch needs, prefixes, categorical domains, exception slots within their feature, in the width a build would choose, and matching words, monotone offsets, predicate coverage, and the varying predicates' order by feature and kind, which their mask ranges rely on); features own disjoint slots and tables and trees disjoint nodes, nodes' mask fields match their predicates, and the mask plan equals one rebuilt from the validated bins and predicates (the rebuild capped at the stored plan's size). Checks that would only guard the outputs are left out, as a writer can produce a valid file with any predictions and the checksum covers accidents: the two heavy-side encodings' agreement, the no-category code having no exceptions, and threshold and codebook order. Validation takes O(R log R) time in the file's records (sorts, and a binary search per split); its scratch holds the rebuilt plan beside the stored one, and ownership and traversal tables. Predictors rebuild their chunks and scratch. Errors areLoadError::Io,LoadError::LayoutandLoadError::Limit. A checksum guards against corruption in addition to, never instead of, validation.layout file: map validated arrays from immutable files. On 64-bit Unix,unsafe fn Forest::map_layout(path)borrows the nodes (with their Fixed leaf values), exception words and slots from a read-only private mapping, with the same validation; the forest and its clones and predictors keep the mapping. Array access is a pointer and length either way, with no owned-or-mapped branch in the kernel. The caller must ensure the file is neither written nor truncated until every forest, clone and predictor from it drops: validation can't protect a live mapping from later writes.docs/layout-file.mdhas the format, limits and the unsafe contract.Evidence
On the private model, Apple M4 Pro, against a normal load of the Treelite file with this stack (
from_bytes, the file read included; heap peaks againstfrom_reader's, the lower):load_layout(owned)map_layout(mapped)Load times include validation (three interleaved rounds of three loads; owned and mapped loads take 0.05 to 0.06 s, normal ones 0.96 to 1.06 s). With the model's configuration, validation's scratch is small: the owned load peaks at the loaded model's heap. It isn't small for every model: with every feature varying, the peak is 1.84 times the loaded heap owned and 2.51 times mapped, mostly the rebuilt mask plan and the ownership tables; on a public generated forest with 1,024 two-sided categorical predicates, 1.37 and 1.64 times. Saving takes 0.05 s; the file is 90.8 MiB, about the loaded model's size. Request latency on a mapped forest equals the normal forest's within noise: at 1, 32 and 512 candidates, the p50 differences of four paired rounds (A B B A) span zero.
experiments/layout-file, a crate outside the workspace that enables onlylayout-file, so they test the production build, with body corruptions whose checksum is recomputed and a file from other sources rejected; CI and the fast checks run it.cargo test --release --features research,layout-fileand the production crate's tests pass. Clippy for the library alone, with and withoutlayout-file(every commit with it), the workspace, and research; fmt; Miri on the codec and owned-storage unsafe paths.Decided in review
layout-filefeature while the format is v1; make it the default once it has a build identity (now in the header) and a compatibility policy, and keep the owned loader for users who load into their own memory.unsafe fncontract, which fits a deploy that writes a new file and renames it into place. Durable publication waits until a consumer needs it.