Skip to content
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,36 @@
# Changelog

## Unreleased

Loading a model with many categorical splits takes much less memory, and so does
the loaded model. For a large model with many categorical splits, the heap's peak
while loading from a file or a reader is 55% lower than 3.2.1's, and the loaded
model's heap 52% lower, with loads 30% faster. The layout and the outputs are
unchanged.

### Memory

- The rank tables' exceptions are counted, then written in place in a table of
their exact size, so the build no longer holds every (code, chain, rank) triple
and a sorted copy, and the loaded model keeps no spare capacity. The cap on the
rank tables is checked before they grow.
- A categorical feature's heavy sets hold each distinct category list's codes once,
shared by its sets with and without missing, listed and complemented, rather
than every split side's list and every set's.
- The build frees its categorical scratch sooner: the heavy sets are trimmed, the
split sides recomputed rather than stored, the distinct categories deduplicated
as they grow, and the splits' words allocated for the pass that writes them.
- The decoders intern category lists by their hash and pool offset rather than by
a copy of each list.
- Production frees the category lists once a model is loaded: it tests categorical
splits through the rank tables only. Research and the tests keep them.

### Load time

- Where a categorical feature's categories are dense enough, the build looks their
codes up by table and collects the distinct categories in bits, rather than by
binary search and a sort of every split's list.

## 3.2.2

3.2.2 closes a denial of service in the loader found by a security review: a model
Expand Down
17 changes: 11 additions & 6 deletions docs/treelite-loading.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,8 +212,8 @@ including caches the system does not report;
replaces the measurement. Every group runs whole: groups of up to 1,024 rows take
masks of 32 to 1,024 bits, a power of two, and wider ones masks of their own width
in 128-bit units. A loaded model keeps the prediction layout,
16 bytes per node plus the bins and the leaf table; the parsed nodes are kept only
with the `research` feature.
16 bytes per node plus the bins and the leaf table; the parsed nodes and the
category lists are kept only with the `research` feature.

## Bounded import

Expand All @@ -222,9 +222,11 @@ file past one fails with `LoadError::Limit`, and trusted inputs may raise them.
They bound input, representation, and decoding work; they are not a promise about
peak process RSS. Large legitimate forests should use binary. The loader's hash
tables (the forest's distinct splits, a JSON tree's node IDs, interned category
lists) hash with keys drawn at random for each load, so a file cannot choose
thresholds or IDs that collide: decoding work stays linear in the input whatever
its values.
lists and the heavy sets' shared lists) hash with keys drawn at random for each
load, so a file cannot choose thresholds, IDs or lists that collide: decoding
work stays linear in the input whatever its values. Category-list fingerprints
use the intern map's random keys too; a hit is compared with the pool's list
before sharing it.

| Cap (`LoadOptions` field) | Default | Protects against |
|---|---:|---|
Expand Down Expand Up @@ -277,7 +279,10 @@ order, from the distinct splits' bins, codebooks and chains, and writes the leaf
tree order is the order of the trees' starts and leaf values, not of their nodes.
Peak memory is therefore the loaded model plus the split table (small for binned
models: thousands of splits against millions of nodes) and a reader's buffer of at
most 1 MiB or one tree's fields. For Expedia's 1,000-tree, depth-16 LightGBM model
most 1 MiB or one tree's fields. A model with categorical splits also holds its
category lists, each distinct one once, while it loads, and each categorical
feature's heavy sets and rank tables while its layout is built; production frees
the lists once loaded. For Expedia's 1,000-tree, depth-16 LightGBM model
(7.7 million nodes, a 636 MB file), `Forest::load` takes 0.27 s with a heap peak of
191 MB, against a loaded model of 186 MB. JSON keeps a
document in memory: simd-json's tape over the input, whose strings it borrows,
Expand Down
Loading
Loading