Repository navigation
[memory] load, parse: a categorical model loads in under half the heap, and the loaded model keeps no category lists or spare capacity - #65
Conversation
27af9ee to
2496558
Compare
category_slots collected every (code, chain, rank) triple of a feature, then counting-sorted them into a second vector before merging each code's ranks by word into the exceptions. A feature with many listed categories held both at once: for a large model with many categorical splits, most of the load's heap peak. The exceptions table also grew by doubling, so the loaded model kept up to half of it unused. A code's ranks arrive in word order, so a first pass over the chains now counts each code's words, its exceptions, and a second writes and merges them in place in a table of exactly that size. The cap is checked before the table grows rather than after, the prefix sums are checked, and the tables are unchanged.
Every categorical feature's heavy sets stay alive from the first pass of bin_map to the second, so their codes and starts are now trimmed, once the per-side lists they were taken from are freed, rather than kept at the capacity that doubling left. The split sides are recomputed from the feature's splits rather than stored, 16 bytes each. The first pass's distinct categories are deduplicated as they grow rather than collected from every split's list first, and that scratch is freed before the covers. The splits' words are allocated for the second pass, which writes them, rather than before the first.
The decoders interned category lists in a map keyed by a copy of each distinct list, so a model's lists were held twice while it decoded, the second time in one small allocation per list: a large part of the decoders' heap for a model with many categorical splits. The intern now maps each list's per-load hash to its offset in the pool and compares the pool's list before sharing it. A different list with the same hash is stored again, unshared, as with interning disabled. The map's RandomState hashes the lists as well as indexing their hashes, with keys drawn per load, so neither the fingerprints nor the lookup have a fixed hash a model can collide under. Reusing the map's hasher needs no second seed or hash implementation. Tests check the per-load fingerprints, a collision's pool comparison, and loading lists that collided under the old fixed hash.
A loaded model kept every categorical split's list, as many bytes as the model's category entries, though production tests categorical splits through the layout's rank tables only: it always precomputes masks by the sweep and runs passes without a scan, so the lists' readers, research's scans and brute-force masks, and the tests' parsed-node walks, never run there. Production now frees the lists after the build; research and the tests keep them; Forest's Debug shows their bytes, zero in production.
heavy_sets copied every split side's list into a vector of codes, then the distinct sets' lists into another, so a feature with many categorical splits held its lists twice while its sets were built, and kept a list once for each of its sets: listed and complemented, with and without missing. That set the load's heap peak for a large model with many categorical splits. The sides are now sorted and deduplicated by their decoded lists, compared as their codes would compare (the codebook keeps the categories' order, and missing's code is below every category's), and each distinct list's codes are held once, after missing's, so that its sets are ranges of them: Sets takes a range for each set rather than adjacent starts. The cap, the covers' order and the sets themselves are unchanged, so the layout is too.
Building a categorical feature's heavy sets looked every listed category's code up by binary search in the feature's codebook, and the first pass found the feature's distinct categories by sorting every split's list together: most of the load's time for a model with many categorical splits. Where a feature's categories are dense enough, at most eight times the entries they replace and 8 MiB, the build now takes codes from a table by category and collects the distinct categories in bits up to the largest; otherwise it searches and sorts as before. Neither table outlives the feature's first pass, and the layout is unchanged. The fixtures' small categories take the tables, and a feature listing category u32::MAX the search.
2496558 to
2f9df0a
Compare
ukaratay
left a comment
There was a problem hiding this comment.
Approving. The in-place exception write and the hash intern check out. A hash hit compares the pool's list before sharing it, so a collision costs sharing, not correctness. Freeing the lists in production is safe: their only readers are behind ABLATE (kernel.rs:88) or in the research scan.
One ask: which commits carry the 55% lower peak and the 26% faster from_reader? If any of them lands within noise, I'd drop it rather than carry its code path.
|
|
||
| /// A table by category, or the distinct categories' bits, is used only when it has | ||
| /// at most this many entries per list entry it replaces; else a search and a sort. | ||
| const DENSE_SPREAD: usize = 8; |
There was a problem hiding this comment.
Was 8 measured or picked? This adds a second path for codes and for the distinct categories. If the dense table isn't a visible share of the load-time win, I'd keep only the search and the sort.
There was a problem hiding this comment.
Picked, not swept. The table is most of the load-time win, though: commit 6 is −33.6% in every round (−35.5% to −30.4%), and without it this PR loads 8.7% slower than 3.2.2. On the private model, every categorical feature's largest category is under three times its distinct categories, so any factor from three up takes the same path there. Eight leaves room for sparser features, and the table stays under 8 MiB.
The unreleased section records what the load-memory changes save and how, and the loading docs say what a model with categorical splits holds while it loads and once loaded.
2f9df0a to
81e1b12
Compare
|
Measured per commit on the private model: the heap figures are exact (dhat), and load time is
The heap columns are shares of 3.2.2's peak and loaded heap. The 55% lower peak is commits 1, 2 and 5. The 26% faster load is commits 3 and 6, net of 2 and 5: those two cost load time but take a quarter off 3.2.2's peak. Commit 7 (skipping the sort of lists that already ascend) was within noise: +1.1% (−2.2% to +4.3%), and −1.5% (−3.7% to +4.3%) applied to commit 5 alone. I've dropped it. Without it, this PR loads in 72% of 3.2.2's time (rounds 70% to 75%). The description has the table. |
Motivation
Loading a model with many categorical splits took a heap several times the loaded model's, and the loaded model kept data that production never reads. On a large private model with many categorical splits, 3.2.2's heap peaked at 2.4 times the loaded model when loading from a file or a reader, and the loaded model held twice what this PR leaves it. The peak was set by the categorical build's intermediates, mostly copies of the category lists:
category_slotsheld every(code, chain, rank)triple, then a counting-sorted copy, before merging each code's ranks by word into the exceptions. The exceptions table grew by doubling, so the loaded model kept up to half of it unused.heavy_setscopied every split side's list as codes, then each distinct set's list again, so a list was held once for each of its sets: listed and complemented, with and without missing.Design
load: write the rank tables' exceptions in place, counted exactly. A code's ranks arrive in word order, so a first pass over the chains counts each code's words, its exceptions, and a second writes and merges them in place, in a table of exactly that size. The cap is checked before the table grows, and the prefix sums are checked.load: free the categorical build's scratch sooner. The heavy sets, which live from the first pass to the second, are trimmed; split sides are recomputed rather than stored (16 bytes each); the first pass's distinct categories are deduplicated as they grow; the splits' words are allocated for the pass that writes them.parse: intern category lists by hash and pool offset. The intern maps a list's hash to its offset in the pool and compares the pool's list before sharing it; a different list with the same hash is stored again, unshared, as with interning disabled. The hash is the map's ownRandomState, keyed per load as in 3.2.2, so a model cannot force collisions; a test loads lists that collided under the old fixed hash.load: free the category lists once loaded, in production. Production always precomputes masks by the sweep and runs passes without a scan, so the lists' readers (research's scans and brute-force masks, the tests' parsed-node walks) never run there. Research and the tests keep the lists;Forest'sDebugshows their bytes, zero in production.load: hold each category list of a feature's heavy sets once. Split sides are sorted and deduplicated by their decoded lists, compared as their codes would compare (the codebook keeps the categories' order, and missing's code is below every category's), and each distinct list's codes are held once, after missing's;Setstakes a range per set rather than adjacent starts, so a list's sets share it. The cap, the covers' order and the sets are unchanged.load: look categories up by table where a feature's are dense. Where a feature's categories are at most eight times as many as the entries they replace, and 8 MiB, codes come from a table by category and the distinct categories from bits; otherwise the search and the sort as before. The eight is picked, not swept: on the private model every categorical feature's largest category is under three times its distinct categories, so any factor from three up takes the same path there.changelog, docs.Not done, measured and dropped: counting every feature's exceptions before one allocation (a synthetic model with 200 to 1,000 categorical features loads as fast with the per-feature exact reservation, and the global count keeps every feature's heavy sets alive longer), freeing the category pool inside the build's first pass, skipping the pool's repacking in production (neither lowers the peak, which is in the first pass), and skipping the sort of category lists that already ascend (load time within noise: +1.1%, rounds −2.2% to +4.3%).
Evidence
On the private model, against 3.2.2, whose load memory and outputs equal 3.2.1's (dhat-rs requested bytes; malloc zone statistics; load times on an Apple M4 Pro, median of six rotated rounds of three loads, with the rounds' range):
Forest::loadorfrom_readerfrom_byteswith the file in memoryfrom_readerPer commit: the share of 3.2.2's
from_readerheap peak and loaded heap that each removes (exact), and its load time against the commit before it:Commits 2 and 5 cost load time but take a quarter off 3.2.2's peak; commit 6 more than pays for them.
layout_statsare equal (chains, chain slots, rank-table bytes, packing). Outputs are bitwise equal to 3.2.2's on synthetic rows (grouped and single) and on generated requests (each candidate, and the requests as served bypredict_fixed). Request latency at 1 candidate is unchanged (p50 101% of 3.2.2's, rounds within ±3%).cargo test --release --features researchpasses.cargo clippywith-D warningsfor the workspace, withtreewalker-bench/research, and for the library alone (-p treewalker-gbdt --lib, research off, which is the only build that compiles the production-only drop in commit 4); every commit passes the last on its own.cargo fmt --check.research(its self dev-dependency), so commit 4's production path is exercised only by a production build: a separate harness crate depending on this branch, which produced the outputs above.