Skip to content

benchmarks.md

Latest commit

 

History

History
310 lines (220 loc) · 15.5 KB

benchmarks.md

File metadata and controls

310 lines (220 loc) · 15.5 KB
Raw

Benchmarks

Running benchmarks

All benchmarks use Criterion.rs and live in src/simlin-engine/benches/.

# Run all benchmarks
cargo bench -p simlin-engine

# Run a specific benchmark suite
cargo bench -p simlin-engine --bench compiler
cargo bench -p simlin-engine --bench simulation
cargo bench -p simlin-engine --bench array_ops

# Run a specific benchmark group within a suite
cargo bench -p simlin-engine --bench compiler -- parse_mdl
cargo bench -p simlin-engine --bench compiler -- bytecode_compile/clearn

Criterion saves results in target/criterion/ and generates HTML reports in target/criterion/<group>/report/index.html.

Benchmark suites

Suite File What it measures
compiler benches/compiler.rs End-to-end compiler pipeline on real models (WRLD3, C-LEARN)
simulation benches/simulation.rs VM execution, slider interaction, compilation of synthetic models
array_ops benches/array_ops.rs Array sum, element-wise add, broadcasting, multi-ref

compiler benchmarks

The compiler suite measures each stage of the compilation pipeline independently using large, real-world models checked in under test/:

  • parse_mdl — MDL text to datamodel::Project (lexing, parsing, conversion)
  • project_build — datamodel::Project to engine Project (unit inference, dependency resolution, topological sort)
  • bytecode_compile — engine Project to CompiledSimulation (bytecode generation)
  • full_pipeline — all stages end-to-end

Models used:

  • wrld3 — World3 model (151 KB, ~3,800 lines), a classic system dynamics model
  • clearn — C-LEARN climate model (1.4 MB, ~53,000 lines), a stress test for the compiler

C-LEARN currently uses builtins that are not yet implemented in the bytecode compiler, so it is automatically skipped for bytecode_compile and full_pipeline. It still participates in parse_mdl and project_build, which are the most allocation-heavy stages.

Node VM-vs-wasm eval benchmark

@simlin/engine can run a model on two backends: the libsimlin VM or a compiled WebAssembly blob. This benchmark compares their simulation (eval) time through the public Model.simulate({}, { engine, enableLtm }) API, on fishbanks, WORLD3, and C-LEARN.

It is an rstest test gated behind RUN_BENCH so it stays out of the default pnpm test (a full C-LEARN run on both engines exceeds the per-test time budget):

# Run all three models on both engines
RUN_BENCH=1 pnpm -C src/engine exec rstest run backend-bench

# Subset the models (comma-separated: fishbanks, wrld3, clearn)
RUN_BENCH=1 BENCH_MODELS=fishbanks,wrld3 pnpm -C src/engine exec rstest run backend-bench

# Exercise Node/V8 acceptance and numerical parity on a large LTM model
RUN_BENCH=1 BENCH_MODELS=clearn BENCH_LTM=on pnpm -C src/engine exec rstest run backend-bench

# Compare both LTM settings (off is the default; choices: off, on, both)
RUN_BENCH=1 BENCH_LTM=both pnpm -C src/engine exec rstest run backend-bench

Use the system Node v24 and build shipping artifacts with env -u DISABLE_WASM_OPT bash src/engine/build.sh before measuring performance. Pre-commit stages a faster gate build whose timings do not represent shipping. The benchmark runs native Node/V8; it does not establish browser UI latency.

It prints a markdown table of the warm median eval time per engine and LTM setting plus the VM/wasm ratio.

What it measures, and what it deliberately excludes:

  • Eval only. The Sim for each (model, engine) is built once in untimed setup; for wasm that one-time cost is the blob compile and instantiate. Each measured iteration is a reset() (also untimed) followed by a timed runToEnd(). Result extraction (getRun/getSeries) is not timed.
  • Median over an explicit warmup. A discard-only warmup runs first, then the harness collects timings adaptively (until a max iteration count or a per-model wall-clock budget) and reports the median. The pure stats/harness lives in src/engine/tests/bench-stats.ts and is always-on unit-tested.
  • Cross-checked before trusted. Before timing, the benchmark runs each model and selected LTM setting on both engines and compares every public variable series within the engine's tolerance. A generated module that V8 rejects fails the test. This checks public trajectories; link-score analysis has separate tests.

Absolute numbers include the async public-API overhead, so the VM/wasm ratio is the figure to compare across runs.

The Rust counterpart is src/simlin-engine/examples/backend_bench.rs, which uses the same eval-vs-eval methodology and median statistic against the lower-level Vm/wasm interfaces.

Results are reported in the PR or chat, not committed: the harness is regenerable, but checked-in numbers go stale and mislead. Do not add a results file.

Node allocator benchmark for the wasm bundle

src/engine/bench/clearn-alloc.mjs compares two or more builds of the libsimlin wasm bundle on the full public-API pipeline for C-LEARN v77 -- Project.openVensim (parse + salsa sync), Model.simulate (the salsa compile plus Vm::new), Sim.runToEnd, every series, the LTM link scores, and dispose -- and prints median and min per stage plus the peak memory.size. It exists to compare global allocators: the compile stage makes tens of millions of small, short-lived allocations on this model, so the allocator the bundle links (src/libsimlin/src/lib.rs) is a first-order term in its time, and a synthetic allocation loop would not show how that plays out on the real allocation mix.

# Build the bundle to compare against on its branch and copy it out of
# src/engine/core/ first: build.sh overwrites the staged artifacts.
node --expose-gc src/engine/bench/clearn-alloc.mjs \
    dlmalloc=/path/to/main/libsimlin-browser.wasm \
    wasmalloc=src/engine/core/libsimlin-browser.wasm

# LTM off only, 20 iterations, exact memory.grow counts (needs wasm-tools on PATH)
node --expose-gc src/engine/bench/clearn-alloc.mjs --ltm off --iters 20 --count-grows \
    a=/path/to/a.wasm b=/path/to/b.wasm

The bundles are interleaved (A, B, A, B, ...) so machine drift is shared rather than attributed to whichever ran last. Each iteration runs on a fresh instance of a module compiled once per bundle: every iteration starts from a cold heap, the state a page load leaves the allocator in, and no iteration inherits fragmentation from the one before, while V8 keeps its optimized code for the shared module, so the warm-up iterations warm the JIT and only the JIT. Instantiation is not timed. Run it under both node 22 and node 24 (V8 12 and 13), with --expose-gc so it can collect between iterations and pinned with taskset to reduce drift; like the eval benchmark, its results belong in the PR or chat, never in a committed file.

Editing latency at the libsimlin FFI

src/libsimlin/examples/editing_latency.rs times, call by call, what a native host pays to edit a diagram through libsimlin: a hit test for every pointer move and press, the tap, drag and nudge planners, the scene a host redraws after an edit, how long an equation edit holds the project's datamodel lock, which is how long a hit test or a press from another thread waits meanwhile, what the simulation after it takes (it compiles with the datamodel released), and what a hover at display rate waits while edits land on another thread.

# World3 (864 drawn elements)
cargo run --release -p simlin --example editing_latency -- test/metasd/WRLD3-03/wrld3-03.mdl

# C-LEARN (4059 drawn elements); fewer samples keep the slow scenarios short
cargo run --release -p simlin --example editing_latency -- "test/xmutil_test_models/C-LEARN v77 for Vensim.mdl" --samples 40

It prints the p50, p90, p99, max and mean of every scenario. It keeps the system allocator, which a host runs on when it builds libsimlin without the mimalloc feature, and its results belong in the PR or chat, never in a committed file.

To see where the time goes, build it with symbols and record it under a sampling profiler, as Profiling describes. The bench profile is release with debug info and no stripping, and cargo writes its examples to target/release/examples/:

cargo build --profile bench -p simlin --example editing_latency
perf record -g -- target/release/examples/editing_latency test/metasd/WRLD3-03/wrld3-03.mdl --samples 600
perf report

Memory at the libsimlin FFI

src/libsimlin/examples/memory_census.rs measures what a native host holds while it works on a model, step by step through the entry points a host calls. A counting global allocator over the system allocator reports the heap each step leaves live and the peak reached during it, and a retained size is attributed by dropping one owner at a time (a run, an undo copy, the compiled project) and reading how far the live count falls.

# Every workload in one process
cargo run --release -p simlin --example memory_census -- "test/xmutil_test_models/C-LEARN v77 for Vensim.mdl"

# One workload per process, beside the process's own peak (GNU time on Linux)
/usr/bin/time -v target/release/examples/memory_census test/metasd/WRLD3-03/wrld3-03.mdl --workload undo --edits 50

The workloads are open, simulate (compile, run and read the series a diagram's sparklines draw), diagnostics, loops (structural loops, a Loops That Matter run and its links, loop discovery), undo (--edits edits landed the way a host lands them, each copied for the undo history, then all undone; --edit picks what each changes: equation, the default, for the first constant's equation, last-equation for the last constant's, move for an aux nudged by a unit, and rename for the last constant renamed and back, and synced-equation for the first constant's equation followed by a diagram sync, since a copy that shares what its edit leaves alone costs only what the edit reaches) and draft (an equation draft previewed on a scratch copy). The live count covers the heap alone: the allocator's own retention and anything outside the heap show only in the process's peak. Like the other harnesses, its results belong in the PR or chat, never in a committed file.

Profiling

Build a benchmark binary for profiling

Criterion benchmark binaries are standalone executables. To build one without running it:

cargo bench -p simlin-engine --bench compiler --no-run

The binary will be in target/release/deps/compiler-<hash>. Find the exact path with:

cargo bench -p simlin-engine --bench compiler --no-run 2>&1 | grep -o 'target/[^ ]*'

CPU profiling with perf (Linux)

# Record a profile (run a single benchmark to keep the profile focused)
perf record -g -- target/release/deps/compiler-* --bench parse_mdl/clearn

# View the report
perf report

# Generate a flamegraph (requires https://github.com/brendangregg/FlameGraph)
perf script | stackcollapse-perf.pl | flamegraph.pl > flamegraph.svg

Alternatively, use cargo-flamegraph:

cargo install flamegraph
cargo flamegraph --bench compiler -- --bench parse_mdl/clearn

CPU profiling with callgrind (valgrind)

Callgrind provides instruction-level profiling and call graphs. It runs the program under emulation, so it's slower but gives precise, deterministic results unaffected by system load.

# Profile a specific benchmark
valgrind --tool=callgrind --callgrind-out-file=callgrind.out \
    target/release/deps/compiler-* --bench parse_mdl/clearn

# Analyze results
callgrind_annotate callgrind.out
# Or use the graphical viewer:
kcachegrind callgrind.out

A release build inlines small hot code into its callers, so a per-function listing has no row for "all hashing" or "all refcounting". scripts/callgrind-by-file.py sums a profile by the source file each instruction came from, which sees through inlining, and lists the functions paying for one file; its header has the commands for profiling one compile of a model with examples/clearn_profile.rs.

Allocation profiling with DHAT (valgrind)

DHAT tracks every allocation: size, lifetime, and access patterns. Useful for finding unnecessary allocations or short-lived temporaries.

valgrind --tool=dhat \
    target/release/deps/compiler-* --bench parse_mdl/clearn

# Opens an interactive viewer in Firefox/Chrome
# The output file is dhat-out-<pid>.txt

View results at https://valgrind.org/docs/manual/dh-manual.html or use dh_view.html from the valgrind distribution.

Allocation profiling with heaptrack

heaptrack is lighter-weight than DHAT and produces flamegraphs of allocation sites.

heaptrack target/release/deps/compiler-* --bench parse_mdl/clearn

# Analyze (TUI)
heaptrack_print heaptrack.compiler-*.zst

# Analyze (GUI)
heaptrack_gui heaptrack.compiler-*.zst

CPU + allocation profiling with gperftools

gperftools provides both CPU and heap profiling via LD_PRELOAD.

# CPU profile
LD_PRELOAD=/usr/lib/libprofiler.so CPUPROFILE=cpu.prof \
    target/release/deps/compiler-* --bench parse_mdl/clearn

# Heap profile
LD_PRELOAD=/usr/lib/libtcmalloc.so HEAPPROFILE=heap.prof \
    target/release/deps/compiler-* --bench parse_mdl/clearn

# View (requires google-pprof or go tool pprof)
pprof --web target/release/deps/compiler-* cpu.prof
pprof --web target/release/deps/compiler-* heap.prof.0001.heap

Allocation counting with the global allocator

For tracking allocation counts and bytes in CI or quick checks, Rust's global allocator can be overridden. This isn't wired up in the benchmarks by default, but you can use it in a one-off test:

use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::atomic::{AtomicUsize, Ordering};

struct CountingAlloc;
static ALLOC_COUNT: AtomicUsize = AtomicUsize::new(0);
static ALLOC_BYTES: AtomicUsize = AtomicUsize::new(0);

unsafe impl GlobalAlloc for CountingAlloc {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        ALLOC_COUNT.fetch_add(1, Ordering::Relaxed);
        ALLOC_BYTES.fetch_add(layout.size(), Ordering::Relaxed);
        unsafe { System.alloc(layout) }
    }
    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        unsafe { System.dealloc(ptr, layout) }
    }
}

#[global_allocator]
static A: CountingAlloc = CountingAlloc;

Comparing results

Criterion automatically compares against the previous run and reports statistical significance. To save an explicit baseline for later comparison:

# Save a baseline
cargo bench -p simlin-engine --bench compiler -- --save-baseline before-change

# Run again after making changes
cargo bench -p simlin-engine --bench compiler -- --baseline before-change

HTML comparison reports are generated in target/criterion/<group>/report/index.html.

Tips

  • Use -- <filter> to run only matching benchmarks (e.g., -- parse_mdl/clearn).
  • Build in release mode (cargo bench does this automatically) for representative profiles.
  • The benchmarks configure measurement_time (10-15s) in code, which gives external profilers enough sustained CPU activity to collect meaningful data.
  • When using valgrind tools, the 10-30x slowdown means you may want to reduce measurement_time in the benchmark source or filter to a single benchmark.
  • For allocation analysis, compare total allocation counts before and after a change rather than absolute numbers — the counts are deterministic across runs.