Skip to content

README.md

Latest commit

 

History

History
51 lines (36 loc) · 4.81 KB

README.md

File metadata and controls

51 lines (36 loc) · 4.81 KB
Raw

ref/ — reference projects

This directory holds git submodules of external projects we study while building beastdb (KV engines, compaction designs, Lean examples, etc.).

Rules

  • Not product source. Lean packages must not depend on ref/ as implementation.
  • Any language is fine here. First-party language purity rules do not apply under ref/.
  • Namecheck skips ref/. Upstream casing and naming are left alone.
  • Prefer submodules (pinned commits) over vendoring copies.
  • Flake productSrc filter drops the top-level ref directory (see flake.nix).
  • Measure-only exception: Nix measure packages may build and link ref/<name> with first-party drivers under bench/ and beastdb-lib (example: beastdb-ycsbc / YCSB-C). Not for Lean product or ffi/ product crates. See AGENTS.md.

Inventory

Short name Upstream URL Pinned commit Purpose of study What beastdb does not import
lmdb https://github.com/LMDB/lmdb.git 704dc7028587983ce6c2a19018d22e85f2d9b8ff SWMR mmap B+tree baseline; freelist/COW pages; multi-process lock lore. Aligns with G7.2 microbench class. No link of liblmdb into product Lean; no env/txn API; Nix beastdb-lmdb-bench is a separate generated C harness, not built from this tree
leveldb https://github.com/google/leveldb.git 7ee830d02b623e8ffe0b95d59a74db1e58da04c5 LSM / memtable / SSTable / leveled compaction contrast to append-only + Levels No SSTable format, bloom filters, or C++ memtable; no LevelDB API; product does not vendor or compile this tree
breccia https://github.com/petertodd/breccia.git 1845f81ea0b2f0a7cc496e9fc56af3a454bda13e Peter Todd append-only blob store: mark words, discoverable offsets, mmap-compatible verbatim blobs, binary search on ordered blobs (DESIGN.md) — compaction/layout inspiration (not a KV API) No Rust crate link; no breccia file format in product; no chattr +a requirement; study only
heed https://github.com/meilisearch/heed.git 86cd1f681953cd5f6870706f6139b851e975975e (tag v0.22.1) Meilisearch heed monorepo (crates heed, heed3, heed-types, heed-traits, lmdb-master*-sys): typed env/db/txn ergonomics, codecs, cookbook, and test suite as a requirements catalog for the first-party Rust crate at ffi/rust/beastdb/ (R17 closed Phases 3–5: thin v0 + selective T0/T1; heed3 tests zero; map docs/HEED-MAP.md) No link of heed / LMDB sys crates into product Lean or into ffi/rust/beastdb as a dependency; not a drop-in LMDB/lmdb.h shim; not full upstream suite as a product gate; heed3 encryption-at-rest is contrast only
ycsb-c https://github.com/basicthinker/YCSB-C.git 6475c7a5c3c7b229224ce5ca3332a58c994b8a09 (master pin) Yahoo! Cloud Serving Benchmark in C++ (YCSB-C): CoreWorkload A–F + pluggable DB drivers Lean/ffi do not depend on this tree. Measure link: beastdb-ycsbc builds upstream + first-party bench/ycsb driver (-db beastdb). Also ycsb-c-ref (TBB baselines) and first-party ycsb-bench approx harness

Nested LevelDB deps (optional): ref/leveldb/third_party/* (googletest, benchmark, …) may be empty after a normal shallow submodule init. They are not required for prior-art architecture reading (include/leveldb/, doc/, db/). Initialize recursively only if you build LevelDB’s own tests/benches.

heed pin: whole Meilisearch monorepo at tag v0.22.1 (crates.io max stable heed / heed3 0.22.1 as of pin date). Nested LMDB sys trees under that monorepo are part of the upstream checkout for reading; recursive init of further nested deps is optional (API/docs triage and Phase 2 study extract do not require building the full upstream suite).

Not a submodule (family note only): RocksDB — industrial LevelDB descendant; discussed in docs/PRIOR-ART.md without pinning the large repo.

Full comparison writeups (architecture, concurrency, durability, lessons adopted / not adopted, non-claims): docs/PRIOR-ART.md.

Add a submodule

git submodule add <repository-url> ref/<short-name>
# optional shallow:
git submodule add --depth 1 <repository-url> ref/<short-name>
git submodule update --init --recursive

Record the addition in CHANGELOG.md (what was added and why) and update the inventory table above plus docs/PRIOR-ART.md.

Clone

git clone --recurse-submodules <beastdb-url>
# existing clone:
git submodule update --init --recursive

After clone without --recurse-submodules, ref/lmdb, ref/leveldb, ref/breccia, ref/heed, and ref/ycsb-c appear empty until git submodule update --init --recursive.