Skip to content

Add CI workflows, BERT fine-tuning example - #3

Merged
nicholasjng merged 13 commits into
masterfrom
ci-workflows
Nov 18, 2026
Merged

nicholasjng merged 13 commits into
masterfrom
ci-workflows

Conversation

@nicholasjng

Copy link
Copy Markdown
Owner

The workflows are adapted from mew's CI setup, CPython 3.15 on macOS and Linux only, with sccache to save time on compilation.

Keeps the uv-lock hook and the locked dependencies current.
Training code already holds framework dtypes, and spelling them as ducky's
short names was a needless translation. Specs normalize any accepted dtype to
its short name when they are created, so the SQL and stream code is unchanged
and an unknown dtype fails there rather than when the dataset is built. No
framework is imported unless one of its dtypes is passed.
Shows where ducky's loader stands in a realistic training loop: a small BERT
fine-tuned on AG News from tokenized Parquet shards, timed against Hugging Face
datasets behind a DataLoader. The loader alone is about 6x faster, but training
is model-bound with either.
@nicholasjng nicholasjng changed the title Add CI workflows, Add CI workflows, BERT fine-tuning example Nov 18, 2026
Mirrors mew, so CI's sanitizer jobs and local runs share one setup.
DUCKY_UBSAN and DUCKY_TSAN build like DUCKY_ASAN, and
scripts/sanitizer-pytest.sh replaces asan-pytest.sh for all three on macOS.
CMake otherwise picks its platform default, Makefiles, as mew's script avoids
too. An existing build/clangd configured with Makefiles has to be deleted once.
profile-gaps.sh ran benchmark files that the restructure for mew 0.3 removed,
and ft-udf-bench.py was a one-off free-threading measurement nothing runs.
A path dependency on a sibling checkout fails wherever there is none, such as
CI, where the uv-lock hook could not resolve it. The lockfile pins the master
commit it resolved, so `uv lock --upgrade-package mew-bench` moves to a newer
one.
PyPI's Linux torch wheels pull in several GB of CUDA libraries, which CI
machines without GPUs download for nothing. PyTorch's CPU index has the same
release, including Python 3.15 and free-threaded wheels. macOS keeps PyPI's
wheels, which carry no CUDA.
MLX's Linux wheels leave out libmlx.so, which comes with its cpu extra, so
importing mlx.core failed there and its tests errored instead of running. The
extra only applies on Linux, where it pulls in mlx-cpu.
Ports mew's CI without wheel publishing: lint through prek-action, the stub
check, tests on Linux and macOS for Python 3.15 and 3.15t, and ASAN, UBSAN and
TSAN runs. Every job but lint builds DuckDB from source, so a shared action
checks out the patched submodule with picky, installs the libcurl headers that
DuckDB's httpfs extension needs on Linux, and caches objects with sccache.
Many comments restated their code, and the prose leaned on semicolons.
Comments now give the reason, contracts such as lock and GIL requirements
stay, and stale notes go: a removed chunk buffer pool, delivered gaps in
notes/capi-gaps.md, finished roadmap items.
scikit-build-core already builds Release and reads the CMake minimum from
CMakeLists.txt, which also exports the compilation database, and DuckDB
already leaves its benchmarks off. The sanitizer overrides pass their CMake
flags, so CMake no longer reads the variables from the environment as well.
Package builds take scikit-build-core from the build-system requirements, so
the build group need not install it.
@nicholasjng
nicholasjng merged commit 94944e2 into master Nov 18, 2026
7 checks passed
@nicholasjng
nicholasjng deleted the ci-workflows branch November 18, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant