Repository navigation
Add CI workflows, BERT fine-tuning example - #3
Merged
Merged
Conversation
Keeps the uv-lock hook and the locked dependencies current.
Training code already holds framework dtypes, and spelling them as ducky's short names was a needless translation. Specs normalize any accepted dtype to its short name when they are created, so the SQL and stream code is unchanged and an unknown dtype fails there rather than when the dataset is built. No framework is imported unless one of its dtypes is passed.
Shows where ducky's loader stands in a realistic training loop: a small BERT fine-tuned on AG News from tokenized Parquet shards, timed against Hugging Face datasets behind a DataLoader. The loader alone is about 6x faster, but training is model-bound with either.
nicholasjng
force-pushed
the
ci-workflows
branch
from
November 18, 2026 15:03
df9f92d to
dbe3390
Compare
nicholasjng
force-pushed
the
ci-workflows
branch
from
November 18, 2026 15:25
dbe3390 to
1b46305
Compare
Mirrors mew, so CI's sanitizer jobs and local runs share one setup. DUCKY_UBSAN and DUCKY_TSAN build like DUCKY_ASAN, and scripts/sanitizer-pytest.sh replaces asan-pytest.sh for all three on macOS.
CMake otherwise picks its platform default, Makefiles, as mew's script avoids too. An existing build/clangd configured with Makefiles has to be deleted once.
profile-gaps.sh ran benchmark files that the restructure for mew 0.3 removed, and ft-udf-bench.py was a one-off free-threading measurement nothing runs.
A path dependency on a sibling checkout fails wherever there is none, such as CI, where the uv-lock hook could not resolve it. The lockfile pins the master commit it resolved, so `uv lock --upgrade-package mew-bench` moves to a newer one.
PyPI's Linux torch wheels pull in several GB of CUDA libraries, which CI machines without GPUs download for nothing. PyTorch's CPU index has the same release, including Python 3.15 and free-threaded wheels. macOS keeps PyPI's wheels, which carry no CUDA.
MLX's Linux wheels leave out libmlx.so, which comes with its cpu extra, so importing mlx.core failed there and its tests errored instead of running. The extra only applies on Linux, where it pulls in mlx-cpu.
Ports mew's CI without wheel publishing: lint through prek-action, the stub check, tests on Linux and macOS for Python 3.15 and 3.15t, and ASAN, UBSAN and TSAN runs. Every job but lint builds DuckDB from source, so a shared action checks out the patched submodule with picky, installs the libcurl headers that DuckDB's httpfs extension needs on Linux, and caches objects with sccache.
Many comments restated their code, and the prose leaned on semicolons. Comments now give the reason, contracts such as lock and GIL requirements stay, and stale notes go: a removed chunk buffer pool, delivered gaps in notes/capi-gaps.md, finished roadmap items.
scikit-build-core already builds Release and reads the CMake minimum from CMakeLists.txt, which also exports the compilation database, and DuckDB already leaves its benchmarks off. The sanitizer overrides pass their CMake flags, so CMake no longer reads the variables from the environment as well. Package builds take scikit-build-core from the build-system requirements, so the build group need not install it.
nicholasjng
force-pushed
the
ci-workflows
branch
from
November 18, 2026 15:44
1b46305 to
b668372
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The workflows are adapted from mew's CI setup, CPython 3.15 on macOS and Linux only, with sccache to save time on compilation.