Repository navigation
add more examples, fix ducky v2 bindings bugs - #5
Merged
Merged
Conversation
…audited defects Loading a dataset stacked and gathered full-size copies because a result could only be drained column by column. to_numpy_folds copies each row once, into the array of its fold. Lists, dicts and NumPy arrays now bind as LIST and STRUCT, so an embedding can be a query parameter. The fixes come from an audit and were reproduced on this base first. Two crashed or leaked (a table function whose factory returns a list, a connection kept alive by its own UDF after execute()); the others returned wrong data without an error, such as a retried to_numpy() after a failure, columns dropped for duplicate names, and an aware time losing its offset.
…n one copy Splits by row hash cannot keep a patient's rows together or keep a rare label's share in every fold. Ranking rows within strata does both. dataset() now reads through to_numpy_folds, which cuts load time by a third to a half on 5M rows and, without a split, peak memory by 45%. BREAKING CHANGE: split(seed=...) without key= now hashes the seed with the row number. Adding it only shifted the same split by seed rows, so folds for a given seed change.
…or JAX and MLX The Titanic example fits in a few kilobytes and one linear layer, so it shows little of DuckDB's share of the work on large data. These scale it up in three directions: rows and network size, math that DuckDB reduces before the array library sees it, and a text pipeline with vector search.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Also improves throughput and cuts memory usage on many workloads.