Support for Databricks deletion vectors - #2271
Conversation
Databricks writes the spec-aligned deletion-vector-v1 envelope in a Delta container, so Iceberg reads failed on Puffin magic. Decode that slice from content_offset when the object is not a Puffin file. Co-authored-by: Cursor <cursoragent@cursor.com>
Spark Iceberg DELETE writes Puffin, so convert those objects to the Databricks UniForm layout before the ClickHouse read. Co-authored-by: Cursor <cursoragent@cursor.com>
…ctors Spark Iceberg DELETE writes Puffin; this customer table stores DVs as version-1 `.bin` objects with `content_offset = 1`. Co-authored-by: Cursor <cursoragent@cursor.com>
The root gitignore drops `*.bin`, so the previous commit omitted the object the test actually reads. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d8e94ed1bb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (isDeletionVectorV1Envelope(envelope, content_size_in_bytes)) | ||
| return IcebergDeletionVectorContainer::SliceOnly; |
There was a problem hiding this comment.
When a Puffin header is damaged—or a .bin object uses an unknown version—this accepts any valid envelope found at the manifest offset as SliceOnly. That skips bindDeletionVectorBlob, including the footer checks for the referenced data file, fields, and cardinality, so malformed input can silently filter rows instead of being rejected. Require either the known Delta 0x01 container layout or an exact bare-envelope layout before taking this path.
AGENTS.md reference: AGENTS.md:L153-L153
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I did not find strong description what can and can't be in non-puffin DV files. So prefer to accept all, until report about specific files that must be declined.
1 participant
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Support for Databricks deletion vectors
Documentation entry for user-facing changes
Databricks put deletion vector into .bin file instead of puffin files.
CI/CD Options
Exclude tests:
Regression jobs to run: