docs: document DELETE and UPDATE for SQL users and table provider authors by michaelsembwever · Pull Request #24567 · apache/datafusion

michaelsembwever · 2026-08-21T20:31:51Z

Which issue does this PR close relate to?

Documentation follow-up for feat: Add DELETE/UPDATE hooks to TableProvider trait and to MemTable implementation #19142, which added the TableProvider row-level DML hooks and closed Support delete_from and update in TableProvider #16959. That PR shipped no documentation.
Related to UPDATE ...FROM bug #19950.

Rationale for this change

Since 52.0.0, DataFusion runs DELETE and UPDATE against a table whose provider implements TableProvider::delete_from() or TableProvider::update(), and the built-in in-memory table implements both. No page in the documentation says so.

A SQL user therefore cannot learn which tables accept the two statements, what a statement returns, or which forms fail. A provider author cannot learn what the planner passes to each hook, or what the hook must return.

Two current behaviours are surprising enough to warn about in the same pass:

A DELETE or an UPDATE whose WHERE clause holds an IN or an EXISTS subquery applies to all rows of the table. The optimizer rewrites the subquery into a LeftSemi Join, so extract_dml_filters() finds no predicate on the target table, and the provider reads the
empty filter list as "no WHERE clause".
```
> create table s1 as values (1), (2), (3);
> create table s2 as values (2);
> delete from s1 where column1 in (select column1 from s2);
-- count 3; s1 is now empty
```
EXPLAIN DELETE and EXPLAIN UPDATE execute the statement on an in-memory table. MemTable changes the rows inside the hook, and the physical planner calls the hook while it builds the plan.

Both behaviours need code fixes, which this PR does not attempt. Until then a reader needs the warning.

What changes are included in this PR?

docs/source/user-guide/sql/dml.md:

A DELETE section and an UPDATE section: syntax, the count result, three-valued logic, and examples.
A "Table support for DELETE and UPDATE" section: which table kinds support the statements, and the exact error text for a table that does not.
A "Limitations" section: the two warnings above, the ignored LIMIT on DELETE, and UPDATE ... FROM.

docs/source/library-user-guide/custom-table-providers.md:

A "Row-Level DML: DELETE and UPDATE" section: what the planner passes to each hook (split AND conjunctions, stripped table qualifiers, target-table predicates only), the single-row count return contract, the two semantic rules a provider must follow, a compiling example,
the clauses a hook never receives, and when the work happens.

No code changes.

Are these changes tested?

Yes.

cargo test --doc -p datafusion library_user_guide_custom_table_providers passes. The new example is a compiled doctest, not an ignore block.
./ci/scripts/doc_prettier_check.sh passes.
Every behavioural statement in the new text was checked against main with temporary sqllogictest cases, rather than read from the code alone: the ignored LIMIT; the pre-statement values in SET a = b, b = a; the error text for an external table and for a view; the scalar
subquery error; the IN and EXISTS all-rows result; and the EXPLAIN side effect. Those cases are not part of this PR, because the last two assert behaviour that should change.

Are there any user-facing changes?

Documentation only. No change to any API.

…hors PR apache#19142 added `TableProvider::delete_from()` and `TableProvider::update()`, and implemented both for `MemTable`, but added no documentation. Add a `DELETE` section and an `UPDATE` section to the SQL user guide, with the syntax, the result shape, which table kinds support the statements, and the current limitations. Add a "Row-Level DML" section to the custom table provider guide, covering what the planner passes to each hook, the `count` result contract, the semantic rules a provider must follow, and a compiling example. Two behaviours found while verifying the documentation are recorded as warnings, since users meet them today: - An `IN` or an `EXISTS` subquery in the `WHERE` clause makes the statement apply to all rows, because the optimizer rewrites the subquery into a join and the predicate never reaches the provider. - `EXPLAIN DELETE` and `EXPLAIN UPDATE` execute the statement on an in-memory table, because `MemTable` changes the rows inside the hook and the hook runs during physical planning. Assisted-by: Claude Code:claude-opus-5

codecov-commenter · 2026-08-21T20:59:24Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.41%. Comparing base (a6e2d3f) to head (e4f2a13).
⚠️ Report is 418 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24567      +/-   ##
==========================================
+ Coverage   81.36%   82.41%   +1.04%     
==========================================
  Files        1117     1138      +21     
  Lines      397872   435320   +37448     
  Branches   397872   435320   +37448     
==========================================
+ Hits       323725   358763   +35038     
+ Misses      55229    54847     -382     
- Partials    18918    21710    +2792

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb · 2026-08-24T19:58:29Z

Both behaviours need code fixes, which this PR does not attempt. Until then a reader needs the warning.

Are there tickets that cover these issues? I agree they sound serious

alamb · 2026-08-24T20:00:30Z

Specifically, I want to make sure the issues are tracked (ideally with a link in the docs as well) so that as we resolve them we also know to come and update the docs

michaelsembwever · 2026-08-25T09:11:45Z

Are there tickets that cover these issues?

@alamb , i've created them

A DELETE or an UPDATE with an IN or an EXISTS subquery in its WHERE clause changes every row #24654
EXPLAIN DELETE and EXPLAIN UPDATE execute the statement on an in-memory table #24656

with PRs attached to each.

michaelsembwever · 2026-09-03T16:47:12Z

@alamb , are we good for merging this now ?

martin-g

I think this PR adds useful information about some current limitations in DataFusion.
It would be nice if every limitation is accompanied with text similar to "This limitation is tracked at issue XYZ"
Once the limitation is implemented it would be easier to detect that the documentation is obsolete and be updated too.
There are opened PRs for some of the limitations already.

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

…ed limitation docs: document DELETE and UPDATE for SQL users and table provider authors PR apache#19142 added `TableProvider::delete_from()` and `TableProvider::update()`, and implemented both for `MemTable`, but added no documentation. Add a `DELETE` section and an `UPDATE` section to the SQL user guide, with the syntax, the result shape, which table kinds support the statements, and the current limitations. Add a "Row-Level DML" section to the custom table provider guide, covering what the planner passes to each hook, the `count` result contract, the semantic rules a provider must follow, and a compiling example. Two behaviours found while verifying the documentation are recorded as warnings, since users meet them today: - An `IN` or an `EXISTS` subquery in the `WHERE` clause makes the statement apply to all rows, because the optimizer rewrites the subquery into a join and the predicate never reaches the provider. - `EXPLAIN DELETE` and `EXPLAIN UPDATE` execute the statement on an in-memory table, because `MemTable` changes the rows inside the hook and the hook runs during physical planning. Every documented limitation ends with the issue that tracks it, in one phrasing a contributor can grep for, so a merged fix makes the obsolete paragraph easy to find: apache#24654 for the subquery cases, apache#24656 for `EXPLAIN`, apache#24998 for the ignored `LIMIT` on a `DELETE`, and apache#19950 for `UPDATE ... FROM`. Which tables support the statements is a capability rather than a defect, so those two lines cite nothing. Assisted-by: Claude Code:claude-opus-5

michaelsembwever · 2026-09-07T15:02:33Z

ideally with a link in the docs as well
…
I think this PR adds useful information about some current limitations in DataFusion.
It would be nice if every limitation is accompanied with text similar to "This limitation is tracked at issue [XYZ (https://github.com/apache/datafusion/issues/XYZ)"
Once the limitation is implemented it would be easier to detect that the documentation is obsolete and be updated too.

Every limitation now ends with "This limitation is tracked at issue NNNNN": #24654 for the subquery cases, #24656 for EXPLAIN, #24998 for the ignored LIMIT, and #19950 for UPDATE ... FROM. Three of the four have an open fix (#24657, ~~#24655~~ #25040, #25005), so whichever of those merges after this PR should drop the matching paragraph.

michaelsembwever · 2026-09-19T13:18:16Z

@alamb , is this good to merge now ?

for clarity sake, (with the issues and PRs that have spun out from this doc PR):

flowchart TD
    DOC["pr#24567: documentation and bug warnings"]
    DOC -. tracks .-> A["issue#24654: WHERE conditions lost"]
    DOC -. tracks .-> B["issue#24656: EXPLAIN changes data"]
    DOC -. tracks .-> C["issue#24998: DELETE ignores LIMIT"]

    A --> AF["pr#24657: protect provider calls"]
    B --> OLD["pr#24655: original execution-time fix"]
    OLD -->|superseded by| NEW["pr#25040: replacement execution-time fix"]
    C --> CF["pr#25005: reject DELETE LIMIT"]

alamb

Thank you @michaelsembwever and sorry for the delay in reviewing

I think this is a very nice addition -- I had several suggestions -- let me know what you think

alamb · 2026-09-20T13:46:02Z

+
+### What the Planner Passes to the Hooks
+
+`filters` holds the `WHERE` predicates as logical `Expr` values, after three transformations:


Would you be willing to move some of this documentation on to the TableProvider::delete_from and TableProvider::update methods themselves?

Here:

datafusion/datafusion/session/src/table.rs

Lines 360 to 380 in 3b16a3d

/// Delete rows matching the filter predicates.

///

/// Returns an [`ExecutionPlan`] producing a single row with `count` (UInt64).

/// Empty `filters` deletes all rows.

// Hand-written `#[async_trait]` expansion to reduce compile time. See

// <https://github.com/apache/datafusion/issues/13814#issuecomment-5292709677>

fn delete_from<'life0, 'life1, 'async_trait>(

&'life0 self,

_state: &'life1 dyn Session,

_filters: Vec<Expr>,

) -> BoxFuture<'async_trait, Result<Arc<dyn ExecutionPlan>>>

where

'life0: 'async_trait,

'life1: 'async_trait,

Self: 'async_trait,

{

Box::pin(ready(not_impl_err!(

"DELETE not supported for {} table",

self.table_type()

)))

}

That will result in them being available in https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableProvider.html as well as the source code so I think it will be more discoverable

Then the idea is that this library user guide can help provide a more user friendly overview / introduction

done in e4f2a13

alamb · 2026-09-20T13:47:57Z

+
+## Table support for DELETE and UPDATE
+
+The table provider does the work for `DELETE` and `UPDATE`. Support is therefore a property of each table:


I think this section and down has too much implementation detail and we don't have to spell out exactly how UPDATE / DELETE is implemented as part of the user guide (targeting SQL users). We could perhaps just say something like "not all table providers support UPDATE and DELETE" ?

done in e4f2a13

alamb · 2026-09-20T13:48:42Z

+
+### Limitations
+
+:::{warning}


I a not sure we need to spell out the deatils of the active bugs in the SQL reference manual. If you think it is valuable, we could list them, but let users follow the links if they want more details

done e4f2a13

alamb · 2026-09-20T13:49:31Z

 +-------+
 ```
+
+## DELETE


I recommend pulling the sql/dml.sql guide updates into their own PR for faster review / merging

done: #25553

docs: move row-level DML contracts into TableProvider API docs Keep the library guide introductory and split the SQL reference changes into a separate PR.

github-actions Bot added the documentation Improvements or additions to documentation label Aug 21, 2026

This was referenced Aug 25, 2026

A DELETE or an UPDATE with an IN or an EXISTS subquery in its WHERE clause changes every row #24654

Open

EXPLAIN DELETE and EXPLAIN UPDATE execute the statement on an in-memory table #24656

Open

martin-g reviewed Sep 7, 2026

View reviewed changes

Comment thread docs/source/user-guide/sql/dml.md Outdated

Comment thread docs/source/user-guide/sql/dml.md Outdated

Comment thread docs/source/user-guide/sql/dml.md Outdated

michaelsembwever and others added 4 commits September 7, 2026 14:23

SQUASH – Update docs/source/user-guide/sql/dml.md

4b5c030

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

SQUASH – Update docs/source/user-guide/sql/dml.md

462ce65

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

SQUASH – Update docs/source/user-guide/sql/dml.md

ca7e59b

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

alamb reviewed Sep 20, 2026

View reviewed changes

SQUASH – review comments

e4f2a13

docs: move row-level DML contracts into TableProvider API docs Keep the library guide introductory and split the SQL reference changes into a separate PR.

michaelsembwever force-pushed the mck/dml-delete-update-docs branch from 5c7006d to e4f2a13 Compare September 20, 2026 19:03

michaelsembwever mentioned this pull request Sep 20, 2026

docs: document DELETE and UPDATE in the SQL reference #25553

Open


		### What the Planner Passes to the Hooks

		`filters` holds the `WHERE` predicates as logical `Expr` values, after three transformations:

	/// Delete rows matching the filter predicates.
	///
	/// Returns an [`ExecutionPlan`] producing a single row with `count` (UInt64).
	/// Empty `filters` deletes all rows.
	// Hand-written `#[async_trait]` expansion to reduce compile time. See
	// <https://github.com/apache/datafusion/issues/13814#issuecomment-5292709677>
	fn delete_from<'life0, 'life1, 'async_trait>(
	&'life0 self,
	_state: &'life1 dyn Session,
	_filters: Vec<Expr>,
	) -> BoxFuture<'async_trait, Result<Arc<dyn ExecutionPlan>>>
	where
	'life0: 'async_trait,
	'life1: 'async_trait,
	Self: 'async_trait,
	{
	Box::pin(ready(not_impl_err!(
	"DELETE not supported for {} table",
	self.table_type()
	)))
	}


		## Table support for DELETE and UPDATE

		The table provider does the work for `DELETE` and `UPDATE`. Support is therefore a property of each table:

Conversation

michaelsembwever commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close relate to?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb commented Aug 24, 2026

Uh oh!

alamb commented Aug 24, 2026

Uh oh!

michaelsembwever commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelsembwever commented Sep 3, 2026

Uh oh!

martin-g left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelsembwever commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelsembwever commented Sep 19, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

michaelsembwever commented Aug 21, 2026 •

edited

Loading

codecov-commenter commented Aug 21, 2026 •

edited

Loading

michaelsembwever commented Aug 25, 2026 •

edited

Loading

michaelsembwever commented Sep 7, 2026 •

edited

Loading