Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 69 additions & 4 deletions datafusion/session/src/table.rs
Original file line number Diff line number Diff line change
Expand Up @@ -359,8 +359,38 @@ pub trait TableProvider: Any + Debug + Sync + Send {

/// Delete rows matching the filter predicates.
///
/// Returns an [`ExecutionPlan`] producing a single row with `count` (UInt64).
/// Empty `filters` deletes all rows.
/// Returns an [`ExecutionPlan`] producing one row in a single non-null
/// `UInt64` column named `count`, containing the number of deleted rows.
/// The default implementation returns a "not implemented" error.
///
/// # Filters
///
/// `filters` contains logical [`Expr`] predicates on the target table.
/// The planner collects them from filters and pushed-down table scan
/// predicates, splits `AND` conjunctions, removes table qualifiers, and
/// deduplicates them. For example, `t.id = 1 AND t.value > 15` becomes
/// separate `id = 1` and `value > 15` expressions.
///
/// Delete a row only when every predicate evaluates to true. SQL
/// three-valued logic applies: a false or `NULL` predicate leaves the row
/// unchanged. Empty `filters` deletes all rows, as for `DELETE FROM t`
/// without a `WHERE` clause.
///
/// # Execution
///
/// This method is called during physical planning, including for `EXPLAIN`.
/// Perform mutations when the returned plan executes, rather than while
/// constructing it, so planning does not change the table.
///
/// # Limitations
///
/// The method receives no row limit ([#24998]). Subqueries in DML
/// expressions are not fully supported ([#24654]); in particular, a
/// subquery rewritten into a join can cause predicates to be lost before
/// this method is called.
///
/// [#24998]: https://github.com/apache/datafusion/issues/24998
/// [#24654]: https://github.com/apache/datafusion/issues/24654
// Hand-written `#[async_trait]` expansion to reduce compile time. See
// <https://github.com/apache/datafusion/issues/13814#issuecomment-5292709677>
fn delete_from<'life0, 'life1, 'async_trait>(
Expand All @@ -381,8 +411,43 @@ pub trait TableProvider: Any + Debug + Sync + Send {

/// Update rows matching the filter predicates.
///
/// Returns an [`ExecutionPlan`] producing a single row with `count` (UInt64).
/// Empty `filters` updates all rows.
/// Returns an [`ExecutionPlan`] producing one row in a single non-null
/// `UInt64` column named `count`, containing the number of affected rows.
/// The default implementation returns a "not implemented" error.
///
/// # Filters
///
/// `filters` follows the same conventions as [`Self::delete_from`]:
/// predicates are combined with `AND`, table qualifiers are removed, and
/// only rows for which every predicate is true are updated. A false or
/// `NULL` predicate leaves the row unchanged. Empty `filters` updates all
/// rows.
///
/// # Assignments
///
/// `assignments` contains `(column_name, Expr)` pairs from the `SET`
/// clause. The planner removes identity assignments and strips table
/// qualifiers from the expressions. Leave columns without an assignment
/// unchanged.
///
/// Evaluate every assignment against the row values from before the
/// statement, and only for matching rows. For example, `SET a = b, b = a`
/// exchanges the two values; an expression such as `100 / divisor` must
/// not be evaluated on rows excluded by the filters.
///
/// # Execution
///
/// Like [`Self::delete_from`], this method is called during physical
/// planning, including for `EXPLAIN`. Perform mutations when the returned
/// plan executes so planning does not change the table.
///
/// # Limitations
///
/// Subqueries have the same limitations as in [`Self::delete_from`]
/// ([#24654]). `UPDATE ... FROM` is not supported ([#19950]).
///
/// [#24654]: https://github.com/apache/datafusion/issues/24654
/// [#19950]: https://github.com/apache/datafusion/issues/19950
// Hand-written `#[async_trait]` expansion to reduce compile time. See
// <https://github.com/apache/datafusion/issues/13814#issuecomment-5292709677>
fn update<'life0, 'life1, 'async_trait>(
Expand Down
13 changes: 13 additions & 0 deletions docs/source/library-user-guide/custom-table-providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -792,6 +792,19 @@ that a `FilterExec` is unnecessary for the `date` predicate, and the second
ensures that only the relevant directories are scanned. The actual file reading
happens later, in the stream produced by `execute()`.

## Row-Level DML: DELETE and UPDATE

A custom table provider can support `DELETE` and `UPDATE` by implementing the optional [`TableProvider::delete_from()`] and [`TableProvider::update()`] methods. Their default implementations return a "not implemented" error.

Each method builds an [ExecutionPlan] that changes the matching rows and returns the number of affected rows in a `count` column. Your provider decides how to apply and persist the changes. Perform the changes when the returned plan executes: these methods run during physical planning, including for `EXPLAIN`.

The method documentation describes the filters, assignments, SQL semantics, and result schema that an implementation must support. [MemTable] provides an implementation of both methods over in-memory batches.

For SQL syntax and examples, see the [DML section](../user-guide/sql/dml.md) of the user guide.

[`tableprovider::delete_from()`]: https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableProvider.html#method.delete_from
[`tableprovider::update()`]: https://docs.rs/datafusion/latest/datafusion/catalog/trait.TableProvider.html#method.update

## Putting It All Together

Here is a minimal but complete example of a custom table provider that generates
Expand Down
Loading