Conversation
Adds a `Syntax vs Semantics` specification that states explicitly what sqlparser's README states for the parser and what DataFusion's planning docs only imply: the parser checks syntax, and semantic checks belong on the LogicalPlan so they apply to every frontend (SQL, DataFrame, Substrait, custom planners) rather than to SQL alone. Also cross links the new page from `DFParser`, the crate level architecture docs, and the "Extending SQL" guide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #25540 +/- ##
========================================
Coverage 82.38% 82.39%
========================================
Files 1138 1138
Lines 434492 434623 +131
Branches 434492 434623 +131
========================================
+ Hits 357975 358097 +122
+ Misses 54873 54857 -16
- Partials 21644 21669 +25 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
When reviewing PRs we regularly ask contributors to move a check out of the SQL
parser or the SQL planner, because:
stated explicitly by sqlparser,
but nowhere in DataFusion's own docs)
SqlToReldoes not apply to plans built by theDataFrame API, Substrait,
datafusion-proto, or other query languages builton DataFusion
Today the only statement of this in DataFusion is implicit, in the
planning overview,
which describes where each phase happens but never says which kind of check
belongs where. That makes the review feedback look like a matter of taste, and
new checks keep landing in the wrong layer.
What changes are included in this PR?
A new specification page,
Syntax vs Semantics(
docs/source/contributor-guide/specification/syntax-vs-semantics.md), which:DFParserchecks syntax only, and why (no catalog/schema access,the AST is also used for round tripping, and parser checks are skipped by
every non-SQL frontend), quoting sqlparser's
Syntax vs SemanticssectionLogicalPlan, in a node'stry_newindatafusion-expr, so that every frontend passes through themSqlToRel,try_new,AnalyzerRule/invariants, execution)SqlToRel(unsupported SQL constructsthat have no plan representation, and
Diagnostic/span context on errors).slttests for the user visible error, so that tests survive acheck moving between layers
checks in the wrong place
It also cross links the new page from the three places a contributor is likely
to be when they write such a check: the
DFParserdocs, the crate levelarchitecture docs in
datafusion/core/src/lib.rs, and the "Extending SQL"library user guide.
Examples in the doc
In the right place (single check, all frontends):
Filter::try_new: non boolean predicates and window functions in a predicatecheck_aggregate_and_window_nesting, fromAggregate::try_new/Window::try_newUnion::try_new, theTypeCoercionanalyzer rule,assert_valid_semantic_planIn the wrong place (SQL only), listed as existing violations rather than precedents:
Aggregates in
WHERE(datafusion/sql/src/select.rs). The same query via theDataFrame API builds the plan and fails much later in physical planning:
Contrast with window functions in
WHERE, checked inFilter::try_new, whereSQL and the DataFrame API give the same error.
'IF NOT EXISTS' cannot coexist with 'REPLACE'andConstraints on Partition Columns are not supported, rejected inDFParseralthough both statements parseColumn reference is not allowed in the DEFAULT expression, rejected inSqlToRelWhat is the testing strategy for this PR?
Docs only, so there are no new tests.
./ci/scripts/doc_prettier_check.sh,cargo fmt --all -- --check, andRUSTDOCFLAGS="-D warnings" cargo doc -p datafusion-sql --no-deps(for the new intra doc link) all pass. The errormessages quoted in the new page were produced by running the queries against
this branch rather than written from memory.
Are there any user-facing changes?
No code changes. New contributor documentation, plus rustdoc pointers to it.