Skip to content

Full join output partitioning - #25554

Open
gstamatakis95 wants to merge 2 commits into
apache:mainfrom
gstamatakis95:full-join-output-partitioning
Open

gstamatakis95 wants to merge 2 commits into
apache:mainfrom
gstamatakis95:full-join-output-partitioning

Conversation

@gstamatakis95

@gstamatakis95 gstamatakis95 commented Sep 20, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

A partitioned Full Outer Join reports UnknownPartitioning even when both inputs are already partitioned the same way on the join keys. Every output row stays in the partition its key was routed to, so the report throws away a real guarantee. A later aggregate or join on the join key then gets a RepartitionExec it does not need.

The left key alone cannot describe the output, because right only rows carry NULL in the left key in every partition. The key that agrees with the partition number is coalesce(left_key, right_key).

What changes are included in this PR?

  • symmetric_join_output_partitioning takes the join keys and the join's equivalence properties. For a Full join whose inputs pass the existing co-partitioning check, it reports the left partitioning restated on CASE WHEN l IS NOT NULL THEN l ELSE r END for each key pair, through Partitioning::adapt. Hash keeps its count, Range keeps its split points and sort options. Anything else still reports UnknownPartitioning.
  • The join records that both coalesce orders of each key pair are equal, so a parent keyed on COALESCE(r.k, l.k) matches as well.
  • HashJoinExec in partitioned mode, SortMergeJoinExec, and SymmetricHashJoinExec pass their keys and equivalence properties into the helper.

The CASE form is what the planner produces for a two argument COALESCE, so a parent's expression compares equal structurally.

What is the testing strategy for this PR?

  • Unit tests in joins/utils.rs cover the Hash, Range, and multi key outputs, the fallbacks to UnknownPartitioning, projections that keep or drop a key, the sort merge and symmetric hash join paths, and the equivalence of both coalesce orders.
  • enforce_distribution.rs tests show no shuffle for a parent join on the coalesced key over Hash and Range inputs, in either key order and over a swapped join, and a shuffle for a parent joon the plain key.
  • range_partitioning.slt tests 49 to 53 show plans and results for an aggregate on COALESCE, a chained Full join, the hash repartitioned variant, the negative GROUP BY l.range_key case with its single NULL group, and the reversed coalesce order.
  • The existing dataframe test that asserted UnknownPartitioning for Full joins now asserts the coalesced Hash partitioning.

Are there any user-facing changes?

Plans with a partitioned Full join followed by an operator keyed on COALESCE of the join keys lose a RepartitionExec. Results do not change. symmetric_join_output_partitioning is crate private, so there is no public API change.

@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Sep 20, 2026
@gstamatakis95
gstamatakis95 marked this pull request as ready for review September 20, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve partitioning through co-partitioned Full Outer Joins

1 participant