Skip to content

perf: mirror compound ASOF key predicates to right - #25728

Merged
jayzhan211 merged 10 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-key-predicate-mirroring
Oct 1, 2026
Merged

jayzhan211 merged 10 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-key-predicate-mirroring

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

#25574 mirrors a selective left_key = literal filter to the right side of an ASOF join. The same equality guarantee applies to any deterministic predicate that references only direct left equality-key columns, including IN, ranges, and OR expressions. Mirroring those predicates avoids sorting and broadcasting right-side groups that cannot match a surviving left row.

Performance

Q10 has 50M rows on each side across 500K equality groups. WHERE l.key IN (42, 43) retains two groups and produces 200 rows with both revisions. On an Apple M4 Max, using independent release builds and two 7-iteration runs per revision, the median elapsed time across all 14 iterations was:

Revision Median elapsed
Current main after #25574 (acf5c89e6) 984.9 ms
This change (5763d505a) 82.3 ms

This selective compound-key workload is about 12.0x faster (91.6% lower elapsed time). The benchmark file and command were the same for both binaries:

target/release/benchmark_runner asof_join --query 10 --iterations 7

What changes are included in this PR?

  • Build an ASOF-specific column mapping for direct equality keys.
  • Mirror a pushed deterministic predicate only when every referenced column has a right-key mapping.
  • Support compound IN, range, and OR predicates while leaving mixed key/non-key predicates entirely off the right input.
  • Add plan coverage and ASOF benchmark Q10.

Coerced and expression-based equality keys remain excluded because they cannot be replaced column-for-column.

What is the testing strategy for this PR?

The new asof_join.slt cases verify compound predicate mirroring and the all-or-nothing behavior for a mixed key/non-key OR predicate. Q10 covers the user-visible selective workload.

Also run:

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D warnings
  • The extended workspace test suite, including all 524 sqllogictest files

Are there any user-facing changes?

No API or result changes. More eligible ASOF joins can sort and retain fewer right-side rows.

@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Sep 25, 2026
@codecov-commenter

codecov-commenter commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.58%. Comparing base (5635e5c) to head (55022b4).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/optimizer/src/push_down_filter.rs 85.71% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25728      +/-   ##
==========================================
- Coverage   82.58%   82.58%   -0.01%     
==========================================
  Files        1145     1145              
  Lines      443657   443651       -6     
  Branches   443657   443651       -6     
==========================================
- Hits       366396   366383      -13     
- Misses      55011    55017       +6     
- Partials    22250    22251       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…cate-mirroring

# Conflicts:
#	benchmarks/sql_benchmarks/asof_join/asof_join.suite
#	datafusion/optimizer/src/push_down_filter.rs
#	datafusion/sqllogictest/test_files/asof_join.slt
@Xuanwo
Xuanwo marked this pull request as ready for review September 26, 2026 18:07
@Xuanwo

Xuanwo commented Sep 26, 2026

Copy link
Copy Markdown
Member Author

Hi @jayzhan211 and @2010YOUY01, #25574 has merged, so I merged the latest main into this follow-up and marked it ready. Could you take a look when you have time? Thanks!

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Xuanwo !

@Xuanwo

Xuanwo commented Sep 30, 2026

Copy link
Copy Markdown
Member Author

Let's merge this PR in 😄 !

@jayzhan211
jayzhan211 disabled auto-merge October 1, 2026 12:18
@jayzhan211
jayzhan211 enabled auto-merge October 1, 2026 12:18
@jayzhan211

Copy link
Copy Markdown
Contributor

Couldn't merge 🤔 re-trigger the CI

@2010YOUY01

Copy link
Copy Markdown
Contributor

Couldn't merge 🤔 re-trigger the CI

If there is a new CI job added in main, we have to merge main in the PR to trigger that new job, and make it mergeable.

(And thanks for the great work on another improvement!)

@jayzhan211
jayzhan211 added this pull request to the merge queue Oct 1, 2026
Merged via the queue into apache:main with commit 242b96c Oct 1, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants