Skip to content

perf: mirror ASOF equality-key filters to right - #25574

Open
Xuanwo wants to merge 5 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-key-filter-mirroring
Open

Xuanwo wants to merge 5 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-key-filter-mirroring

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

When a filter selects one ASOF equality-key group on the left, rows in other right-side groups cannot match any surviving left row. The broadcast join still sorts and retains those right rows unless the same key filter is applied to the right input.

Performance

Q09 has 1M rows on each side across 10K equality groups. WHERE l.key = 42 retains one group and produces 100 rows with both revisions. On an Apple M4 Max, using release builds and two 7-iteration runs per revision, the median elapsed time across all 14 iterations was:

Revision Median elapsed
main without this optimization (b172f73f0) 21.5 ms
This change (b3fd5f1f9) 2.8 ms

This selective keyed workload is about 7.8x faster (87% lower elapsed time). The benchmark file and command were the same for both binaries:

target/release/benchmark_runner asof_join --query 9 --iterations 7

The branch was subsequently synced through 21a3215b6; those upstream-only commits do not touch the optimizer or ASOF paths exercised by Q09.

What changes are included in this PR?

  • Mirror a pushed left_key = literal predicate to the corresponding right key when both join keys are direct columns of the same type.
  • Keep the original left filter and ASOF null-padding behavior unchanged. Other predicate shapes and coerced or expression-based keys are not inferred.
  • Add result and plan coverage in asof_join.slt, plus a keyed selective workload as ASOF benchmark Q09.

What is the testing strategy for this PR?

The SLT cases check the mirrored plan, matching and unmatched results, a right-side IS NULL filter, and a left disjunction that must not prune the right input. Q09 runs the user-visible query with 1M rows per side across 10K groups, selecting one group.

Are there any user-facing changes?

No API or result changes. Eligible ASOF joins can sort and retain fewer right-side rows.

@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Sep 21, 2026
@codecov-commenter

codecov-commenter commented Sep 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.07407% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.50%. Comparing base (21a3215) to head (bf38b31).

Files with missing lines Patch % Lines
datafusion/optimizer/src/push_down_filter.rs 74.07% 4 Missing and 3 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #25574   +/-   ##
=======================================
  Coverage   82.50%   82.50%           
=======================================
  Files        1140     1140           
  Lines      438050   438077   +27     
  Branches   438050   438077   +27     
=======================================
+ Hits       361410   361434   +24     
+ Misses      54833    54830    -3     
- Partials    21807    21813    +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…r-mirroring

# Conflicts:
#	benchmarks/sql_benchmarks/asof_join/asof_join.suite
#	datafusion/optimizer/src/push_down_filter.rs
#	datafusion/sqllogictest/test_files/asof_join.slt
@Xuanwo
Xuanwo marked this pull request as ready for review September 24, 2026 12:57
@Xuanwo

Xuanwo commented Sep 24, 2026

Copy link
Copy Markdown
Member Author

Hi @jayzhan211 and @2010YOUY01, this follow-up is now restacked and ready for review. Would you mind taking a look when you have time? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants