Describe the bug
Filtering a window ranking to the first ten rows retains the full input
estimate. The reduced example estimates 73,049 rows and emits ten; the window
input itself is estimated correctly.
To Reproduce
From the repository root, with
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
repro_dir=$(mktemp -d)
mkdir -p "$repro_dir/data"
curl --fail --location \
https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/date_dim.parquet \
-o "$repro_dir/data/date_dim.parquet"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT * FROM (
SELECT d_date_sk, ROW_NUMBER() OVER (ORDER BY d_date_sk) AS rn
FROM date_dim
) t WHERE rn <= 10;
SELECT * FROM (
SELECT d_date_sk, RANK() OVER (ORDER BY d_date_sk) AS rn
FROM date_dim
) t WHERE rn <= 10;
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.
| Predicate |
FilterExec node |
Estimated rows |
Actual rows |
| ROW_NUMBER() <= 10 |
0.0 |
73,049 |
10 |
| RANK() <= 10 |
0.0 |
73,049 |
10 |
Expected behavior
Expose usable ranking-column statistics. A global ROW_NUMBER() <= 10 has an
upper bound of ten rows. RANK needs tie-aware estimation, and partitioned
windows need partition-count estimates; neither can generally use that same
global bound.
Additional context
Reduced from the ranking filters in TPC-DS Q49/Q67/Q70, with ROW_NUMBER added as
a stronger semantic control. BoundedWindowAggExec currently appends unknown
column statistics for its window outputs. This concerns estimates, independently
of whether execution can stop early.
Part of #25610.
Describe the bug
Filtering a window ranking to the first ten rows retains the full input
estimate. The reduced example estimates 73,049 rows and emits ten; the window
input itself is estimated correctly.
To Reproduce
From the repository root, with
PR #25570 applied:
Observed at
this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.
0.00.0Expected behavior
Expose usable ranking-column statistics. A global ROW_NUMBER() <= 10 has an
upper bound of ten rows. RANK needs tie-aware estimation, and partitioned
windows need partition-count estimates; neither can generally use that same
global bound.
Additional context
Reduced from the ranking filters in TPC-DS Q49/Q67/Q70, with ROW_NUMBER added as
a stronger semantic control. BoundedWindowAggExec currently appends unknown
column statistics for its window outputs. This concerns estimates, independently
of whether execution can stop early.
Part of #25610.