Describe the bug
String equality and pattern predicates all receive the same 20% fallback. LIKE
and NOT LIKE also get identical estimates even though they partition this
non-null input.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
--parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT o_orderkey FROM orders WHERE o_comment LIKE '%special%requests%';
SELECT o_orderkey FROM orders WHERE o_comment NOT LIKE '%special%requests%';
SELECT n_nationkey FROM nation WHERE n_name = 'GERMANY';
SELECT s_suppkey FROM supplier WHERE s_comment LIKE '%Customer%Complaints%';
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.
| SELECT |
FilterExec node |
Estimated rows |
Actual rows |
| Orders LIKE |
0 |
300,000 |
16,082 |
| Orders NOT LIKE |
0 |
300,000 |
1,483,918 |
| Nation equality |
0 |
5 |
1 |
| Supplier LIKE |
0 |
2,000 |
4 |
Expected behavior
Use NDVs/frequencies for equality where available, distinguish pattern forms,
and make LIKE/NOT LIKE estimates complementary over non-null rows.
Additional context
The predicates come from TPC-H Q11/Q13/Q16. Related:
filter selectivity #14237.
Part of #25610.
Describe the bug
String equality and pattern predicates all receive the same 20% fallback. LIKE
and NOT LIKE also get identical estimates even though they partition this
non-null input.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
Observed with
tpchgen-cli1.1.1 at6c320561b5.
Inspect the SELECT reports; ignore the empty
SETreports.0000Expected behavior
Use NDVs/frequencies for equality where available, distinguish pattern forms,
and make LIKE/NOT LIKE estimates complementary over non-null rows.
Additional context
The predicates come from TPC-H Q11/Q13/Q16. Related:
filter selectivity #14237.
Part of #25610.