Skip to content

Distinguish string equality, LIKE and NOT LIKE estimates #25614

Description

@gabotechs

Describe the bug

String equality and pattern predicates all receive the same 20% fallback. LIKE
and NOT LIKE also get identical estimates even though they partition this
non-null input.

To Reproduce

From the repository root, with the CLI fix from
PR #25570 applied:

cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
  --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT o_orderkey FROM orders WHERE o_comment LIKE '%special%requests%';

SELECT o_orderkey FROM orders WHERE o_comment NOT LIKE '%special%requests%';

SELECT n_nationkey FROM nation WHERE n_name = 'GERMANY';
SELECT s_suppkey FROM supplier WHERE s_comment LIKE '%Customer%Complaints%';
SQL
target/ci/dfbench statistics \
  --path "$repro_dir/data" --query_path "$repro_dir/repro.sql"

Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.

SELECT FilterExec node Estimated rows Actual rows
Orders LIKE 0 300,000 16,082
Orders NOT LIKE 0 300,000 1,483,918
Nation equality 0 5 1
Supplier LIKE 0 2,000 4

Expected behavior

Use NDVs/frequencies for equality where available, distinguish pattern forms,
and make LIKE/NOT LIKE estimates complementary over non-null rows.

Additional context

The predicates come from TPC-H Q11/Q13/Q16. Related:
filter selectivity #14237.

Part of #25610.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions