MDEV-38722 server crash hp_rec_key_cmp

[sanja-byelkin]

https://jira.mariadb.org/browse/MDEV-38722

Problem

In an extended set operation (select_unit_ext, used when EXCEPT ALL or
INTERSECT ALL is present) the HEAP unique index is released once, at the
node pointed to by union_distinct, assuming every operation after it is a
plain UNION ALL that can be unfolded. That does not hold for a trailing
EXCEPT ALL: select_unit_ext::send_data() still calls find_unique_row()
for it (only the UNION ALL branch checks is_index_enabled), so it
dereferenced the already released index and the server crashed in
hp_rec_key_cmp().

Minimal reproducer:

SELECT 1 INTERSECT SELECT 1 UNION ALL SELECT 1 EXCEPT ALL SELECT 1;

Fix

• optimize_bag_operation() postpones the index release past such a trailing
  EXCEPT ALL, so the index is available while it is executed.
• Keeping the index alive uncovered a second problem: a leading INTERSECT
  that stays inline in the extended operation left the rows that did not match
  every INTERSECT operand in the temporary table (the materialized case is
  handled by select_unit::send_eof(), the inline select_unit_ext override
  did not), producing a wrong multiset once the crash was gone.
  select_unit_ext::send_eof() already scans the temporary table (to reset the
  duplicate counter, fold INTERSECT ALL counters, or unfold), so the
  filtering of the non-matching INTERSECT records is folded into those loops
  (guarded by filter_intersect) and the table is still scanned only once.

Test

Regression test added to main/intersect_all.test; verified the flat form
matches the parenthesized (materialized) form across the mixed cases, in
default and Oracle mode, with --ps-protocol/--view-protocol.

🤖 Generated with Claude Code

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[gemini-code-assist]

Code Review

This pull request addresses a server crash (MDEV-38722) in hp_rec_key_cmp by filtering out records that do not match every operand in an inline INTERSECT DISTINCT subsequence and postponing index release for EXCEPT ALL operations. The review feedback correctly identifies a bug in the filter_intersect condition: when INTERSECT DISTINCT is followed by INTERSECT ALL, the filtering is incorrectly skipped because both share the INTERSECT_TYPE linkage. To fix this, the condition should also verify that the next select is distinct.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

If an inline INTERSECT DISTINCT is followed by an INTERSECT ALL operation, the INTERSECT DISTINCT subsequence ends. However, because next_sl->get_linkage() == INTERSECT_TYPE is true for both INTERSECT DISTINCT and INTERSECT ALL, filter_intersect will incorrectly evaluate to false. This prevents the non-matching records of the INTERSECT DISTINCT phase from being filtered out, leading to incorrect query results.

To fix this, we should also check next_sl->distinct to ensure we only skip filtering if the next operation is indeed another INTERSECT DISTINCT.

  bool filter_intersect= curr_op_type == INTERSECT_DISTINCT &&
                         !(next_sl &&
                           next_sl->get_linkage() == INTERSECT_TYPE &&
                           next_sl->distinct);

[sanja-byelkin]

Thanks. I looked into this specific case (an inline INTERSECT DISTINCT followed by INTERSECT ALL) and it produces correct results as written.

When the next operation is INTERSECT ALL, select_unit_ext::send_eof() already enters the scan loop through the is_next_intersect_all condition and folds the DISTINCT result into the INTERSECT ALL counters (duplicate_cnt/additional_cnt). The rows that did not match the DISTINCT phase are then removed by the INTERSECT ALL counting/unfold, so they do not leak into the result — the filtering is effectively still done, just by the following INTERSECT ALL rather than by filter_intersect.

I verified this with a range of INTERSECT DISTINCT → INTERSECT ALL queries — the leftover value both absent from and present in the following INTERSECT ALL operand, chained INTERSECT ALL, different multiplicities, trailing UNION ALL/EXCEPT ALL, and ORACLE mode. In every case the flat form matches its parenthesized (materialized) equivalent, e.g.:

-- ta=(1,1,2,2,3,4,4,4) tb=(1,2,4,4,5) tc=(2,2,4,6)
SELECT i FROM ta INTERSECT SELECT i FROM tb INTERSECT ALL SELECT i FROM tc;
-- => 2, 4   (matches ((ta INTERSECT tb) INTERSECT ALL tc))

I also tried the suggested && next_sl->distinct: it yields identical results on the whole intersect_all suite and on these cases, i.e. it is behavior-neutral here rather than a fix. So filter_intersect intentionally targets only the terminal INTERSECT of a DISTINCT subsequence; when an INTERSECT ALL follows, the is_next_intersect_all path performs the cleanup.

[Copilot]

Pull request overview

Fixes a server crash and correctness issues in extended set operations (select_unit_ext) involving mixed INTERSECT/UNION ALL/EXCEPT ALL, specifically ensuring the HEAP unique index isn’t disabled too early and filtering out non-matching INTERSECT DISTINCT rows during the existing single table scan.

Changes:

• Postpone unique-index disabling in optimize_bag_operation() so EXCEPT ALL still has a valid unique index when select_unit_ext::send_data() calls find_unique_row().
• Add filter_intersect handling in select_unit_ext::send_eof() to delete rows whose intersect counter doesn’t match curr_step, folded into the existing scan/unfold loops.
• Add regression coverage in main/intersect_all.test (with updated expected output) for the crash reproducer and mixed-operation equivalence checks.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
sql/sql_union.cc	Prevents early unique-index disabling before EXCEPT ALL and filters non-matching INTERSECT DISTINCT rows during the existing scan/unfold pass.
mysql-test/main/intersect_all.test	Adds regression tests for the crash reproducer and mixed set-operation cases (flat vs parenthesized/materialized forms).
mysql-test/main/intersect_all.result	Updates expected output for the new regression test cases.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[midenok]

The whole test is lowercase sql, it would be nice to keep it lowercase. This is on your discretion, the fix LGTM.