Implement SQL grouping extensions: ROLLUP, CUBE, GROUPING SETS, GROUPING(), GROUP BY DISTINCT by livius2 · Pull Request #9029 · FirebirdSQL/firebird

livius2 · 2026-05-17T14:53:10Z

Hi

A few years ago I made an earlier attempt at implementing ROLLUP / CUBE support in Firebird.
That prototype was based on joining the input stream with a procedure producing a sequence of numbers/masks.
It worked for simple cases, but the approach turned out to be hard to generalize, difficult to integrate cleanly with the optimizer/executor, and not a good long-term fit for Firebird internals.

After several years of following Firebird development and repository changes, I decided to revisit this feature with a cleaner implementation strategy.

This pull request implements SQL extended grouping support, including:

GROUP BY ROLLUP (...)
GROUP BY CUBE (...)
GROUP BY GROUPING SETS (...)
mixed grouping elements, for example GROUP BY a, ROLLUP(b, c)
empty grouping set ()
GROUPING() with multiple arguments, SQL feature T433
GROUP BY DISTINCT, SQL feature T434
GROUPING_ID as a compatibility extension

The implementation lowers advanced grouping to existing aggregate and UNION ALL machinery, without adding new BLR opcodes, ODS changes.

This is a flattened commit from my local repo, there were too many commits and experiments there.
Please note that all English texts come from a translator.
I will add tests to Firebird-qa soon.
Tests added now:
FirebirdSQL/firebird-qa#38

GROUP BY ROLLUP(...) GROUP BY CUBE(...) GROUP BY GROUPING SETS (...) GROUP BY () GROUPING(...) GROUPING_ID(...) GROUP BY DISTINCT

sim1984 · 2026-05-18T06:39:36Z

The implementation lowers advanced grouping to existing aggregate and UNION ALL machinery, without adding new BLR opcodes, ODS changes.

This is unfortunate; at least aggregate functions containing only WITH ROLLUP can be executed in a single pass (without re-executing the query). This is unlikely to be possible with CUBE and GROUPING SETS, but there are options there, such as repeating groupings based on records stored in the "Record Buffer".

livius2 · 2026-05-18T08:14:16Z

I agree. This implementation intentionally uses lowering to the existing aggregate + UNION ALL infrastructure as a first correctness-focused step, avoiding new BLR, ODS, record source, or executor changes.

This is not meant to be the final optimal execution strategy. ROLLUP is a good candidate for a future single-pass implementation, as its grouping levels are hierarchical and can be produced from one ordered/grouped stream. CUBE and arbitrary GROUPING SETS are less straightforward, but there are still possible optimizations, for example sharing/materializing the base stream or reusing buffered records instead of re-executing the full input for every grouping set.

For this PR, the main goal is SQL semantics and integration with existing DSQL/BLR paths. Native execution and optimizer improvements can be developed as a follow-up without changing the SQL surface introduced here.

sim1984 · 2026-05-18T08:28:09Z

I don't see any reason to change the ODS. The introduction of new BLR verbs may be necessary to improve implementation efficiency, but it's not necessary.

The main question: do you plan to implement support in the optimizer yourself or wait for someone else to do it?

livius2 · 2026-05-18T08:50:54Z

Yes, if I understand correctly, by optimizer support you mean a future native execution strategy, for example single-pass execution for pure ROLLUP cases.

I do plan to work on that, but I would prefer to do it after this PR is accepted and merged. This PR has already required a significant amount of work, and I would like to avoid investing in a larger implementation before knowing whether this approach is acceptable.

# Conflicts: # src/dsql/parse-conflicts.txt

# Conflicts: # src/common/ParserTokens.h # src/dsql/parse-conflicts.txt # src/dsql/parse.y

dyemanov

This is the first part of the review, the hardest changes inside pass1.cpp are still to be reviewed a bit later (the next week, hopefully).

dyemanov · 2026-07-03T14:05:44Z

+				}
+			}
+			else
+				$$->legacyGroup = NULL;


I have a feeling that fixup of legacyGroup does not belong to the parser and should better be done inside dsqlPass().

Done in 25eb2a9

dyemanov · 2026-07-03T14:10:48Z

+		};
+
+		explicit Element(Type aType = Type::SIMPLE, ValueListNode* aItems = NULL,
+				GroupingClause* aGroupingSets = NULL)


nullptr instead of NULL, please

Done in cfc8434

dyemanov · 2026-07-03T14:11:17Z

+	{
+		ALL,
+		DISTINCT
+	};


This could be just a bool, I suppose.

Done in 3282fff

dyemanov · 2026-07-03T14:19:16Z

+	{
+		explicit Dimension(MemoryPool&)
+			: expr(NULL),
+			  index(0)


Here and in other places -- constant-based initialization should better be done in the member declaration, e.g. unsigned index = 0;. And NestConst already initializes to nullptr by default, no need to assign it explicitly.

Done in 7fede78

dyemanov · 2026-07-03T14:25:59Z

+
+dsc* GroupingNode::execute(thread_db* /*tdbb*/, Request* /*request*/) const
+{
+	fb_assert(false);


Does it mean <group operation> is temporarily disallowed (returns NULL) until proper BLR codes are added in another PR?

No, GROUPING/GROUPING_ID are not temporarily disallowed and they do not return NULL.

The node exists only as a DSQL-level representation parsed from SQL. For supported contexts, advanced grouping is lowered before BLR generation, and GROUPING/GROUPING_ID are replaced with branch-specific integer constants: 0/1 for GROUPING and a bit mask for GROUPING_ID.

So GroupingNode::execute() should never be reached in a valid compiled request. The fb_assert(false) is a defensive guard for an internal lowering failure. Similarly, genBlr() raises an error if a GROUPING expression reaches BLR generation without being lowered.

This PR intentionally avoids new BLR opcodes. Proper native BLR/executor support could be added later as an optimization path, but it is not required for the SQL feature to work in this implementation.

…be nested inside another grouping set, matching the parser and SQL specification.

…ation and lowering logic

Implemented SQL support for advanced grouping:

c806e55

GROUP BY ROLLUP(...) GROUP BY CUBE(...) GROUP BY GROUPING SETS (...) GROUP BY () GROUPING(...) GROUPING_ID(...) GROUP BY DISTINCT

livius2 mentioned this pull request May 17, 2026

tests for grouping sets, rollup, cube FirebirdSQL/firebird-qa#38

Open

Merge branch 'master' into grouping_sets__flattened

6452298

# Conflicts: # src/dsql/parse-conflicts.txt

This was referenced May 22, 2026

SELECT ... GROUP BY ... WITH CUBE [CORE650] #1016

Open

Add standard SQL "extended grouping capabilities" - ROLLUP, CUBE, GROUPING SETS [CORE4995] #5283

Open

This was linked to issues May 24, 2026

Add standard SQL "extended grouping capabilities" - ROLLUP, CUBE, GROUPING SETS [CORE4995] #5283

Open

SELECT ... GROUP BY ... WITH CUBE [CORE650] #1016

Open

livius2 added 3 commits June 22, 2026 22:48

deleting an unnecessarily copied comment

b36ac24

Merge branch 'master' into grouping_sets__flattened

6f17370

# Conflicts: # src/common/ParserTokens.h # src/dsql/parse-conflicts.txt # src/dsql/parse.y

update pares conflicts txt

303b681

dyemanov self-requested a review July 3, 2026 13:07

dyemanov requested changes Jul 3, 2026

View reviewed changes

livius2 added 8 commits July 4, 2026 08:04

Update grouping sets documentation to mention that GROUPING SETS may …

0e50834

…be nested inside another grouping set, matching the parser and SQL specification.

Use a single GroupingNode for GROUPING and GROUPING_ID, sharing valid…

d0b6e7c

…ation and lowering logic

Merge branch 'master' into grouping_sets__flattened

4a3d3af

Reuse quantifier_opt for GROUP BY duplicate mode

8a16dd9

Use member initializers in grouping nodes

7fede78

Use nullptr

cfc8434

Use bool for grouping set distinct mode

3282fff

Move grouping legacy list maintenance out of parser

25eb2a9

Uh oh!

Uh oh!

Conversation

livius2 commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sim1984 commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

livius2 commented May 18, 2026

Uh oh!

sim1984 commented May 18, 2026

Uh oh!

livius2 commented May 18, 2026

Uh oh!

dyemanov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

livius2 commented May 17, 2026 •

edited

Loading

sim1984 commented May 18, 2026 •

edited

Loading