Describe the bug
On Spark 5, GpuGroupPartitionsExec can execute planning-time partition groups after its child
has been replaced by a plan with a different partitioning. This can silently omit input
partitions and return wrong results.
SPARK-59289 binds a CPU GroupPartitionsExec's grouping to the childPartitioning it was planned
over. If a later rewrite changes the child's partitioning, Spark drops the keyed partitioning
claim and refuses row or columnar execution. The GPU replacement instead snapshots
groupedPartitions.map(_._2) into GpuGroupPartitionsExecInfo and does not retain or validate the
planned child partitioning.
Steps/Code to reproduce bug
The following regression test can be added to GroupPartitionsExecSuite and run with
buildver=500. It plans groups over three input partitions, replaces the child with a
four-partition GPU leaf, confirms that the CPU operator rejects the stale grouping, and then
executes the same stale groups through the GPU operator.
test("Spark 5 GPU execution must reject stale GroupPartitionsExec grouping") {
withGpuSparkSession { _ =>
val attr = AttributeReference("id", IntegerType, nullable = false)()
val originalChild = spy(LocalTableScanExec(Seq(attr), Nil, None))
doReturn(KeyedPartitioning(
Seq(attr),
Seq(InternalRow(1), InternalRow(1), InternalRow(2))))
.when(originalChild).outputPartitioning
val plannedCpu = GroupPartitionsExecTestShim(
originalChild,
enableSortedMerge = false)
assert(plannedCpu.groupedPartitions.map(_._2) == Seq(Seq(0, 1), Seq(2)))
val changedChild = GroupPartitionsTestGpuLeaf(
Seq(attr),
Seq(Seq[Integer](10), Seq[Integer](20), Seq[Integer](30), Seq[Integer](40)),
Seq.empty)
val staleCpu = plannedCpu.withNewChildren(Seq(changedChild))
.asInstanceOf[GroupPartitionsExec]
val cpuError = intercept[SparkException](staleCpu.executeColumnar())
assert(cpuError.getMessage.contains(
"no longer reports the partitioning it was planned over"))
val staleGpu = GpuGroupPartitionsExec(
changedChild,
GpuGroupPartitionsExecInfo(staleCpu, Seq.empty))
val actual = GpuColumnarToRowExec(staleGpu)
.executeCollect()
.map(_.getInt(0))
.toSeq
assert(actual == Seq(10, 20, 30, 40))
}
}Observed result:
Spark 5 GPU execution must reject stale GroupPartitionsExec grouping *** FAILED ***
ArraySeq(10, 20, 30) did not equal List(10, 20, 30, 40)
Tests: succeeded 17, failed 1
The stale groups are [[0, 1], [2]], so GPU execution never schedules parent partition 3 and
silently drops value 40.
Validation command:
env -u SPARK_HOME mvn -f scala2.13/pom.xml -Dbuildver=500 -Dcuda.version=cuda13 -DwildcardSuites=com.nvidia.spark.rapids.GroupPartitionsExecSuite clean package
Expected behavior
The GPU replacement must not execute partition groups when the current child no longer reports
the partitioning those groups were planned over. It should preserve Spark 5's rejection/fallback
behavior instead of silently reading an incomplete set of parent partitions.
Environment details (please complete the following information)
Additional context
This reproducer explicitly constructs the post-planning child replacement. It proves the GPU
operator returns an incomplete result if that state reaches conversion or execution; it does not
yet establish that a current built-in Spark SQL or AQE rule makes the state reachable in a normal
query. The fix should carry and validate the planned child partitioning at the appropriate GPU
conversion/execution boundary and add a Spark 5 regression test modeled on Spark's SPARK-59289
coverage.
Describe the bug
On Spark 5,
GpuGroupPartitionsExeccan execute planning-time partition groups after its childhas been replaced by a plan with a different partitioning. This can silently omit input
partitions and return wrong results.
SPARK-59289 binds a CPU
GroupPartitionsExec's grouping to thechildPartitioningit was plannedover. If a later rewrite changes the child's partitioning, Spark drops the keyed partitioning
claim and refuses row or columnar execution. The GPU replacement instead snapshots
groupedPartitions.map(_._2)intoGpuGroupPartitionsExecInfoand does not retain or validate theplanned child partitioning.
Steps/Code to reproduce bug
The following regression test can be added to
GroupPartitionsExecSuiteand run withbuildver=500. It plans groups over three input partitions, replaces the child with afour-partition GPU leaf, confirms that the CPU operator rejects the stale grouping, and then
executes the same stale groups through the GPU operator.
Observed result:
The stale groups are
[[0, 1], [2]], so GPU execution never schedules parent partition 3 andsilently drops value
40.Validation command:
Expected behavior
The GPU replacement must not execute partition groups when the current child no longer reports
the partitioning those groups were planned over. It should preserve Spark 5's rejection/fallback
behavior instead of silently reading an incomplete set of parent partitions.
Environment details (please complete the following information)
5.0.0-20260921.005531-72, Scala 2.13, JDK 174a4bedd5738d1e44a8bc5bcf76e6425c1a797d1ebuildver=500, CUDA 13 artifactAdditional context
expectedPartitionKeysaccessor but does not preserve the newchildPartitioningvalidation introduced by the same Spark change.KeyedPartitioninginstances share the samepartitionKeysreference.execution.
This reproducer explicitly constructs the post-planning child replacement. It proves the GPU
operator returns an incomplete result if that state reaches conversion or execution; it does not
yet establish that a current built-in Spark SQL or AQE rule makes the state reachable in a normal
query. The fix should carry and validate the planned child partitioning at the appropriate GPU
conversion/execution boundary and add a Spark 5 regression test modeled on Spark's SPARK-59289
coverage.