Skip to content

Commit 32aff86

Browse files
ulysses-youcloud-fan
authored andcommitted
[SPARK-39447][SQL][3.2] Avoid AssertionError in AdaptiveSparkPlanExec.doExecuteBroadcast
This is a backport of #36974 for branch-3.2 ### What changes were proposed in this pull request? Change `currentPhysicalPlan` to `inputPlan ` when we restore the broadcast exchange for DPP. ### Why are the changes needed? The currentPhysicalPlan can be wrapped with broadcast query stage so it is not safe to match it. For example: The broadcast exchange which is added by DPP is running before than the normal broadcast exchange(e.g. introduced by join). ### Does this PR introduce _any_ user-facing change? yes bug fix ### How was this patch tested? add test Closes #37087 from ulysses-you/inputplan-3.2. Authored-by: ulysses-you <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent be891ad commit 32aff86

File tree

2 files changed

+20
-1
lines changed

2 files changed

+20
-1
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -658,7 +658,7 @@ case class AdaptiveSparkPlanExec(
658658
// node to prevent the loss of the `BroadcastExchangeExec` node in DPP subquery.
659659
// Here, we also need to avoid to insert the `BroadcastExchangeExec` node when the newPlan is
660660
// already the `BroadcastExchangeExec` plan after apply the `LogicalQueryStageStrategy` rule.
661-
val finalPlan = currentPhysicalPlan match {
661+
val finalPlan = inputPlan match {
662662
case b: BroadcastExchangeLike
663663
if (!newPlan.isInstanceOf[BroadcastExchangeLike]) => b.withNewChildren(Seq(newPlan))
664664
case _ => newPlan

sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1597,6 +1597,25 @@ class DynamicPartitionPruningV1SuiteAEOff extends DynamicPartitionPruningV1Suite
15971597
class DynamicPartitionPruningV1SuiteAEOn extends DynamicPartitionPruningV1Suite
15981598
with EnableAdaptiveExecutionSuite {
15991599

1600+
test("SPARK-39447: Avoid AssertionError in AdaptiveSparkPlanExec.doExecuteBroadcast") {
1601+
val df = sql(
1602+
"""
1603+
|WITH empty_result AS (
1604+
| SELECT * FROM fact_stats WHERE product_id < 0
1605+
|)
1606+
|SELECT *
1607+
|FROM (SELECT /*+ SHUFFLE_MERGE(fact_sk) */ empty_result.store_id
1608+
| FROM fact_sk
1609+
| JOIN empty_result
1610+
| ON fact_sk.product_id = empty_result.product_id) t2
1611+
| JOIN empty_result
1612+
| ON t2.store_id = empty_result.store_id
1613+
""".stripMargin)
1614+
1615+
checkPartitionPruningPredicate(df, false, false)
1616+
checkAnswer(df, Nil)
1617+
}
1618+
16001619
test("SPARK-37995: PlanAdaptiveDynamicPruningFilters should use prepareExecutedPlan " +
16011620
"rather than createSparkPlan to re-plan subquery") {
16021621
withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",

0 commit comments

Comments
 (0)