[SPARK-42259][SQL] ResolveGroupingAnalytics should take care of Python UDAF

cloud-fan · cloud-fan · commit 80e8df11d7e2 · 2023-02-01T17:40:56.000+08:00
This is a long-standing correctness issue with Python UDAF and grouping analytics. The rule `ResolveGroupingAnalytics` should take care of Python UDAF when matching aggregate expressions. bug fix Yes, the query result was wrong before existing tests Closes #39824 from cloud-fan/python. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 1219c84) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -617,7 +617,7 @@ class Analyzer(override val catalogManager: CatalogManager)
         // AggregateExpression should be computed on the unmodified value of its argument
         // expressions, so we should not replace any references to grouping expression
         // inside it.
-        case e: AggregateExpression =>
+        case e if AggregateExpression.isAggregate(e) =>
           aggsBuffer += e
           e
         case e if isPartOfAggregation(e) => e

Original file line number	Diff line number	Diff line change
`@@ -617,7 +617,7 @@ class Analyzer(override val catalogManager: CatalogManager)`
`617`	`617`	`// AggregateExpression should be computed on the unmodified value of its argument`
`618`	`618`	`// expressions, so we should not replace any references to grouping expression`
`619`	`619`	`// inside it.`
`620`		`- case e: AggregateExpression =>`
	`620`	`+ case e if AggregateExpression.isAggregate(e) =>`
`621`	`621`	`aggsBuffer += e`
`622`	`622`	`e`
`623`	`623`	`case e if isPartOfAggregation(e) => e`