[FEATURE REQUEST]: Change the API of UDF that returns Row

[elvaliuliuliu]

Currently, the UDF that returns Row using GenericRow and Schema as below:

public static Func<Column, Column> Udf<T>(Func<T, GenericRow> udf, StructType returnType)
{
    return CreateUdf(udf.Method.ToString(), UdfUtils.CreateUdfWrapper(udf), returnType).Apply1;
}

This will cause issues for chaining UDFs. We need to modify the API to return Row.

[imback82]

Can you share the PySpark experience if the Row inside the chained UDF has schema?

[elvaliuliuliu]

I have the following chained UDF in PySpark:

# create dataframe
df = spark.range(0,3);
schema = StructType([StructField("col1", IntegerType(), True), StructField("col2", StringType(), True)])
# UDF that returns Row
def udfFun1(s):
    return [1, "abc"]
udf1 = F.udf(udfFun1, StructType(schema))
udfDf1 = df.select(udf1(df["id"]))

# printSchema() prints
# root
# |-- udfFun1(id): struct (nullable = true)
# |    |-- col1: integer (nullable = true)
# |    |-- col2: string (nullable = true)
udfDf1.printSchema()

# show() prints
# +-----------+
# |udfFun1(id)|
# +-----------+
# |   [1, abc]|
# |   [1, abc]|
# |   [1, abc]|
# +-----------+
udfDf1.show()

# UDF that takes in Row
def udfFun2(s):
    return "hello"
udf2 = F.udf(udfFun2, StringType())
# Chained UDF
udfDf2 = df.select(udf2(udf1(df["id"])))

# printSchema() prints
# root
# |-- udfFun2(udfFun1(id)): string (nullable = true)
udfDf2.printSchema()

# show() prints
# +--------------------+
# |udfFun2(udfFun1(id))|
# +--------------------+
# |               hello|
# |               hello|
# |               hello|
# +--------------------+
udfDf2.show()

Can we tell if the Row generated by udf1 has a schema by this example?

[imback82]

Can't you update udfFun2 to look into s, which should be Row object?

[elvaliuliuliu]

Can't you update udfFun2 to look into s, which should be Row object?

As per offline discussion, Schema is not passed in chained UDF in PySpark.

[elvaliuliuliu]

This has been resolved via #411.