-
Notifications
You must be signed in to change notification settings - Fork 328
Fix the chained UDF with Row type #411
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs
Outdated
Show resolved
Hide resolved
src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs
Outdated
Show resolved
Hide resolved
src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs
Outdated
Show resolved
Hide resolved
src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs
Outdated
Show resolved
Hide resolved
src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs
Outdated
Show resolved
Hide resolved
src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs
Outdated
Show resolved
Hide resolved
src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs
Outdated
Show resolved
Hide resolved
src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfComplexTypesTests.cs
Outdated
Show resolved
Hide resolved
/// <summary> | ||
/// Constructor for the schema-less Row class used for chained UDFs. | ||
/// </summary> | ||
/// <param name="genericRow">GenericRow to convert from</param> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there is no conversion happening here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This constructor is for converting GenericRow
to schema-less Row
, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
well, it's just setting the given value to its member (not "convert"ing it)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it, make sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks!
This is a follow PR of #376 which will fix the following chained UDF issue with Row type (#407):
We compared with PySpark UDFs and found that
Schema
is not passed in chained UDF cases. So schema-lessRow
is created to handle chained UDF.