Support for Row type Ser/De and exposing the CreateDataFrame API #338

Niharikadutta · 2019-11-15T02:39:56Z

We are excited to review your PR.

So we can do the best job, please check:

There's a descriptive title that will make sense to other developers some time from now.
There's associated issues. All PR's should have issue(s) associated - unless a trivial self-evident change such as fixing a typo. You can use the format Fixes #nnnn in your description to cause GitHub to automatically close the issue(s) when your PR is merged.
Your change description explains what the change does, why you chose your approach, and anything else that reviewers should know.
You have included any necessary tests in the same PR.

This PR addresses #161 and supports the following:

User can create a DataFrame by giving a List of Rows and StructType schema.
Row type Serialization and De-serialization.
Exposed GenericRow implementation of Spark (that does not take a schema) on .NET for Spark

…reateDataFrame that takes in strings

examples/Microsoft.Spark.CSharp.Examples/Sql/Batch/Basic.cs

src/csharp/Microsoft.Spark/Sql/GenericRow.cs

src/csharp/Microsoft.Spark/Sql/Row.cs

.../microsoft-spark-2.4.x/src/main/scala/org/apache/spark/api/dotnet/DotnetBackendHandler.scala

… it from the data

examples/Microsoft.Spark.CSharp.Examples/Sql/Batch/Basic.cs

src/csharp/Microsoft.Spark/Interop/Ipc/PayloadHelper.cs

src/csharp/Microsoft.Spark.UnitTest/Sql/GenericRowTests.cs

src/csharp/Microsoft.Spark/Interop/Ipc/PayloadHelper.cs

src/scala/microsoft-spark-2.3.x/src/main/scala/org/apache/spark/api/dotnet/SerDe.scala

src/csharp/Microsoft.Spark/Sql/SparkSession.cs

src/csharp/Microsoft.Spark/Sql/GenericRow.cs

Co-Authored-By: Steve Suh <[email protected]>

src/scala/microsoft-spark-2.4.x/src/main/scala/org/apache/spark/api/dotnet/SerDe.scala

src/csharp/Microsoft.Spark.E2ETest/IpcTests/Sql/SparkSessionTests.cs

src/scala/microsoft-spark-3.0.x/src/main/scala/org/apache/spark/sql/api/dotnet/SQLUtils.scala

src/csharp/Microsoft.Spark/Sql/SparkSession.cs

src/scala/microsoft-spark-3.0.x/src/main/scala/org/apache/spark/sql/api/dotnet/SQLUtils.scala

Co-Authored-By: Steve Suh <[email protected]>

…t schema inference

src/csharp/Microsoft.Spark.E2ETest/IpcTests/Sql/SparkSessionTests.cs

src/csharp/Microsoft.Spark.UnitTest/Sql/GenericRowTests.cs

src/csharp/Microsoft.Spark/Sql/SparkSession.cs

src/scala/microsoft-spark-3.0.x/src/main/scala/org/apache/spark/sql/api/dotnet/SQLUtils.scala

src/csharp/Microsoft.Spark/Sql/SparkSession.cs

src/scala/microsoft-spark-2.3.x/src/main/scala/org/apache/spark/api/dotnet/SerDe.scala

src/csharp/Microsoft.Spark.E2ETest/IpcTests/Sql/SparkSessionTests.cs

suhsteve

LGTM. Just have a few nits. Thanks @Niharikadutta

src/csharp/Microsoft.Spark.E2ETest/IpcTests/Sql/SparkSessionTests.cs

src/csharp/Microsoft.Spark/Sql/SparkSession.cs

suhsteve · 2020-01-09T17:54:48Z

Looks like tests failed. Can you fix those ?

Niharikadutta · 2020-01-10T00:47:30Z

@imback82 Could you please review this PR? Thanks!

imback82

I have few minor comments, but generally looks good.

src/csharp/Microsoft.Spark/Sql/Row.cs

src/csharp/Microsoft.Spark.E2ETest/IpcTests/Sql/SparkSessionTests.cs

…a/spark into nidutta/RowTypeSerDeSupport

imback82

LGTM. Thanks @Niharikadutta!

Niharikadutta added 9 commits November 14, 2019 17:26

Initial commit for Row Ser/De and exposing CreateDataFrame

044a01e

Reverting changes to Basic.cs

b1d8dfc

Merge branch 'master' into nidutta/RowTypeSerDeSupport

5c7dd03

Added example in Basic.cs about how to use CreateDataFrame

b6b82be

Removing unneccessary comments

ae31dd2

Removing whitespace

9546229

Added GenericRow support and some tests, removed comments and added C…

ca094d7

…reateDataFrame that takes in strings

Exposing GenericRow tests and cleaned up comments.

3c58aa7

Made GenericRow as internal

74ff143

Niharikadutta changed the title ~~[WIP] Support for Row type Ser/De and exposing the CreateDataFrame API~~ Support for Row type Ser/De and exposing the CreateDataFrame API Dec 10, 2019