[FEATURE REQUEST]: Foreach and Foreachbatch output sinks

[jatcar95]

I am unable to find a way to use Foreach and Foreachbatch sinks with .NET for Spark. Are these supported features, and if not, are there plans to support them in the future?

Here is the Spark documentation I'm referring to: http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch

[imback82]

@jatcar95 thanks for the suggestion. This is currently not supported, but we are open to support this feature depending on the community interests. (We are also very open to accept PRs to support this).

[josh-furman]

Would love to see this feature :)

[dannydwarren]

One of the struggles we are running into is joining new data in a Kafka stream with data from the beginning of the stream. We think we have a pattern in the Scala API proven out to write to Postgres via JDBC inside of the .foreachBatch then joining the batch with the Postgres table immediately after and reducing all the records for the key in Postgres down to a single record and publishing that reduced record to a different stream in Kafka.

We've spent the last day trying to do this in C#, but without the .foreachBatch option the C# API is lacking key functionality we need.

[AFFogarty]

Please assign to me.

[imback82]

@AFFogarty Can you share the initial proposal?

[imback82]

@suhsteve can you share the initial proposal when it's ready?