[Feature] Write your feature request here!🚀

[kira-lin]

Here we will maintain a list of requested features, to make it easier to track.

• 1. Add custom resource requirements for RayDP executor actors. This has been implemented for RayDP master actors thanks to @pang-wu . Related issue Having worker nodes specific for Spark jobs #119
• 2. Release conda package. Related issue conda-forge package? #124
• 3. Support Ray Dataset Pipelining with Spark Dataframe
• 4. Support streaming. Spark Streaming/Flink/Kafka? Need more use cases. Related issue Integrate Spark Streaming with Ray/Ray Datasets #213 请问本项目是否有计划支持Flink #291
• 5. Provide a guide for using RayDP in Ray on Kubernetes. Related issue Failed validating 'additionalProperties' in schema when using Python 3.8 for docker example #305
• 6. Integration with Spark native SQL engine Gluten

Feel free to write yours below! Also welcome to share your use case or scenario of RayDP, this will help us better understand your feature request, and how to further optimize performance, thanks!

[pang-wu]

@kira-lin Is it possible to make the point you listed with numbers so we can up vote?
I would vote for No. 1, 3, 4

[kira-lin]

@kira-lin Is it possible to make the point you listed with numbers so we can up vote? I would vote for No. 1, 3, 4

Thanks, good idea!

[kira-lin]

@pang-wu Regarding to issue No. 4, are you currently dealing with streaming data in your workflow? What tool should RayDP provide in your expectation?