Investigate and potentially add support for spark connect

[razvan]

Spark Connect

Spark 3.5 introduces a new client called Spark Connect.

The use case seems to be thin clients that connect to a running spark driver.

This probably means that the operator needs to be able to start spark connect servers without spark applications and publish a service for "connect" clients.

Roadmap

Rough roadmap to GA:

• POC: can set up a spark-connect server with kubernetes as resource manager, basic integration test
• minimal CRD: drop the stateful set, minimum configuration for the server (jvm props, logging)
  • server
    • deployment with one replica
    • jvm arg overrides
    • config overrides
    • env overrides
    • log configuration and aggregation with vector
    • pod overrides
    • resource requests
    • status and transition events
    • reconciliation operation (paused, stopped, etc)
  • executor
    • jvm arg overrides
    • config overrides
    • env overrides
    • log configuration and aggregation
    • resource requests
    • pod affinity
• add preliminary documentation
• expose Prometheus metrics
• integrate with the history server See: doc: comment on spark history integration #559
• integrate with the listener op
• create a new demo

Related PRs

• feat: add support for SparkConnect #539
• fix(spark-k8s): refactor for Spark Connect docker-images#1034
• feat: Add Deployments to ClusterResources operator-rs#992
• feat: add DeploymentConditionBuilder operator-rs#993
• feat: add spark connect client to image build flows docker-images#1051
• doc: comment on spark history integration #559
• feat: expose services with listener classes #562

[timrobertson100]

We have started exploring Spark Connect at GBIF.org. Our primary use case is to explore having a long running spark cluster hold an in memory cached table, for apps to do filtered data egress with minimal startup cost.

[timrobertson100]

@razvan - thank you for your work. When you are ready, we will be interested in helping to test.

[razvan]

@timrobertson100 - we merged preliminary support for spark connect deployments in the main branch. Looking forward for your feedback!

[dshershov]

Awesome! Good job guys!

[lfrancke]

@razvan This is in "Done" but not closed. Anything left to do?

Could you please add release notes and a link to the docs?

[razvan]

This release adds experimental support for Spark Connect. The Spark operator watches for SparkConnectServer custom resources. Preliminary documentation is available here [1] and the existing Taxi Data Anomaly Detection demo [2] has been retrofitted to use a JupterLab client running against a Spark Connect server.

[1] https://docs.stackable.tech/home/nightly/spark-k8s/usage-guide/spark-connect/
[2] https://docs.stackable.tech/home/nightly/demos/jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data/

[timrobertson100]

Thank you very much. We have been watching but have not yet had time to test - it's on our backlog.