[AUDIT] test the RAPIDS plugin with the spark-connect plugin

[gerashegalov]

Spark connect modifies how jars are associated with the spark context / session apache/spark@caa3df4

Add tests ensuring, spark-rapids works with connect.

[razajafri]

@mattahrens @sameerz
How important do you guys think this feature is for 24.04?

[gerashegalov]

I presume the main driver for connect will be support for an LTS of Databricks 14 https://docs.databricks.com/en/release-notes/runtime/14.0.html. 14.3 LTS was released on Feb 1 https://docs.databricks.com/en/release-notes/runtime/index.html

[razajafri]

What about Apache Spark 3.5.1? Since 14.3 is branching off of it.

[gerashegalov]

What about Apache Spark 3.5.1? Since 14.3 is branching off of it.

Upstream users have a choice, whereas it does not look like Shared Cluster users can opt out https://docs.databricks.com/en/release-notes/runtime/14.0.html#introducing-spark-connect-in-shared-cluster-architecture

[sameerz]

We can revisit when we work on 4.0 or Databricks 14.x shims. For 3.5.1 it is not critical.

[gerashegalov]

Related user question #10611