[FEA] spark 3.5.x use at least java 11 to build

[liurenjie1024]

Is your feature request related to a problem? Please describe.
See this github action build

Iceberg 1.7.2 requires at least java 11, while we are currently still using java 8 for ci.

Describe the solution you'd like
Upgrade ci jdk to at least java 11 for spark 3.5.x, since iceberg currently only support 3.5.x.

Describe alternatives you've considered
No.

Additional context

[liurenjie1024]

cc @pxLi PTAL, this is blocking #12921

[pxLi]

cc @pxLi PTAL, this is blocking #12921

We currently maintain separate JDK requirements across our scala releases: JDK8 for Scala 2.12 (spark32x-35x) and JDK17 for Scala 2.13 (spark33x-40x). Any changes to our minimum supported JDK version would require production-level approval and consideration. cc @sameerz @GaryShen2008 @mattahrens

[gerashegalov]

We can separate the issue into two parts.

1. What JDK version to use to build the plugin artifacts?
2. What JDK version to use to run and test the plugin, based on what min JDK version is the JDK release target defined in 1)

Currently, regardless of the Build JDK (1) we produce the classfiles runnable with JDK8 by default java.major.version.
JDK17 is used to be able to have the Scala2.13 multi-shim build work with 2.13 Spark 4.0.0 artifacts with minJDK=17 at the same time as with 2.13 Spark 3.5.6 build with minJDK=8

Our users can use the same dist jar for running the plugin with Spark 3.5.6 Scala 2.13 on JDK 8 and Spark 4.0.0 Scala 2.13 on JDK 17.

For our Iceberg 1.7+ users Iceberg is already a forcing function to use JDK11+ to run their Spark apps regardless of the Plugin. For Spark 4.0.0 users JDK17+ is already required regardless. We can reinforce in our documentation.

Thus, we can simplify our build requirements by enforcing JDK 17 for building of both 2.12 and 2.13 Shim builds uniformly for all supported Spark builds

[liurenjie1024]

I agree with @gerashegalov that we can build with jdk 17, produce class files for jdk 8, and test against different jdk version. For iceberg test, we could skip it jdk version is less than 11.

One concern with this approach is that if we used an api that's included in jdk 17, but not in jdk 8, then user may still experience error when running with jdk 8.

[pxLi]

We can separate the issue into two parts.

1. What JDK version to use to build the plugin artifacts?
2. What JDK version to use to run and test the plugin, i.e. what JDK version is the JDK release target defined in 1)

Currently, regardless of the Build JDK (1) we produce the classfiles runnable with JDK8 by default java.major.version. JDK17 is used to be able to have the Scala2.13 multi-shim build work with 2.13 Spark 4.0.0 artifacts with minJDK=17 at the same time as with 2.13 Spark 3.5.6 build with minJDK=8

Our users can use the same dist jar for running the plugin with Spark 3.5.6 Scala 2.13 on JDK 8 and Spark 4.0.0 Scala 2.13 on JDK 17.

For our Iceberg 1.7+ users Iceberg is already a forcing function to use JDK11+ to run their Spark apps regardless of the Plugin. For Spark 4.0.0 users JDK17+ is already required regardless. We can reinforce in our documentation.

Thus, we can simplify our build requirements by enforcing JDK 17 for building of both 2.12 and 2.13 Shim builds uniformly for all supported Spark builds

@gerashegalov @liurenjie1024

Given the significant effort required to update our default JDK build version across the multiple projects and CI setups, would it be feasible to enable iceberg 1.7+ support specifically for the scala2.13 artifacts as an interim solution?

This approach aligns with spark's plan to discontinue scala2.12 release after 4.0+ and prevents the need for all stakeholders to update their environments.

[liurenjie1024]

Given the significant effort required to update our default JDK build version across the multiple projects and CI setups, would it be feasible to enable iceberg 1.7+ support specifically for the scala2.13 artifacts as an interim solution?

It depends on how urgent we upgrade to iceberg 1.7+. cc @sameerz @gerashegalov @GaryShen2008

[gerashegalov]

One concern with this approach is that if we used an api that's included in jdk 17, but not in jdk 8, then user may still experience error when running with jdk 8.

This scenario is impossible with the current build. --release 8 option would fail it already at the build time.