Allow the project to be built with Java 16 #25171

dreis2211 · 2021-02-10T15:04:35Z

Hi,

this PR gets us one step further into supporting JDK 16 (see #24402).

I'm at a stage where I have random failures on my machine that seem related to network errors & timeouts, so I'm both confident and desperate enough to share my first draft.

Essentially, the PR replaces the functionality of buildJavaHome with a new property toolchainVersion that controls whether or not an optional toolchain should be configured for compilation, javadoc and test runs.

I should note that the spring-boot-gradle-plugin has multiple problems. None of the current versions support JDK 16 and thus a lot of tests fail. Unfortunately, you can't provide an empty stream in GradleCompatibilityExtension.provideTestTemplateInvocationContexts. I had some hacky code that would workaround that but I figured that a simple exclude would suffice for the time being. E.g. I used the following:

./gradlew build -x :spring-boot-project:spring-boot-tools:spring-boot-gradle-plugin:test -x :spring-boot-project:spring-boot-tools:spring-boot-gradle-plugin:validatePlugins -PtoolchainVersion=16

As you can see, I excluded the validatePlugins task as well, which is failing due to gradle/gradle#15538.

The big issue is something else though. With JDK 16 and more specifically JEP-396 we have strong encapsulation enabled by default. That breaks a lot of tests and libraries. Just to name a few:

Any AbstractServletWebServerFactoryTests due to the URL.factory field being reset.
CliTester for a similar reason
JsonbTesterTests due to a bug in Johnzon
A couple of tests related to LDAP because of AbstractContextSource cannot be instantiated on JDK16+ spring-ldap#570
...

I decided to add --illegal-access=warn so we get warned, but at the same time have access allowed for the time being.

Feel free to test around with this PR. Again - I didn't have a single test run today that was green, but all test failures were random and successful when run in isolation.

Next steps

If this PR is merged eventually, I would tackle the actual pipeline. I thought this might be a bit much for this PR, given that I need to add a secondary JDK to the CI images. (I can hopefully reuse some logic though that I build in the past for Boot already)

Let me know what you think.
Christoph

See spring-projectsgh-24402

dreis2211 · 2021-02-10T15:42:34Z

settings.gradle

@@ -31,10 +31,6 @@ rootProject.name="spring-boot-build"
 settings.gradle.projectsLoaded {
 	gradleEnterprise {
 		buildScan {
-			if (settings.gradle.rootProject.hasProperty('buildJavaHome')) {


The idea/hope is that the build scans show the actual toolchain, but we'll see. Couldn't test this really.

Here's a scan against 2.4.x with -PtoolchainVersion=16: https://ge.spring.io/s/yadj75gujofi6

I wonder why 2.4.x is targeted here? I guess we would need Groovy 3 as well in the 2.4.x mainline to make this work. See #24946

Support for Java 16 starts with Spring Framework 5.3 which is what Spring Boot 2.4+ uses. We need a GA release of Spring Boot that folks using Java 15 could rely on once 16 is out (given that 15 will be effectively EOL at that time).

As for Groovy 3, perhaps we could upgrade to it on the Java 16 build?

I think I would need a little help on getting this to work on 2.4.x - I currently don't see a good way to do this sort of conditional dependency upgrade

Well, this would only happen for the actual pipelines e.g. via -x :spring-boot-project:spring-boot-cli:test. I wouldn't know what I should update in this PR - cluttering every CLI test with @DisabledForJreRange seems like an overkill.

I'd rather find a way to disable tests for the CLI module altogether with Java 16. When we had a Maven build we used profiles to disable Surefire. I guess Gradle should have a similar feature.

If you know one that is also aware of the toolchain version, that would be great. We could also use that for the gradle plugin then. I just don't know how we could exclude a whole module based on the toolchain version - apart from coming up with our own way. Let me know if I should experiment around that.

I think we'll have to come up with our own way. Alternatively, we could use @DisabledForJreRange all over the place. One advantage of that is the tests are disabled no matter how they're launched.

I'm currently writing a custom plugin that lets you specify the maximum compatible java version inside the build.gradle for that particular submodule. The plan is that this information is consumed in order to dynamically disable the things that run with the optional toolchain.

wilkinsona · 2021-02-11T09:37:06Z

Thanks very much, @dreis2211. Here's a build scan for the proposed changes: https://ge.spring.io/s/ovgtv4h7lapkc.

The failures in the CLI appear to be due to Groovy not supported Java 16 byte code. The JarCommandIT tests appear to be failing due to Groovy's version of ASM not supporting Java 16 byte code. It's not apparent from the test's output, but running in my IDE revealed the following output from the jar command:

Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 60
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:196)
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:177)
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:163)
	at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:284)
	at org.codehaus.groovy.ast.decompiled.AsmDecompiler.parseClass(AsmDecompiler.java:81)
	at org.codehaus.groovy.control.ClassNodeResolver.findDecompiled(ClassNodeResolver.java:251)
	at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:189)
	at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:169)
	at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:125)
	at org.codehaus.groovy.ast.decompiled.AsmReferenceResolver.resolveClassNullable(AsmReferenceResolver.java:57)
	at org.codehaus.groovy.ast.decompiled.Annotations.createAnnotationNode(Annotations.java:40)
	at org.codehaus.groovy.ast.decompiled.DecompiledClassNode.addAnnotations(DecompiledClassNode.java:268)
	at org.codehaus.groovy.ast.decompiled.DecompiledClassNode.lazyInitSupers(DecompiledClassNode.java:184)
	at org.codehaus.groovy.ast.decompiled.DecompiledClassNode.getInterfaces(DecompiledClassNode.java:98)
	at org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:375)
	at org.codehaus.groovy.ast.ClassNode.getAllInterfaces(ClassNode.java:438)
	at org.codehaus.groovy.ast.ClassNode.getAllInterfaces(ClassNode.java:440)
	at org.codehaus.groovy.ast.ClassNode.getAllInterfaces(ClassNode.java:430)
	at org.codehaus.groovy.control.ResolveVisitor.setRedirect(ResolveVisitor.java:526)
	at org.codehaus.groovy.control.ResolveVisitor.resolveNestedClass(ResolveVisitor.java:485)
	at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:462)
	at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:428)
	at org.codehaus.groovy.control.ResolveVisitor.resolveOrFail(ResolveVisitor.java:343)
	at org.codehaus.groovy.control.ResolveVisitor.visitAnnotations(ResolveVisitor.java:1330)
	at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:51)
	at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1465)
	at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:230)
	at org.codehaus.groovy.control.CompilationUnit$13.call(CompilationUnit.java:700)
	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:965)
	... 17 more

Support for Java 16 has landed in ASM but it has not yet been released. Framework is avoiding this problem by patching its own version of ASM to pull in the latest changes prior to them being released. The ASM release with Java 16 is expected to be "at the same time the JDK 16 will be released" so we may have to live with this for a while by skipping those tests on Java 16.

dreis2211 · 2021-02-11T09:39:48Z

@wilkinsona Thanks. I'm wondering why the test you mentioned didn't fail on my machine.

But seeing no indication of the toolchain, I think I will bring back a custom value.

wilkinsona · 2021-02-11T09:43:40Z

The ASM release with Java 16 is expected to be "at the same time the JDK 16 will be released" so we may have to live with this for a while by skipping those tests on Java 16.

Things seem to have happened more quickly on the ASM side than the ticket linked to above suggested. According to https://asm.ow2.io/versions.html, ASM 9.0 – that includes Java 16 support – was released in September.

wilkinsona · 2021-02-11T09:45:17Z

But seeing no indication of the toolchain, I think I will bring back a custom value.

I couldn't see any indication either. I raised it with the Gradle team and they confirmed it's not yet surfaced in the builds scans. They suggested using custom values for now.

wilkinsona · 2021-02-11T09:59:38Z

I'm wondering why the test you mentioned didn't fail on my machine.

Me too. Turns out there's a bug in CommandLineInvoker. If there's more than one zip in the build output it'll use the first one it finds which may not be the right one. As a result, the tests were using 2.4's CLI with Groovy 2.5.x. They pass with 2.5's CLI and Groovy 3.0.x.

See spring-projectsgh-24402

dreis2211 · 2021-02-11T12:59:52Z

@wilkinsona Thanks for checking that. I indeed did some clean runs in between.

I added the custom toolchain value to the scans. Let me know if you miss anything on this PR still.

dreis2211 · 2021-02-22T11:59:00Z

@philwebb I saw you changed the title, but I'm personally not comfortable with saying "Support Java 16" just yet. The issues over at spring-projects/spring-ldap#570 and the Gradle Plugin not being supported at all maybe gives the wrong impression to users that read the changelog and see that. In the end, it's more "Support Java 16 builds" and one step further into supporting Java 16.

wilkinsona · 2021-02-22T12:11:20Z

We're happy with stating that Spring Boot itself supports Java 16, even if that's not true across the board. For example, our Java 9 support excluded Cassandra for several months. If necessary, we can do the same with Spring LDAP and Gradle. The latter's really not too bad as users can still use Java 16 at runtime as long as they run Gradle itself on an earlier JVM.

dreis2211 · 2021-02-22T12:17:47Z

Fair enough...

See spring-projectsgh-24402

dreis2211 · 2021-02-23T18:53:46Z

@wilkinsona @snicoll I've pushed a revised version that extracts the toolchain configuration into its own plugin/extension. That felt cleaner overall, but let me know what you think of it.

The custom plugin + extension has the possibility to set a maximum compatible java version and disables the toolchain based tasks if it doesn't match (see the build.gradle of spring-boot-cli for an example) The positive side-effect of this is that there is no need for custom task exclusions via ./gradlew build -x excludedTask anymore, which will be benefitial for the overall pipeline complexity.

See gh-25171

wilkinsona · 2021-03-04T19:16:12Z

This is great. Thanks very much, @dreis2211. I particularly like the new approach to configuring the toolchain stuff via the plugin and extension.

wilkinsona · 2021-03-04T19:18:11Z

Here is a scan for a 2.4.x build using a Java 16 toolchain.

Support running builds with custom toolchain version

ffb0dbe

See spring-projectsgh-24402

spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Feb 10, 2021

dreis2211 commented Feb 10, 2021

View reviewed changes

snicoll added the for: team-attention An issue we'd like other members of the team to review label Feb 11, 2021

dreis2211 added 2 commits February 11, 2021 13:54

Set custom toolchain value inside build scans

289f889

See spring-projectsgh-24402

Update copyright of changed file

da50fce

philwebb added this to the 2.5.x milestone Feb 19, 2021

philwebb added type: enhancement A general enhancement and removed status: waiting-for-triage An issue we've not yet triaged type: enhancement A general enhancement labels Feb 19, 2021

philwebb changed the title ~~Support running builds with custom toolchain version~~ Support Java 16 Feb 19, 2021

philwebb modified the milestones: 2.5.x, 2.4.x Feb 19, 2021

philwebb removed the for: team-attention An issue we'd like other members of the team to review label Feb 19, 2021

Extract toolchain support into its own plugin

534aab1

See spring-projectsgh-24402

wilkinsona self-assigned this Mar 4, 2021

wilkinsona changed the title ~~Support Java 16~~ Allow the project to be built with Java 16 Mar 4, 2021

wilkinsona added type: task A general task and removed type: enhancement A general enhancement labels Mar 4, 2021

wilkinsona mentioned this pull request Mar 4, 2021

Allow the project to be built with Java 16 #25497

Closed

wilkinsona pushed a commit that referenced this pull request Mar 4, 2021

Allow the project to be built with Java 16

1ccd8da

See gh-25171

wilkinsona added a commit that referenced this pull request Mar 4, 2021

Polish "Allow the project to be built with Java 16"

1ce6e79

See gh-25171

wilkinsona closed this in 5235047 Mar 4, 2021

wilkinsona modified the milestones: 2.4.x, 2.4.4 Mar 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow the project to be built with Java 16 #25171

Allow the project to be built with Java 16 #25171

dreis2211 commented Feb 10, 2021

dreis2211 Feb 10, 2021

snicoll Feb 23, 2021

dreis2211 Feb 23, 2021 •

edited

Loading

snicoll Feb 23, 2021 •

edited

Loading

dreis2211 Feb 23, 2021

dreis2211 Feb 23, 2021

snicoll Feb 23, 2021

dreis2211 Feb 23, 2021

wilkinsona Feb 23, 2021

dreis2211 Feb 23, 2021

wilkinsona commented Feb 11, 2021

dreis2211 commented Feb 11, 2021 •

edited

Loading

wilkinsona commented Feb 11, 2021

wilkinsona commented Feb 11, 2021

wilkinsona commented Feb 11, 2021 •

edited

Loading

dreis2211 commented Feb 11, 2021

dreis2211 commented Feb 22, 2021

wilkinsona commented Feb 22, 2021

dreis2211 commented Feb 22, 2021

dreis2211 commented Feb 23, 2021 •

edited

Loading

wilkinsona commented Mar 4, 2021

wilkinsona commented Mar 4, 2021

Allow the project to be built with Java 16 #25171

Allow the project to be built with Java 16 #25171

Conversation

dreis2211 commented Feb 10, 2021

Next steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dreis2211 Feb 23, 2021 • edited Loading

Choose a reason for hiding this comment

snicoll Feb 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wilkinsona commented Feb 11, 2021

dreis2211 commented Feb 11, 2021 • edited Loading

wilkinsona commented Feb 11, 2021

wilkinsona commented Feb 11, 2021

wilkinsona commented Feb 11, 2021 • edited Loading

dreis2211 commented Feb 11, 2021

dreis2211 commented Feb 22, 2021

wilkinsona commented Feb 22, 2021

dreis2211 commented Feb 22, 2021

dreis2211 commented Feb 23, 2021 • edited Loading

wilkinsona commented Mar 4, 2021

wilkinsona commented Mar 4, 2021

dreis2211 Feb 23, 2021 •

edited

Loading

snicoll Feb 23, 2021 •

edited

Loading

dreis2211 commented Feb 11, 2021 •

edited

Loading

wilkinsona commented Feb 11, 2021 •

edited

Loading

dreis2211 commented Feb 23, 2021 •

edited

Loading