Asynchronous deserialization performed by Hazelcast may fail due to the wrong ClassLoader being used

[dsyer]

I discovered this in a long yak-shaving session migrating an app to Boot 2.4.1 and Hazelcast 4.0.3.

This fails when you run it from the jar:

@SpringBootApplication
public class IssueApplication {
	public static void main(String[] args) {
		SpringApplication.run(IssueApplication.class, args);
	}
	@Bean
	CommandLineRunner runner(HazelcastInstance hazelcast) {
		IMap<String, Foo> sessions = hazelcast.getMap("foos");
		return args -> {
			sessions.set("foo", new Foo("foo"));
			System.err.println(sessions.get("foo"));
			sessions.getAsync("foo").whenComplete((u, t) -> System.err.println(u)).toCompletableFuture().get();
		};
	}
}
class Foo implements Serializable {
	private String value;
	public Foo() {
	}
	public Foo(String value) {
		this.value = value;
	}
	public String getValue() {
		return this.value;
	}
	public void setValue(String value) {
		this.value = value;
	}
	@Override
	public String toString() {
		return "Foo [value=" + this.value + "]";
	}
}

It's fine when you run in an IDE, or with java -classpath .... The problem is that Hazelcast uses the Thread.crrentThread().getContextClassLoader() by default, and in the async background thread this is the JarLauncher not the AppClassLoader.

I worked around it with this

		@Bean
		Config hazelcastConfig() {
			Config config = new Config();
			config.setClassLoader(SpringApplication.class.getClassLoader());
			...
			return config;
		}

but the issue is more generic really - probably any library that uses JDK fork-join utilities will end up with the same class loader.

[wilkinsona]

The problem is that Hazelcast uses the Thread.crrentThread().getContextClassLoader() by default, and in the async background thread this is the JarLauncher not the AppClassLoader.

I assume that JarLauncher here should be LaunchedURLClassLoader. I think this may also be the wrong way round. In a Boot app you want the TCCL to be the LaunchedURLClassLoader as this is the class loader the can load the application's classes and those of its dependencies. The AppClassLoader can only see the launcher and the JDK.

Unfortunately, the JDK's common fork-join pool uses the app class loader as its TCCL. This makes it unsuitable for use in a packaged Boot application and, I suspect, in any environment with a custom class loader such as a Servlet container. We've seen this problem before (#15737) and, in general terms, I don't think we can do anything about it.

In this specific case, it looks like the auto-configuration could be improved by setting the class loader automatically (assuming that's possible while still honouring the rest of the user's configuration).

[dsyer]

Hmm. I think maybe there is still something in Hazelcast which we could ask them to fix.

This app works fine (IDE and JAR) when Hazelcast is not on the classpath:

@SpringBootApplication
public class IssueApplication {

	public static void main(String[] args) {
		SpringApplication.run(IssueApplication.class, args);
	}

	@Bean
	CommandLineRunner runner() {
		return args -> {
			ForkJoinPool.commonPool().submit(() -> {
				System.err.println(ClassUtils.resolveClassName(Foo.class.getName(), null));
			}).get();
		};
	}

}
...

and then it fails from a JAR if you just put Hazelcast on the classpath but never use it:

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-01-14 14:46:30.747 ERROR 1000765 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.IllegalStateException: Failed to execute CommandLineRunner
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:807) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:788) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1311) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1300) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at com.example.IssueApplication.main(IssueApplication.java:16) ~[classes!/:0.0.1-SNAPSHOT]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:107) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Could not find class [com.example.Foo]
	at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006) ~[na:na]
	at com.example.IssueApplication.lambda$runner$1(IssueApplication.java:24) ~[classes!/:0.0.1-SNAPSHOT]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Could not find class [com.example.Foo]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600) ~[na:na]
	... 16 common frames omitted
Caused by: java.lang.IllegalArgumentException: Could not find class [com.example.Foo]
	at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:334) ~[spring-core-5.3.3.jar!/:5.3.3]
	at com.example.IssueApplication.lambda$runner$0(IssueApplication.java:23) ~[classes!/:0.0.1-SNAPSHOT]
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[na:na]
Caused by: java.lang.ClassNotFoundException: com.example.Foo
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[na:na]
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
	at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
	at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
	at org.springframework.util.ClassUtils.forName(ClassUtils.java:284) ~[spring-core-5.3.3.jar!/:5.3.3]
	at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:324) ~[spring-core-5.3.3.jar!/:5.3.3]
	... 7 common frames omitted

Adding Hazelcast to the classpath (and an empty hazelcast.xml) causes our autoconfiguration to be created, which in turn seems to mess with the default ForkJoinPool.

[wilkinsona]

That's odd. The common pool uses DefaultForkJoinWorkerThreadFactory which "creates a new ForkJoinWorkerThread using the system class loader as the thread context class loader". I don't know how Hazelcast could be messing with that. Sounds like some digging is required.

[dsyer]

This is even weirder. If you have a 2 node cluster it doesn't fail from the JAR. It only fails with one node.

[dsyer]

That's rubbish, sorry. Brain fart probably. It always fails from the JAR for the reason you said (the system class loader is the wrong one).

UPDATE: it sometimes fails. Especially with Java 8 (as opposed to 11, where ForkJoinPool always uses the system class loader).

[dsyer]

So maybe we need to set the class loader in Hazelcast Config in our autoconfig?

[wilkinsona]

Yep, I think so. Now we need to figure out how to do that across all the various configuration options that we support.

[dsyer]

I think they all end up in a Config object somehow. There's a utility in HC, or something, for loading the XML. And the Config is mutable, so we can just create it in whatever way we find the user is trying to suggest and set the class loader. Maybe?