ClassNotFoundException when calling a FeignBlockingLoadBalancerClient in a parallel stream (multiple threads)

[didiez]

Describe the bug
When calling a @FeignClient (FeignBlockingLoadBalancerClient) in a parallel stream (stream.parallel()), the following error arises, only when executing it as an uber/executable jar.

It seems to be related (kind of) to some other ClassLoader issues (spring-projects/spring-boot#24836, spring-projects/spring-boot#15737 )

2021-08-04 13:11:44.818 ERROR 93528 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.IllegalStateException: Failed to execute CommandLineRunner
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:794) ~[spring-boot-2.5.3.jar!/:2.5.3]
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:775) ~[spring-boot-2.5.3.jar!/:2.5.3]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:345) ~[spring-boot-2.5.3.jar!/:2.5.3]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-2.5.3.jar!/:2.5.3]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) ~[spring-boot-2.5.3.jar!/:2.5.3]
        at es.didiez.feignparallelclassloaderbug.Application.main(Application.java:16) ~[classes!/:0.0.1-SNAPSHOT]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[feign-parallel-classloader-bug-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[feign-parallel-classloader-bug-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[feign-parallel-classloader-bug-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[feign-parallel-classloader-bug-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Could not find class [org.springframework.boot.autoconfigure.condition.OnPropertyCondition]
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737) ~[na:na]
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) ~[na:na]
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[na:na]
        at es.didiez.feignparallelclassloaderbug.Application.lambda$run$1(Application.java:25) ~[classes!/:0.0.1-SNAPSHOT]
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:791) ~[spring-boot-2.5.3.jar!/:2.5.3]
        ... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: Could not find class [org.springframework.boot.autoconfigure.condition.OnPropertyCondition]
        at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:334) ~[spring-core-5.3.9.jar!/:5.3.9]
        at org.springframework.context.annotation.ConditionEvaluator.getCondition(ConditionEvaluator.java:124) ~[spring-context-5.3.9.jar!/:5.3.9]
        at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:96) ~[spring-context-5.3.9.jar!/:5.3.9]
        at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:88) ~[spring-context-5.3.9.jar!/:5.3.9]
        at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:71) ~[spring-context-5.3.9.jar!/:5.3.9]
        at org.springframework.context.annotation.AnnotatedBeanDefinitionReader.doRegisterBean(AnnotatedBeanDefinitionReader.java:254) ~[spring-context-5.3.9.jar!/:5.3.9]
        at org.springframework.context.annotation.AnnotatedBeanDefinitionReader.registerBean(AnnotatedBeanDefinitionReader.java:147) ~[spring-context-5.3.9.jar!/:5.3.9]
        at org.springframework.context.annotation.AnnotatedBeanDefinitionReader.register(AnnotatedBeanDefinitionReader.java:137) ~[spring-context-5.3.9.jar!/:5.3.9]
        at org.springframework.context.annotation.AnnotationConfigApplicationContext.register(AnnotationConfigApplicationContext.java:168) ~[spring-context-5.3.9.jar!/:5.3.9]
        at org.springframework.cloud.context.named.NamedContextFactory.createContext(NamedContextFactory.java:122) ~[spring-cloud-context-3.0.3.jar!/:3.0.3]
        at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:101) ~[spring-cloud-context-3.0.3.jar!/:3.0.3]
        at org.springframework.cloud.context.named.NamedContextFactory.getInstances(NamedContextFactory.java:181) ~[spring-cloud-context-3.0.3.jar!/:3.0.3]
        at org.springframework.cloud.openfeign.loadbalancer.FeignBlockingLoadBalancerClient.execute(FeignBlockingLoadBalancerClient.java:85) ~[spring-cloud-openfeign-core-3.0.3.jar!/:3.0.3]
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:119) ~[feign-core-10.12.jar!/:na]
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89) ~[feign-core-10.12.jar!/:na]
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100) ~[feign-core-10.12.jar!/:na]
        at es.didiez.feignparallelclassloaderbug.$Proxy60.getAlbum(Unknown Source) ~[na:0.0.1-SNAPSHOT]
        at es.didiez.feignparallelclassloaderbug.Application.lambda$run$0(Application.java:24) ~[classes!/:0.0.1-SNAPSHOT]
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[na:na]
        at java.base/java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177) ~[na:na]
        at java.base/java.util.stream.Streams$RangeLongSpliterator.forEachRemaining(Streams.java:228) ~[na:na]
        at java.base/java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:763) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[na:na]
        at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) ~[na:na]
        at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[na:na]
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) ~[na:na]
Caused by: java.lang.ClassNotFoundException: org.springframework.boot.autoconfigure.condition.OnPropertyCondition
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[na:na]
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
        at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
        at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
        at org.springframework.util.ClassUtils.forName(ClassUtils.java:284) ~[spring-core-5.3.9.jar!/:5.3.9]
        at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:324) ~[spring-core-5.3.9.jar!/:5.3.9]
        ... 29 common frames omitted

Sample
Sample application with the steps to reproduce the problem https://github.com/didiez/feign-parallel-classloader-bug

[OlgaMaciaszek]

Spring Cloud: 2020.0.3, Spring Boot: 2.5.3.

[OlgaMaciaszek]

@didiez Have you tried this with various jdks? Do you get this issue always or only while running with a specific jdk?

[didiez]

@didiez Have you tried this with various jdks? Do you get this issue always or only while running with a specific jdk?

Yes, the issue is reproducible at least with the following jdks:

• JDK 11: OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
• JDK 15: OpenJDK 64-Bit Server VM (build 15.0.3+3-Ubuntu-120.04, mixed mode, sharing)
• JDK 16: OpenJDK 64-Bit Server VM (build 16+36-2231, mixed mode, sharing)

[didiez]

As a hacky workaround, we have registered a custom LoadBalancerClientFactory like this:

@Bean
@ConditionalOnMissingBean
public LoadBalancerClientFactory loadBalancerClientFactory(ObjectProvider<List<LoadBalancerClientSpecification>> configurations) {
    LoadBalancerClientFactory clientFactory = new LoadBalancerClientFactory(){            
        @Override
        protected AnnotationConfigApplicationContext createContext(String name) {
            // FIXME: temporary switch classloader to use the correct one when creating the context
            ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader();
            Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());
            AnnotationConfigApplicationContext context = super.createContext(name);
            Thread.currentThread().setContextClassLoader(originalClassLoader);
            return context;
        }
    };

    clientFactory.setConfigurations(configurations.getIfAvailable(Collections::emptyList));
    return clientFactory;
}

[OlgaMaciaszek]

Ok, thanks for the update. This looks like a duplicate of #475.