CompletableFuture create client triggered a deadlock

[DDAaTao]

Describe the bug
I have a demand to get two FeignClient data when Spring starts up. For various reasons, one of the FeignClients I used to start CompletableFuture asynchronously resulted in a deadlock situation.

I can avoid this problem by changing my code logic, but I don't understand why the feignClient load timing is designed this way ( like #1334 ), and what is the deeper reason for this deadlock.

A simple way to avoid deadlocks - Thread.sleep(3000);

Sample code - Jdk8 spring-cloud-context:2.1.6.RELEASE

package com.siiri.cts.open.biz.config;

import com.siiri.cts.open.biz.feign.AccountFeignClient;
import com.siiri.cts.open.biz.feign.OrganizationFeignClient;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.concurrent.CompletableFuture;

/**
 * @author van
 */
@Slf4j
@Configuration
public class TestDeadLockConfig {

    @Autowired
    private OrganizationFeignClient organizationFeignClient;
    @Autowired
    private AccountFeignClient accountFeignClient;

    @Bean
    public FwtTestBean initBean()  {
        CompletableFuture.runAsync(() -> {
            log.info("=========organizationFeignClient 启动前");   // Thread.sleep(3000) can avoid this deadlock
            organizationFeignClient.getPersonOne("1");
            log.info("=========organizationFeignClient 启动后");
        });

        log.info("=== after organizationFeignClient 启动前");
        accountFeignClient.getByAccountName("fwt");
        log.info("==== after organizationFeignClient 启动后");

        return new FwtTestBean(1, "fwt");
    }
}

Sample code - jstack dump

Found one Java-level deadlock:
=============================
"ForkJoinPool.commonPool-worker-2":
  waiting to lock monitor 0x000000002c456c18 (object 0x00000005c2284aa8, a java.util.concurrent.ConcurrentHashMap),
  which is held by "main"
"main":
  waiting to lock monitor 0x000000002bda1db8 (object 0x000000071649d0e0, a java.util.concurrent.ConcurrentHashMap),
  which is held by "ForkJoinPool.commonPool-worker-2"

Java stack information for the threads listed above:
===================================================
"ForkJoinPool.commonPool-worker-2":
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:179)
        - waiting to lock <0x00000005c2284aa8> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:493)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:520)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:491)
        at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:227)
        at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:231)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1419)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1218)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1175)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostPro
cessor.java:595)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.
java:376)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1404)      
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:592)       
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$162/2081000371.getObject(Unknown Source)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        - locked <0x000000071a98c5a8> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
        - locked <0x000000071a98c200> (a java.lang.Object)
        at org.springframework.cloud.context.named.NamedContextFactory.createContext(NamedContextFactory.java:136)
        at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:101)
        - locked <0x000000071649d0e0> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getContext(SpringClientFactory.java:131)
        at org.springframework.cloud.context.named.NamedContextFactory.getInstance(NamedContextFactory.java:145)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getInstance(SpringClientFactory.java:121)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getClientConfig(SpringClientFactory.java:75)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.getClientConfig(LoadBalancerFeignClient.java:97)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:81)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:110)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
        at com.sun.proxy.$Proxy154.getPersonOne(Unknown Source)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig.lambda$initBean$0(TestDeadLockConfig.java:28)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$Lambda$565/2000449863.run(Unknown Source)
        at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1626)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java)
        at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1618)
        at java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"main":
        at org.springframework.cloud.context.named.NamedContextFactory.getContext(NamedContextFactory.java:100)
        - waiting to lock <0x000000071649d0e0> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getContext(SpringClientFactory.java:131)
        at org.springframework.cloud.context.named.NamedContextFactory.getInstance(NamedContextFactory.java:145)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getInstance(SpringClientFactory.java:121)
        at org.springframework.cloud.netflix.ribbon.SpringClientFactory.getClientConfig(SpringClientFactory.java:75)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.getClientConfig(LoadBalancerFeignClient.java:97)
        at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:81)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:110)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
        at com.sun.proxy.$Proxy153.getByAccountName(Unknown Source)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig.initBean(TestDeadLockConfig.java:33)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$EnhancerBySpringCGLIB$$3776b5f.CGLIB$initBean$0(<generated>)
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$EnhancerBySpringCGLIB$$3776b5f$$FastClassBySpringCGLIB$$bc6c3884.invoke(<generated>)       
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)       
        at com.siiri.cts.open.biz.config.TestDeadLockConfig$$EnhancerBySpringCGLIB$$3776b5f.initBean(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:456)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory
.java:1320)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1159)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)       
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$162/2081000371.getObject(Unknown Source)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        - locked <0x00000005c2284aa8> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
        - locked <0x00000005c23cb290> (a java.lang.Object)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1204)
        at com.siiri.cts.open.web.OpenServiceWebApplication.main(OpenServiceWebApplication.java:42)

Found 1 deadlock.

[OlgaMaciaszek]

Duplicate of spring-cloud/spring-cloud-openfeign#475. Also, it's an issue in OpenFeign, not in commons.

[DDAaTao]

I noticed that the corresponding issue for openfeign was always open, so I wondered if there was another solution.

I think there are two ways to solve this problem（deadlock）, one is from Openfeign to deal with load timing, and the other is from Commons to try to solve the problem of asynchronously initializing LoadBalancer.

Just speculating. welcome to point out mistakes. Thanks

@OlgaMaciaszek

[OlgaMaciaszek]

Hello @DDAaTao we thought this problem would be best solved in OF, but if you have a different idea, definitely feel free to create a draft - we'll take a look and discuss it here. (please make sure to tag me in any replies here, cause since the issue is closed, I might not notice it otherwise).