Exception is thrown when stopping application

[ramunask]

Exception is thrown when stopping application through Tomcat Manager. Application is developed using Spring Boot application with enabled Elastic Micrometer and deployed as WAR to Tomcat.

Stack trace:

2020-03-24 18:12:27.563 [ajp-nio-8014-exec-142] WARN  i.m.c.u.i.l.AbstractInternalLogger.log (AbstractInternalLogger.java:122) - Failed to apply the value function for the gauge 'spring.integration.handlers'. Note that subsequent logs will be logged at debug level.
org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name '_org.springframework.integration.errorLogger.handler': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:208)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:608)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:596)
	at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1226)
	at org.springframework.integration.config.IntegrationManagementConfigurer.lambda$registerComponentGauges$2(IntegrationManagementConfigurer.java:412)
	at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:54)
	at io.micrometer.elastic.ElasticMeterRegistry.writeGauge(ElasticMeterRegistry.java:298)
	at io.micrometer.core.instrument.Meter.match(Meter.java:107)
	at io.micrometer.elastic.ElasticMeterRegistry.lambda$publish$2(ElasticMeterRegistry.java:194)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$SubList$2.forEachRemaining(ArrayList.java:1510)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at io.micrometer.elastic.ElasticMeterRegistry.publish(ElasticMeterRegistry.java:206)
	at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafely(PushMeterRegistry.java:48)
	at io.micrometer.core.instrument.push.PushMeterRegistry.close(PushMeterRegistry.java:83)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:339)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:273)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:571)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:543)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1040)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:504)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1033)
	at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1057)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1026)
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:975)
	at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:517)
	at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:112)
	at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4730)
	at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5396)
	at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
	at org.apache.catalina.manager.ManagerServlet.undeploy(ManagerServlet.java:1521)
	at org.apache.catalina.manager.HTMLManagerServlet.undeploy(HTMLManagerServlet.java:660)
	at org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:219)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:136)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:643)
	at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:348)
	at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:53)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:394)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1579)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:834)

[shakuzen]

This looks like another variety of spring-projects/spring-boot#19557 where the gauge function for spring.integration.handlers gets beans from the application context, which isn't generally allowed in a destroy method. The final publish of the registry is happening on application shutdown as part of the meter registry's destroy method.

While this is annoying and I'd like to improve the situation, at worst your last publish of metrics on shutdown fails, which is not generally critical.

[ramunask]

@shakuzen, thanks for explanation. Yes, it's not critical and let's hope it will be fixed in Spring Integration project :)

[renukeswarchinta-rmal]

This looks like another variety of spring-projects/spring-boot#19557 where the gauge function for spring.integration.handlers gets beans from the application context, which isn't generally allowed in a destroy method. The final publish of the registry is happening on application shutdown as part of the meter registry's destroy method.

While this is annoying and I'd like to improve the situation, at worst your last publish of metrics on shutdown fails, which is not generally critical.

``
I have spring boot application and run as command line runner application. it starts and do specific and closes. In this application we are trying to push metrics to Cloudwatch. Iam @ServiceActuator for messaging(cloud/onpremise) .
I get same error while shutting the down the application and metrics are not published to cloudwatch. Below is the stacktrace.

2021-04-08 20:30:13,559 DEBUG [restartedMain] [] [] service.impl.ImageDeleteServiceImpl: Started to find & publish message non associated images to delete for tenant=rdpsitt4
2021-04-08 20:30:13,954 DEBUG [restartedMain] [] [] config.schema.TenantConnectionProvider: Release connection for tenant=rdpsitt4
2021-04-08 20:30:13,990 INFO [restartedMain] [] [] org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@82c3ed8{HTTP/1.1, (http/1.1)}{0.0.0.0:1917}
2021-04-08 20:30:13,990 INFO [restartedMain] [] [] org.eclipse.jetty.server.session.HouseKeeper: node0 Stopped scavenging
2021-04-08 20:30:13,997 INFO [restartedMain] [] [] org.eclipse.jetty.server.handler.ContextHandler: Stopped o.s.b.w.e.j.JettyEmbeddedWebAppContext@417de324{application,/,[file:///C:/Users/kcr/AppData/Local/Temp/1/jetty-docbase.16655779792455135873.1917/],UNAVAILABLE}
2021-04-08 20:30:14,013 INFO [restartedMain] [] [] org.springframework.integration.endpoint.EventDrivenConsumer: Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2021-04-08 20:30:14,014 INFO [restartedMain] [] [] org.springframework.integration.channel.AbstractSubscribableChannel: Channel 'application-1.errorChannel' has 0 subscriber(s).
2021-04-08 20:30:14,014 INFO [restartedMain] [] [] org.springframework.integration.endpoint.AbstractEndpoint: stopped bean '_org.springframework.integration.errorLogger'
2021-04-08 20:30:14,014 INFO [restartedMain] [] [] org.springframework.integration.endpoint.EventDrivenConsumer: Removing {message-handler:cloudConfiguration.imsDeleteQMessagePublisher.serviceActivator} as a subscriber to the 'imsDeleteOutboundChannel' channel
2021-04-08 20:30:14,015 INFO [restartedMain] [] [] org.springframework.integration.channel.AbstractSubscribableChannel: Channel 'application-1.imsDeleteOutboundChannel' has 0 subscriber(s).
2021-04-08 20:30:14,015 INFO [restartedMain] [] [] org.springframework.integration.endpoint.AbstractEndpoint: stopped bean 'cloudConfiguration.imsDeleteQMessagePublisher.serviceActivator'
2021-04-08 20:30:14,029 INFO [restartedMain] [] [] org.springframework.scheduling.concurrent.ExecutorConfigurationSupport: Shutting down ExecutorService
2021-04-08 20:30:14,031 INFO [restartedMain] [] [] org.springframework.scheduling.concurrent.ExecutorConfigurationSupport: Shutting down ExecutorService 'taskScheduler'
2021-04-08 20:30:14,031 INFO [restartedMain] [] [] org.springframework.scheduling.concurrent.ExecutorConfigurationSupport: Shutting down ExecutorService 'applicationTaskExecutor'
2021-04-08 20:30:14,065 WARN [restartedMain] [] [] io.micrometer.core.util.internal.logging.AbstractInternalLogger: Failed to apply the value function for the gauge 'spring.integration.channels'. Note that subsequent logs will be logged at debug level.
org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'nullChannel': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:212)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:624)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:612)
at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1243)
at org.springframework.integration.config.IntegrationManagementConfigurer.lambda$registerComponentGauges$1(IntegrationManagementConfigurer.java:448)
at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:54)
at io.micrometer.cloudwatch.CloudWatchMeterRegistry$Batch.gaugeData(CloudWatchMeterRegistry.java:169)
at io.micrometer.cloudwatch.CloudWatchMeterRegistry$Batch.access$700(CloudWatchMeterRegistry.java:165)
at io.micrometer.cloudwatch.CloudWatchMeterRegistry.lambda$metricData$0(CloudWatchMeterRegistry.java:152)
at io.micrometer.core.instrument.Meter.match(Meter.java:107)
at io.micrometer.cloudwatch.CloudWatchMeterRegistry.lambda$metricData$4(CloudWatchMeterRegistry.java:151)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at io.micrometer.cloudwatch.CloudWatchMeterRegistry.metricData(CloudWatchMeterRegistry.java:161)
at io.micrometer.cloudwatch.CloudWatchMeterRegistry.publish(CloudWatchMeterRegistry.java:100)
at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafely(PushMeterRegistry.java:52)
at io.micrometer.core.instrument.push.PushMeterRegistry.close(PushMeterRegistry.java:89)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:339)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:273)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:579)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:551)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:512)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:170)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:979)
ImageDeletionServiceAppStarter.main(ImageDeletionServiceAppStarter.java:23)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
2021-04-08 20:30:17,944 INFO [restartedMain] [] [] org.springframework.orm.jpa.AbstractEntityManagerFactoryBean: Closing JPA EntityManagerFactory for persistence unit 'default'
2021-04-08 20:30:17,951 INFO [restartedMain] [] [] com.zaxxer.hikari.HikariDataSource: HikariPool-1 - Shutdown initiated...
2021-04-08 20:30:17,958 INFO [restartedMain] [] [] com.zaxxer.hikari.HikariDataSource: HikariPool-1 - Shutdown completed.

Process finished with exit code 0
``

[renukeswarchinta-rmal]

@shakuzen will metrics will be published to any metric system(In my case it is AWS CloudWatch). Is it while shutting down or when we do a increament on MeterRegistry.