Stackoverflow when compiling callsite chain on a background thread

[aspnet-hello]

From @davidfowl on Monday, July 24, 2017 8:28:43 AM

See for details aspnet/Mvc#6410.

There isn't an isolated repro yet.

Copied from original issue: aspnet/DependencyInjection#555

[aspnet-hello]

From @xxjthxx on Friday, August 4, 2017 4:58:27 AM

I can confirm we have this issue as well. I don't know if this is the solution, but I can say that refactoring helped for this instance. This exception is very troubling though as it crashes kestrel
Service with dependencies:
Controller
-- Service1
-- Service2
---- Service1
What helped is moving common code to a separate Service
Controller
-- Service1
---- Service3 (common methods from Service1)
-- Service2
---- Service3 (common methods from Service1)

[aspnet-hello]

From @Eilon on Monday, August 7, 2017 2:30:24 PM

@pakrym can you take a look?

[aspnet-hello]

From @muratg on Tuesday, September 5, 2017 9:56:34 AM

Could anyone provide an isolated repro app?

cc @alohaninja, @xxjthxx

[aspnet-hello]

From @xxjthxx on Wednesday, September 6, 2017 12:03:55 PM

@muratg as I said before, this is a bug we where experiencing on certain data and certain e2e test in our environment. Although the setup was very specific the bug was very real because it was consistently reproduced it with our data. We didn't find simple repro to send it to you unfortunately. As a last resort we switched to Autofac, which helped.

[aspnet-hello]

From @davidfowl on Wednesday, September 6, 2017 11:42:33 PM

This bug doesn't seem to have a high number of hits so it's likely something specific that the 2 reported applications are doing. We need more information before we can do anything else here.

[aspnet-hello]

From @papaytl185 on Friday, October 6, 2017 2:05:49 PM

We are also running into this issue now, we have a few services that inject a fairly large dependency tree of services. Whats strange is that it seems to run well in 32 bit mode but we do get the stack overflow every so often. Once I run it in 64 bit, the first request loads fine and brings back the page with no memory spike, then on the 2nd request, the memory spikes like crazy until it consumes everything on my machine.

I'm going to try autofac or simple injector to see if it helps.

If anyone has any ideas id appreciate it.

[aspnet-hello]

From @pakrym on Friday, October 6, 2017 2:21:11 PM

@papaytl185 can you please share a memory dump from when the process starts to grow memory to [email protected]. It might help us to diagnose the issue because we weren't able to get any repos until now.

[aspnet-hello]

From @papaytl185 on Saturday, October 7, 2017 8:46:12 AM

@pakrym the memory dump is going to be huge like 5 gb plus, if I cant get it to dump fast enough before it grows to 20gb, I can't email that.

Do you have another recommendation?
On a side note, I tried lightinject last night, finally got everything running, but it takes awhile to boot up to my login page. After I click login, the site just hangs. So I must have a dependency wrong or something somewhere, maybe the lifetime IDK.

While I was messing around with lightinject, I found a setting called AddControllersAsServices() which is part of IMvcBuilder. When I used this option with default microsoft dependency injection, the memory spiked to 4-5gb and then settled back down to 3ish without crashing my app. So maybe that can help others.

I still want to figure out why lightinject isn't working correctly and if my service dependencies are too big for microsoft dependency injection to see how lightinject handles it. Or if I have some singleton or other lifetime scope incorrect masking an issue somewhere..

[aspnet-hello]

From @davidfowl on Saturday, October 7, 2017 9:44:44 AM

When I used this option with default microsoft dependency injection, the memory spiked to 4-5gb and then settled back down to 3ish without crashing my app. So maybe that can help others.

@papaytl185 It might make sense to do a live debug session with you. That just sounds insane.

[aspnet-hello]

From @papaytl185 on Saturday, October 7, 2017 11:44:18 AM

@davidfowl that would be great, I could really use some help with this. How / when should we set this up? I could do a live assist with you and walk you through the startup etc.

[aspnet-hello]

From @papaytl185 on Saturday, October 7, 2017 12:00:14 PM

@davidfowl Does the attached DebugDiag memory analysis report help identify anything for you?
memorydump.pdf

[aspnet-hello]

From @pakrym on Monday, October 9, 2017 9:16:31 AM

@papaytl185 It definitely shows concerning amounts of Expression related data structures that are used to optimize DI operations.

[aspnet-hello]

From @papaytl185 on Monday, October 9, 2017 11:16:59 AM

Do you have any recommendations on what I should do?

[aspnet-hello]

From @pakrym on Monday, October 9, 2017 11:43:45 AM

Are you able to take timeline profile with dotTrace? It would show both memory allocation source and where time is spent during startup.

[aspnet-hello]

From @papaytl185 on Monday, October 9, 2017 12:10:15 PM

Ill try to use dotTrace, I keep trying to use visual studio to grab a snapshot but it crashes due to memory.

Some other tidbits.
It gets through the startup fine and I can load my home controller with just static html.
Using Microsoft Depedency Injection
Once I jump to my AccountController it spikes.

Using LightInject:
It loads my AccountController without a spike, however once I try to login to the application it does nothing and just hangs. As if its blocked, the console says User Authenticated and never continues..

It has to be the way I am registering something in DI? I know the below proabably wont help much without the entire context of code. But maybe you see something since I'm pulling at straws.
Images below also.

My account controller injects:

        private readonly ABSignInManager _signInManager;
        private readonly ABUserManager _userManager;
        private readonly ABRoleManager _roleManager;
        private readonly IHostLogger _hostLogger;
        private readonly IHostingEnvironment _hostingEnvironment;
        private readonly IUrlPrefix _urlPrefix;
        private readonly IDistributedCache _distributedCache;

And my startup registers:

        container.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();

        container.AddScoped<ABHostContext>();
        container.AddScoped<HostLogContext>();
        container.AddScoped<HostEntityLockContext>();
        container.AddScoped<DealerContext>();
        container.AddScoped<DealerLogContext>(); //should result in a new context shared for the dealer log
        container.AddScoped<DealerEntityLockContext>(); //should result in a new context shared for entity locks

        //repository overrides 
        container.AddScoped<IHostRepository<Log>, HostLogRepository<Log>>();
        container.AddScoped<IRepositoryAsync<Domain.Models.Dealer.Log>, DealerLogRepository<Domain.Models.Dealer.Log>>();

        container.AddScoped<IRepositoryAsync<aEntityLock>, DealerEntityLockRepository<aEntityLock>>();

        //Loggers
        container.AddScoped<ISerilogFactory, SerilogFactory>();
        container.AddScoped<IHostLogger, HostLogger>();
        container.AddScoped<IHostLogService, BusinessLogic.HostServices.LogService>();

        container.AddScoped<Serilog.ILogger, DealerLogger>();
        container.AddScoped<IDealerLogService, BusinessLogic.DealerServices.LogService>();

        container.AddScoped<IUrlPrefix, UrlPrefix>();
        container.AddScoped<IDealerDocumentFolder, DealerDocumentFolder>();
        container.AddScoped<IDealerEncryptionKey, DealerEncryptionKey>();
        container.AddScoped<IDealerInfo, DealerInfo>();
        container.AddScoped<IClaimsGenerator, ClaimsGenerator>();
        //host services
        
        

        container.AddScoped((provider) => Postal.Email.CreateEmailService(provider));
        
        //Misc
        container.AddScoped<IUtilityWrapper, UtilityWrapper>();
        container.AddScoped<IHttpSessionManager, HttpSessionManager>();
        container.AddScoped<IHttpHostContextWrapper, HttpHostContextWrapper>();

        container.AddScoped<IHostConnectionStringService, HostConnectionStringService>();
        container.AddScoped<IDealerConnectionStringService, DealerConnectionStringService>();
        //container.AddSingleton(typeof(RedisHubLifetimeManager<>), typeof(RedisHubLifetimeManager<>));
        //services.AddSingleton(typeof(HubLifetimeManager<>), typeof(RedisPresenceHublifetimeManager<>));
        //services.AddSingleton(typeof(IUserTracker<>), typeof(RedisUserTracker<>));

        container.AddScoped<ABSession>();
        //add a forwarding registration that will allow us to request ISedonaOneSession and get the same object.
        //We can slowly replace references to ABSession with ISedonaOneSession, once all direct references to 
        //ABSession are replaced we can provide a separate implementation for mobile that doesn't rely on HttpContext
        container.AddScoped<ISedonaOneSession>(x => x.GetService<ABSession>());

        container.AddScoped<IMessageBus, MessageBus>();

        //Register Generic Repositories
        container.AddScoped(typeof(IHostRepository<>), typeof(HostRepository<>));
        //container.AddTransient(typeof(ITempDataRepository<>), typeof(TempDataRepository<>));
        container.AddScoped(typeof(IRepositoryAsync<>), typeof(DealerRepository<>));

        //Register Generic Services
        container.AddScoped(typeof(IHostService<>), typeof(HostService<>));
        container.AddScoped(typeof(IService<>), typeof(DealerService<>));

        //Register Generic Manager
        container.AddScoped(typeof(IValidator<>), typeof(SimpleValidator<>));
        container.AddScoped(typeof(IConverter<,>), typeof(AutoConverter<,>));
        container.AddScoped(typeof(IManager<,>), typeof(Manager<,>));

        // Refresh token services
        container.AddScoped<IRefreshTokenService, OpenIddictService>();`

Garbage collections is in like an infinite loop it looks like:

[aspnet-hello]

From @papaytl185 on Tuesday, October 10, 2017 8:13:08 AM

@davidfowl SimpleInjector works, keeps my memory around 1gb. Its crazy that there can be so many differences between containers. I also need to refactor my classes to use less dependencies and better follow SRP. I think that is the root cause. I have some classes with 10-15 injections.

[aspnet-hello]

From @Eilon on Friday, October 13, 2017 10:36:09 AM

@pakrym is there any further action for us to take here?

[aspnet-hello]

From @pakrym on Friday, October 13, 2017 10:55:16 AM

@Eilon there still no isolated repro or memory dump so not a lot of actionable items for now.

[aspnet-hello]

From @Eilon on Friday, October 13, 2017 11:11:54 AM

Ok we'll give it a few more days and close if we can't repro.

[aspnet-hello]

From @papaytl185 on Friday, October 13, 2017 11:50:26 AM

I can give someone a memory dump, just need to know how and where to upload a 5-6 GB file. On a side note, I finally ended up using the autofac extension with 3 lines of code and it works perfect. My app now runs at 600-700 MB, so there is something going on in Microsoft's dependency injection with large dependency graphs. I just don't know how to give you a repro without my project.

[aspnet-hello]

From @pakrym on Friday, October 13, 2017 11:51:21 AM

@papaytl185 You can use onedrive/dropbox and share a link via email

[aspnet-hello]

From @papaytl185 on Friday, October 13, 2017 11:57:55 AM

@pakrym Sounds good, I will start uploading the memory dump to my one drive account tonight. Once it finishes I'll send you an email with the link. Hopefully it helps, I grabbed the dump right when it started skyrocketing and snagged it at about 4.1 GB.

[aspnet-hello]

From @pakrym on Thursday, October 19, 2017 3:20:31 PM

@papaytl185 any luck with memory dump?

[aspnet-hello]

From @Zoxive on Tuesday, January 2, 2018 9:13:38 AM

@pakrym I can get you a memory dump. I am not a normal use case however. (Im in the slow process of converting a non dotnet core proj to dotnet core. Step one is staying on MVC 5 but swapping out Unity for DependancyInjection which is where I'm at) I'm having the similar symptoms but it only happens every other request.

Just like @alohaninja mentioned in aspnet/Mvc#6410 if i comment out the magical Interlocked.Increment block (https://github.com/aspnet/DependencyInjection/blob/release/2.0.0/src/Microsoft.Extensions.DependencyInjection/ServiceProvider.cs#L92) it works fine.

Also I've tried recreating simpler project that reproduces it to no avail.

[aspnet-hello]

From @pakrym on Tuesday, January 2, 2018 10:13:07 AM

@Zoxive thank you for the memory dump. From the quick look it indeed shows very large System.Linq.Expressions.Expression<System.Func<Microsoft.Extensions.DependencyInjection.ServiceProvider, System.Object>> with a lot of nesting. I'll investigate further.

[aspnet-hello]

From @Zoxive on Tuesday, January 2, 2018 10:16:58 AM

@pakrym let me know if you need anything else.

[pakrym]

@Zoxive
Looking at SO stack I can see GetRequiredService call from RegisterEntityActionsAndServices.AnonymousMethod__3_1, are you sure that it doesn't produce circular dependency?

Microsoft.Extensions.DependencyInjection.Abstractions.dll!Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService<Infor.Keystone.Runtime.Entities.Actions.Factories.EntityActionStepFactory>(System.IServiceProvider provider)	Unknown
Infor.Keystone.Runtime.dll!Infor.Keystone.Runtime.Entities.EntitiesUnity.RegisterEntityActionsAndServices.AnonymousMethod__3_1(System.IServiceProvider ioc)	Unknown
Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(Microsoft.Extensions.DependencyInjection.ServiceLookup.FactoryCallSite factoryCallSite, Microsoft.Extensions.DependencyInjection.ServiceProvider provider)	Unknown

[fc1943s]

@pakrym
.net core 2.1 preview2
I get the error on a big 50 project solution, with hundreds of services registered, I'm not sure how could I provide a way to reproduce.
Getting the error on VS debug and on the new dotnet preview2 docker alpine images.
Also compiled both from windows and the new dotnet sdk docker image.

The error happens only the third time I call GetServices on a few specific types.

[pakrym]

I opened a separate issue for this: #3054

[Zoxive]

Small update on my end. I'm in the process of converting my solution to dotnet core + kestrel and the same error happens there as well.

System.InvalidProgramException: 'Common Language Runtime detected an invalid program.'

[ghawkescs]

@Zoxive I tried your approach from above:

Using 2.1 which defaults to ServiceProviderMode.Dynamic same thing, every other request fails with a stackoverflow.
I switched it to Mode = ServiceProviderMode.Runtime and it works everytime.
Switching to Mode = ServiceProviderMode.Compiled fails everytime.

However the Mode on the Options object is internal and I cannot figure out anyway to set it in my project. I am upgrading a very large project from AspNetCore 1.1 to AspNetCore 2.0 and I am getting the SO error from the AspNetCore DI as discussed in this thread and other GitHub issues. I tried upgrading to the AspNetCore 2.1.1 nuget packages but the SO is still occurring.

Should this version of AspNetCore resolve the SO-DI issue?

@pakrym Any suggestions would be great as my team is getting stuck on this issue. Thank you.

[ghawkescs]

An update to my comment above. We downloaded the Microsoft.Extensions.DependencyInjection 2.1.1 source and rebuilt it using the ServiceProviderMode.Runtime flag. We then hosted the nuget package locally and referenced it throughout our code. This appears to have fixed our StackoverflowException for the time being.

@pakrym Is there an ETA for a fix being provided in the official package? Thank you.

[Zoxive]

@ghawkescs I was doing the same, but found it easier for me to just set the mode via reflection.

// Workaround until this is resolved https://github.com/aspnet/Home/issues/2737
var type = options.GetType();
var propertyInfo = type.GetProperty("Mode", BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic);
propertyInfo.SetValue(options, 1); // SET TO RUNTIME

[ghawkescs]

@Zoxive - Thank you, that would work for us too.

[muratg]

It's not clear what we'll be able to do in 2.2 milestone. Tentatively moving this to 3.0.

[dbredvick]

@Zoxive where are you doing that configuration? I'm running into this issue on a large project and I'd like to use the workaround you found.

[Zoxive]

@dbredvick
Inside UseDefaultServiceProvider() in Program.cs

//
.UseDefaultServiceProvider((context, options) =>
{
    options.ValidateScopes = !context.HostingEnvironment.IsProduction();

    // Workaround till they fix this https://github.com/aspnet/Home/issues/2737
    var type = options.GetType();
    var propertyInfo = type.GetProperty("Mode", BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic);
    propertyInfo.SetValue(options, 1); // SET TO RUNTIME
})
//

[davidfowl]

@pakrym it would be great if we could get some of these applications to test the new version of DI.

[Zoxive]

@davidfowl @pakrym I'll try it out Monday

[Zoxive]

@davidfowl @pakrym Tested Dynamic, Expressions and ILEmit with the preview packages briefly seems to be working.

So when this is finally released i can remove my forced "Runtime" mode!

Thanks!

[jonlanceley]

Thanks for the work around Zoxive which seems to work.
It would be nice to know when v3 will be out of preview though