[GC] DATAS Startup time significantly worse than with workstation GC

[MichalStrehovsky]

Motivated by aspnet/Benchmarks#1908 where we were making startup time with DATAS even worse during the .NET 8 endgame, but this has been a problem from the beginning.

Comparing with workstation GC, server GC with DATAS seems to be making startup time about 2x worse for Native AOT in the Stage1 benchmark. It would be good to do something about it. This GC configuration (that we made the default for ASP.NET in 8.0) appears to be making startup a lot worse than it could be. 2x worse feels excessive.

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Motivated by aspnet/Benchmarks#1908 where we were making startup time with DATAS even worse during the .NET 8 endgame, but this has been a problem from the beginning.

Comparing with workstation GC, server GC with DATAS seems to be making startup time about 2x worse for Native AOT in the Stage1 benchmark. It would be good to do something about it. This GC configuration (that we made the default for ASP.NET in 8.0) appears to be making startup a lot worse than it could be. 2x worse feels excessive.

Author:	MichalStrehovsky
Assignees:	-
Labels:	area-GC-coreclr
Milestone:	-

[mrsharm]

CC: @eduardo-vp

Worked on collecting some data for this and in a nutshell for a 28 core Windows machine (which seemed to have repro'd the issue, as well):

application	WKS vs. DATAS w/o Any Changes (32 * 28 = 896 Initial Spin)	WKS vs. DATAS w/ YP Spin = 10	WKS vs. DATAS w/ YP Spin = 100	WKS vs. DATAS w/ YP Spin = 400
Start Time (ms)	18.75%	6.25%	10.94%	17.19%

• From the data, the lower we set the initial spin count, the less the start-up vs. WKS.
• The experiment used a modified version of the runtime that doesn't hold the Spin Count Unit stagnant throughout but only inits it with a said value: mrsharm@618320b
• Will need to follow up with a PR with the real fix and the numbers from running all the benchmarks.

Repro steps for each of the 4 configurations

Stage1Aot_Windows.wks_0:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences "Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion)" --application.framework net9.0 --application.options.collectCounters true --application.buildArguments /p:IlcStandaloneGCSupport=true --application.environmentVariables DOTNET_GCName=clrgcexp.dll --application.environmentVariables DOTNET_gcServer=0 --application.environmentVariables DOTNET_GCDynamicAdaptationMode=0 --application.collect true --application.collectStartup true --application.collectArguments "GCCollectOnly" --application.options.traceOutput C:\Users\musharm\source\repos\performance\src\benchmarks\gc\GC.Infrastructure\Configurations\ASPNetBenchmarks\DATAS_Results\wks_0\Stage1Aot_Windows.gc.etl.zip --chart --chart-type hex --application.aspNetCoreVersion 9.0.0-preview.6.24320.4 --application.runtimeVersion 9.0.0-preview.6.24319.11 --application.sdkVersion 9.0.100-preview.6.24321.2 --application.framework net9.0

Stage1Aot_Windows.datas_nospin_0:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences "Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion)" --application.framework net9.0 --application.options.collectCounters true --application.buildArguments /p:IlcStandaloneGCSupport=true --application.environmentVariables DOTNET_GCName=clrgcexp.dll --application.environmentVariables DOTNET_gcServer=1 --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.collect true --application.collectStartup true --application.collectArguments "GCCollectOnly" --application.options.traceOutput C:\Users\musharm\source\repos\performance\src\benchmarks\gc\GC.Infrastructure\Configurations\ASPNetBenchmarks\DATAS_Results\datas_nospin_0\Stage1Aot_Windows.gc.etl.zip --chart --chart-type hex --application.aspNetCoreVersion 9.0.0-preview.6.24320.4 --application.runtimeVersion 9.0.0-preview.6.24319.11 --application.sdkVersion 9.0.100-preview.6.24321.2 --application.framework net9.0

Stage1Aot_Windows.datas_spin10_0:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences "Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion)" --application.framework net9.0 --application.options.collectCounters true --application.buildArguments /p:IlcStandaloneGCSupport=true --application.environmentVariables DOTNET_GCName=clrgcexp.dll --application.environmentVariables DOTNET_gcServer=1 --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.environmentVariables DOTNET_GCSpinCountUnit=10

Stage1Aot_Windows.datas_spin100_0:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences "Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion)" --application.framework net9.0 --application.options.collectCounters true --application.buildArguments /p:IlcStandaloneGCSupport=true --application.environmentVariables DOTNET_GCName=clrgcexp.dll --application.environmentVariables DOTNET_gcServer=1 --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.environmentVariables DOTNET_GCSpinCountUnit=100

Stage1Aot_Windows.datas_spin400_0:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences "Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion)" --application.framework net9.0 --application.options.collectCounters true --application.buildArguments /p:IlcStandaloneGCSupport=true --application.environmentVariables DOTNET_GCName=clrgcexp.dll --application.environmentVariables DOTNET_gcServer=1 --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 --application.environmentVariables DOTNET_GCSpinCountUnit=400 --application.collect true --application.collectStartup true --application.collectArguments "GCCollectOnly" --application.options.traceOutput C:\Users\musharm\source\repos\performance\src\benchmarks\gc\GC.Infrastructure\Configurations\ASPNetBenchmarks\DATAS_Results\datas_spin400_0\Stage1Aot_Windows.gc.etl.zip --chart --chart-type hex --application.aspNetCoreVersion 9.0.0-preview.6.24320.4 --application.runtimeVersion 9.0.0-preview.6.24319.11 --application.sdkVersion 9.0.100-preview.6.24321.2 --application.framework net9.0