Something went wrong with outliers when using --launchCount #2432
Description
I have been playing around with --launchCount and noticed this when looking at the data:
BenchmarkDotNet v0.13.7-nightly.20230717.35, Windows 11 (10.0.22621.2283/22H2/2022Update/SunValley2)
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK 8.0.100-preview.7.23376.3
[Host] : .NET 6.0.22 (6.0.2223.42425), X64 RyuJIT AVX2
Job-QYHTSN : .NET 9.0.0 (42.42.42.42424), X64 RyuJIT AVX2
EnvironmentVariables=DOTNET_JitCSEHash=26ab038f,DOTNET_JitDisasmSummary=1,DOTNET_JitStdOutFile=D:\bugs\cse\base.4a.list,DOTNET_JitCSEMask=4 PowerPlanMode=00000000-0000-0000-0000-000000000000 Arguments=/p:EnableUnsafeBinaryFormatterSerialization=true
Toolchain=CoreRun IterationTime=250.0000 ms LaunchCount=100
MaxIterationCount=20 MinIterationCount=15 WarmupCount=1
| Method | Count | Mean | Error | StdDev | Median | Min | Max | Gen0 | Gen1 | Allocated |
|---|---|---|---|---|---|---|---|---|---|---|
| Dictionary | 512 | 13.71 μs | 0.052 μs | 0.598 μs | 13.82 μs | 12.44 μs | 17.10 μs | 7.6273 | 0.7133 | 46.97 KB |
// * Hints *
Outliers
TryAddDefaultSize.Dictionary: EnvironmentVariables=DOTNET_JitCSEHash=26ab038f,DOTNET_JitDisasmSummary=1,DOTNET_JitStdOutFile=D:\bugs\cse\base.4a.list,DOTNET_JitCSEMask=4, PowerPlanMode=00000000-0000-0000-0000-000000000000, Arguments=/p:EnableUnsafeBinaryFormatterSerialization=true, Toolchain=CoreRun, IterationTime=250.0000 ms, LaunchCount=100, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1 -> Something went wrong with outliers: Size(WorkloadActual) = 1548, Size(WorkloadActual/Outliers) = 34, Size(Result) = 1467), OutlierMode = RemoveUpper
This was running --filter "System.Collections.TryAddDefaultSize<String>.Dictionary" from dotnet/performance using their config setup and a locally built corerun. But I suspect it will repro with a normal build.