Benchmarks for #116057 (EgorBo)

[EgorBot]

Processing dotnet/runtime#116057 (comment) command:

Command

-amd -intel -arm

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    private static Dictionary<string, int> Dictionary
        // We don't need to fill the dictionary as we're only benchmarking the hashcode operation.
        = new() {{ "1", 2 }};

    [Benchmark]
    [ArgumentsSource(nameof(GetKeys))]
    public bool Contains(string key) => Dictionary.ContainsKey(key);

    private static IEnumerable<string> GetKeys()
    {
        foreach (int len in (int[])[1, 10, 20, 30, 40, 50, 75, 100, 1000, 10000])
            yield return new string('a', len);
    }
}

(EgorBot will reply in this issue)

[EgorBot]

❌ Benchmark validation failed

cc @EgorBo (logs). EgorBot manual: link.

[EgorBo]

@EgorBot -arm -amd -intel

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    private static Dictionary<string, int> Dictionary
        // We don't need to fill the dictionary as we're only benchmarking the hashcode operation.
        = new() {{ "1", 2 }};

    [Benchmark]
    [ArgumentsSource(nameof(GetKeys))]
    public bool Contains(string key) => Dictionary.ContainsKey(key);

    public static IEnumerable<string> GetKeys()
    {
        foreach (int len in (int[])[1, 10, 20, 30, 40, 50, 75, 100, 1000, 10000])
            yield return new string('a', len);
    }
}

[EgorBot]

Benchmark results on linux_azure_genoa

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
AMD EPYC 9V74, 1 CPU, 16 logical and 8 physical cores
  Job-UFGQFL : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-SGSYBA : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI

Method	Toolchain	key	Mean	Error	Ratio
Contains	Main	a	5.898 ns	0.0013 ns	1.00
Contains	PR	a	6.156 ns	0.0019 ns	1.04
Contains	Main	aaaaaaaaaa	6.433 ns	0.0050 ns	1.00
Contains	PR	aaaaaaaaaa	6.443 ns	0.0011 ns	1.00
Contains	Main	aaaaaaaaaaaaaaaaaaaa	8.645 ns	0.0067 ns	1.00
Contains	PR	aaaaaaaaaaaaaaaaaaaa	8.635 ns	0.0011 ns	1.00
Contains	Main	aaaaa(...)aaaaa [30]	9.249 ns	0.0108 ns	1.00
Contains	PR	aaaaa(...)aaaaa [30]	9.430 ns	0.0020 ns	1.02
Contains	Main	aaaaa(...)aaaaa [40]	12.269 ns	0.0047 ns	1.00
Contains	PR	aaaaa(...)aaaaa [40]	12.753 ns	0.0010 ns	1.04
Contains	Main	aaaaa(...)aaaaa [50]	13.095 ns	0.0058 ns	1.00
Contains	PR	aaaaa(...)aaaaa [50]	14.758 ns	0.0045 ns	1.13
Contains	Main	aaaaa(...)aaaaa [75]	19.227 ns	0.0065 ns	1.00
Contains	PR	aaaaa(...)aaaaa [75]	15.741 ns	0.0036 ns	0.82
Contains	Main	aaaa(...)aaaa [100]	22.772 ns	0.0268 ns	1.00
Contains	PR	aaaa(...)aaaa [100]	18.676 ns	0.0050 ns	0.82
Contains	Main	aaaa(...)aaaa [1000]	206.707 ns	0.0958 ns	1.00
Contains	PR	aaaa(...)aaaa [1000]	55.444 ns	0.1823 ns	0.27
Contains	Main	aaa(...)aaa [10000]	2,050.498 ns	0.3053 ns	1.00
Contains	PR	aaa(...)aaa [10000]	389.271 ns	0.3163 ns	0.19

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

[EgorBot]

Benchmark results on linux_azure_cobalt100

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
azure_cobalt100
  Job-PCUMCK : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
  Job-VDCGEX : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD

Method	Toolchain	key	Mean	Error	Ratio
Contains	Main	a	7.136 ns	0.0013 ns	1.00
Contains	PR	a	6.834 ns	0.0021 ns	0.96
Contains	Main	aaaaaaaaaa	7.001 ns	0.0009 ns	1.00
Contains	PR	aaaaaaaaaa	6.968 ns	0.0010 ns	1.00
Contains	Main	aaaaaaaaaaaaaaaaaaaa	10.881 ns	0.0006 ns	1.00
Contains	PR	aaaaaaaaaaaaaaaaaaaa	10.395 ns	0.0013 ns	0.96
Contains	Main	aaaaa(...)aaaaa [30]	10.999 ns	0.0014 ns	1.00
Contains	PR	aaaaa(...)aaaaa [30]	10.839 ns	0.0007 ns	0.99
Contains	Main	aaaaa(...)aaaaa [40]	14.434 ns	0.0019 ns	1.00
Contains	PR	aaaaa(...)aaaaa [40]	19.117 ns	0.0011 ns	1.32
Contains	Main	aaaaa(...)aaaaa [50]	14.793 ns	0.0009 ns	1.00
Contains	PR	aaaaa(...)aaaaa [50]	22.769 ns	0.0013 ns	1.54
Contains	Main	aaaaa(...)aaaaa [75]	22.219 ns	0.0017 ns	1.00
Contains	PR	aaaaa(...)aaaaa [75]	23.945 ns	0.0012 ns	1.08
Contains	Main	aaaa(...)aaaa [100]	26.025 ns	0.0036 ns	1.00
Contains	PR	aaaa(...)aaaa [100]	28.577 ns	0.0014 ns	1.10
Contains	Main	aaaa(...)aaaa [1000]	250.081 ns	0.0271 ns	1.00
Contains	PR	aaaa(...)aaaa [1000]	169.024 ns	0.0062 ns	0.68
Contains	Main	aaa(...)aaa [10000]	2,492.973 ns	0.2946 ns	1.00
Contains	PR	aaa(...)aaa [10000]	1,595.048 ns	0.0941 ns	0.64

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

[EgorBot]

Benchmark results on linux_azure_cascadelake

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
Intel Xeon Platinum 8370C CPU 2.80GHz, 1 CPU, 16 logical and 8 physical cores
  Job-PUCUGS : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-ONGJWX : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI

Method	Toolchain	key	Mean	Error	Ratio
Contains	Main	a	6.817 ns	0.0150 ns	1.00
Contains	PR	a	6.508 ns	0.0151 ns	0.95
Contains	Main	aaaaaaaaaa	6.263 ns	0.0608 ns	1.00
Contains	PR	aaaaaaaaaa	6.547 ns	0.0278 ns	1.05
Contains	Main	aaaaaaaaaaaaaaaaaaaa	10.183 ns	0.0774 ns	1.00
Contains	PR	aaaaaaaaaaaaaaaaaaaa	9.659 ns	0.0219 ns	0.95
Contains	Main	aaaaa(...)aaaaa [30]	10.965 ns	0.0011 ns	1.00
Contains	PR	aaaaa(...)aaaaa [30]	10.780 ns	0.0017 ns	0.98
Contains	Main	aaaaa(...)aaaaa [40]	14.253 ns	0.0559 ns	1.00
Contains	PR	aaaaa(...)aaaaa [40]	14.910 ns	0.0067 ns	1.05
Contains	Main	aaaaa(...)aaaaa [50]	15.068 ns	0.0139 ns	1.00
Contains	PR	aaaaa(...)aaaaa [50]	16.843 ns	0.0024 ns	1.12
Contains	Main	aaaaa(...)aaaaa [75]	22.663 ns	0.0077 ns	1.00
Contains	PR	aaaaa(...)aaaaa [75]	17.879 ns	0.0061 ns	0.79
Contains	Main	aaaa(...)aaaa [100]	26.029 ns	0.0125 ns	1.00
Contains	PR	aaaa(...)aaaa [100]	20.844 ns	0.0093 ns	0.80
Contains	Main	aaaa(...)aaaa [1000]	261.494 ns	0.0259 ns	1.00
Contains	PR	aaaa(...)aaaa [1000]	71.795 ns	0.0257 ns	0.27
Contains	Main	aaa(...)aaa [10000]	2,584.381 ns	0.2983 ns	1.00
Contains	PR	aaa(...)aaa [10000]	560.577 ns	0.1273 ns	0.22

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

[EgorBo]

@EgorBot -amd -intel -arm

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    private static Dictionary<string, int> Dictionary
        // We don't need to fill the dictionary as we're only benchmarking the hashcode operation.
        = new() {{ "1", 2 }};

    [Benchmark]
    [ArgumentsSource(nameof(GetKeys))]
    public bool Contains(string key) => Dictionary.ContainsKey(key);

    private static IEnumerable<string> GetKeys()
    {
        foreach (int len in (int[])[1, 2, 3, 10, 20, 30, 40, 50, 63, 64, 70, 100, 120, 128, 1000, 10000])
            yield return new string('a', len);
    }
}

[EgorBot]

❌ Benchmark validation failed

cc @EgorBo (logs). EgorBot manual: link.

[EgorBo]

@EgorBot -amd -intel -arm

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    private static Dictionary<string, int> Dictionary
        // We don't need to fill the dictionary as we're only benchmarking the hashcode operation.
        = new() {{ "1", 2 }};

    [Benchmark]
    [ArgumentsSource(nameof(GetKeys))]
    public bool Contains(string key) => Dictionary.ContainsKey(key);

    public static IEnumerable<string> GetKeys()
    {
        foreach (int len in (int[])[1, 2, 3, 10, 20, 30, 40, 50, 63, 64, 70, 100, 120, 128, 1000, 10000])
            yield return new string('a', len);
    }
}

[EgorBot]

Benchmark results on linux_azure_genoa

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
AMD EPYC 9V74, 1 CPU, 16 logical and 8 physical cores
  Job-BJLMKI : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-NTYPJN : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI

Method	Toolchain	key	Mean	Error	Ratio
Contains	Main	a	5.968 ns	0.0024 ns	1.00
Contains	PR	a	5.880 ns	0.0009 ns	0.99
Contains	Main	aa	5.333 ns	0.0007 ns	1.00
Contains	PR	aa	5.079 ns	0.0011 ns	0.95
Contains	Main	aaa	5.609 ns	0.0012 ns	1.00
Contains	PR	aaa	5.351 ns	0.0010 ns	0.95
Contains	Main	aaaaaaaaaa	6.424 ns	0.0011 ns	1.00
Contains	PR	aaaaaaaaaa	6.698 ns	0.0028 ns	1.04
Contains	Main	aaaaaaaaaaaaaaaaaaaa	8.646 ns	0.0069 ns	1.00
Contains	PR	aaaaaaaaaaaaaaaaaaaa	8.915 ns	0.0139 ns	1.03
Contains	Main	aaaaa(...)aaaaa [30]	9.255 ns	0.0163 ns	1.00
Contains	PR	aaaaa(...)aaaaa [30]	9.726 ns	0.0088 ns	1.05
Contains	Main	aaaaa(...)aaaaa [40]	14.139 ns	0.0214 ns	1.00
Contains	PR	aaaaa(...)aaaaa [40]	12.000 ns	0.0219 ns	0.85
Contains	Main	aaaaa(...)aaaaa [50]	13.092 ns	0.0102 ns	1.00
Contains	PR	aaaaa(...)aaaaa [50]	12.682 ns	0.0134 ns	0.97
Contains	Main	aaaaa(...)aaaaa [63]	15.800 ns	0.0065 ns	1.00
Contains	PR	aaaaa(...)aaaaa [63]	14.784 ns	0.0504 ns	0.94
Contains	Main	aaaaa(...)aaaaa [64]	15.808 ns	0.0107 ns	1.00
Contains	PR	aaaaa(...)aaaaa [64]	14.067 ns	0.0030 ns	0.89
Contains	Main	aaaaa(...)aaaaa [70]	17.053 ns	0.0069 ns	1.00
Contains	PR	aaaaa(...)aaaaa [70]	14.695 ns	0.0024 ns	0.86
Contains	Main	aaaa(...)aaaa [100]	22.778 ns	0.0188 ns	1.00
Contains	PR	aaaa(...)aaaa [100]	19.337 ns	0.0040 ns	0.85
Contains	Main	aaaa(...)aaaa [120]	26.708 ns	0.0307 ns	1.00
Contains	PR	aaaa(...)aaaa [120]	22.553 ns	0.0041 ns	0.84
Contains	Main	aaaa(...)aaaa [128]	28.358 ns	0.0558 ns	1.00
Contains	PR	aaaa(...)aaaa [128]	20.538 ns	0.0428 ns	0.72
Contains	Main	aaaa(...)aaaa [1000]	206.625 ns	0.0587 ns	1.00
Contains	PR	aaaa(...)aaaa [1000]	55.756 ns	0.1391 ns	0.27
Contains	Main	aaa(...)aaa [10000]	2,051.798 ns	0.3873 ns	1.00
Contains	PR	aaa(...)aaa [10000]	389.687 ns	0.3023 ns	0.19

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

[EgorBot]

Benchmark results on linux_azure_cobalt100

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
azure_cobalt100
  Job-DZSNTL : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
  Job-XTOYGC : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD

Method	Toolchain	key	Mean	Error	Ratio
Contains	Main	a	7.040 ns	0.0009 ns	1.00
Contains	PR	a	5.955 ns	0.0020 ns	0.85
Contains	Main	aa	5.804 ns	0.0011 ns	1.00
Contains	PR	aa	5.319 ns	0.0008 ns	0.92
Contains	Main	aaa	6.214 ns	0.0010 ns	1.00
Contains	PR	aaa	5.276 ns	0.0018 ns	0.85
Contains	Main	aaaaaaaaaa	7.124 ns	0.0009 ns	1.00
Contains	PR	aaaaaaaaaa	6.442 ns	0.0011 ns	0.90
Contains	Main	aaaaaaaaaaaaaaaaaaaa	10.815 ns	0.0047 ns	1.00
Contains	PR	aaaaaaaaaaaaaaaaaaaa	9.480 ns	0.0016 ns	0.88
Contains	Main	aaaaa(...)aaaaa [30]	11.357 ns	0.0008 ns	1.00
Contains	PR	aaaaa(...)aaaaa [30]	10.960 ns	0.0084 ns	0.96
Contains	Main	aaaaa(...)aaaaa [40]	14.012 ns	0.0019 ns	1.00
Contains	PR	aaaaa(...)aaaaa [40]	13.665 ns	0.0017 ns	0.98
Contains	Main	aaaaa(...)aaaaa [50]	14.705 ns	0.0023 ns	1.00
Contains	PR	aaaaa(...)aaaaa [50]	14.230 ns	0.0016 ns	0.97
Contains	Main	aaaaa(...)aaaaa [63]	17.666 ns	0.0016 ns	1.00
Contains	PR	aaaaa(...)aaaaa [63]	16.976 ns	0.0032 ns	0.96
Contains	Main	aaaaa(...)aaaaa [64]	17.686 ns	0.0016 ns	1.00
Contains	PR	aaaaa(...)aaaaa [64]	17.130 ns	0.0030 ns	0.97
Contains	Main	aaaaa(...)aaaaa [70]	19.661 ns	0.0018 ns	1.00
Contains	PR	aaaaa(...)aaaaa [70]	18.691 ns	0.0021 ns	0.95
Contains	Main	aaaa(...)aaaa [100]	25.934 ns	0.0039 ns	1.00
Contains	PR	aaaa(...)aaaa [100]	25.669 ns	0.0120 ns	0.99
Contains	Main	aaaa(...)aaaa [120]	30.781 ns	0.0057 ns	1.00
Contains	PR	aaaa(...)aaaa [120]	31.218 ns	0.0047 ns	1.01
Contains	Main	aaaa(...)aaaa [128]	33.385 ns	0.0035 ns	1.00
Contains	PR	aaaa(...)aaaa [128]	34.263 ns	0.0069 ns	1.03
Contains	Main	aaaa(...)aaaa [1000]	250.878 ns	0.0455 ns	1.00
Contains	PR	aaaa(...)aaaa [1000]	169.736 ns	0.0136 ns	0.68
Contains	Main	aaa(...)aaa [10000]	2,493.119 ns	0.8691 ns	1.00
Contains	PR	aaa(...)aaa [10000]	1,594.650 ns	0.0617 ns	0.64

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

[EgorBot]

Benchmark results on linux_azure_cascadelake

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
Intel Xeon Platinum 8370C CPU 2.80GHz, 1 CPU, 16 logical and 8 physical cores
  Job-BCXAWP : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-LZKWTZ : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI

Method	Toolchain	key	Mean	Error	Ratio
Contains	Main	a	6.822 ns	0.0653 ns	1.00
Contains	PR	a	6.248 ns	0.0048 ns	0.92
Contains	Main	aa	5.717 ns	0.0221 ns	1.00
Contains	PR	aa	4.654 ns	0.0112 ns	0.81
Contains	Main	aaa	5.574 ns	0.0013 ns	1.00
Contains	PR	aaa	4.800 ns	0.0154 ns	0.86
Contains	Main	aaaaaaaaaa	6.129 ns	0.0652 ns	1.00
Contains	PR	aaaaaaaaaa	6.526 ns	0.0070 ns	1.06
Contains	Main	aaaaaaaaaaaaaaaaaaaa	9.842 ns	0.0406 ns	1.00
Contains	PR	aaaaaaaaaaaaaaaaaaaa	9.820 ns	0.0386 ns	1.00
Contains	Main	aaaaa(...)aaaaa [30]	10.937 ns	0.0024 ns	1.00
Contains	PR	aaaaa(...)aaaaa [30]	10.207 ns	0.0468 ns	0.93
Contains	Main	aaaaa(...)aaaaa [40]	14.688 ns	0.0208 ns	1.00
Contains	PR	aaaaa(...)aaaaa [40]	13.944 ns	0.0185 ns	0.95
Contains	Main	aaaaa(...)aaaaa [50]	15.118 ns	0.0137 ns	1.00
Contains	PR	aaaaa(...)aaaaa [50]	14.537 ns	0.0151 ns	0.96
Contains	Main	aaaaa(...)aaaaa [63]	18.436 ns	0.0236 ns	1.00
Contains	PR	aaaaa(...)aaaaa [63]	17.422 ns	0.0061 ns	0.94
Contains	Main	aaaaa(...)aaaaa [64]	18.440 ns	0.0071 ns	1.00
Contains	PR	aaaaa(...)aaaaa [64]	15.066 ns	0.0075 ns	0.82
Contains	Main	aaaaa(...)aaaaa [70]	19.719 ns	0.0053 ns	1.00
Contains	PR	aaaaa(...)aaaaa [70]	16.251 ns	0.0042 ns	0.82
Contains	Main	aaaa(...)aaaa [100]	26.027 ns	0.0100 ns	1.00
Contains	PR	aaaa(...)aaaa [100]	20.761 ns	0.0038 ns	0.80
Contains	Main	aaaa(...)aaaa [120]	30.702 ns	0.0191 ns	1.00
Contains	PR	aaaa(...)aaaa [120]	25.226 ns	0.0052 ns	0.82
Contains	Main	aaaa(...)aaaa [128]	32.759 ns	0.0266 ns	1.00
Contains	PR	aaaa(...)aaaa [128]	21.959 ns	0.0275 ns	0.67
Contains	Main	aaaa(...)aaaa [1000]	261.421 ns	0.0154 ns	1.00
Contains	PR	aaaa(...)aaaa [1000]	67.318 ns	0.0258 ns	0.26
Contains	Main	aaa(...)aaa [10000]	2,583.893 ns	0.1799 ns	1.00
Contains	PR	aaa(...)aaa [10000]	558.398 ns	0.0300 ns	0.22

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.