Benchmarks for #115339 (am11)

[EgorBot]

Processing dotnet/runtime#115339 (comment) command:

Command

-windows_intel -use32bit

using BenchmarkDotNet.Attributes;

public class Bench
{
    [Benchmark]
    public double[] AllocDoubleArray(int size) => new double[size];

    [Benchmark]
    public double[] AllocDoubleArrayFixed() => new double[32];
}

(EgorBot will reply in this issue)

[EgorBot]

❌ Benchmark validation failed

cc @am11 (logs). EgorBot manual: link.

[EgorBot]

❌ Benchmark validation failed

cc @EgorBo (logs). EgorBot manual: link.

[EgorBo]

@EgorBot -windows_intel -use32bit

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    [Benchmark]
    [Arguments(1024)]
    public double[] AllocDoubleArray(int size) => new double[size];

    [Benchmark]
    public double[] AllocDoubleArrayFixed() => new double[32];
}

[EgorBo]

Ah, I guess the benchmark was just incorrect so the verifier was right (just error message is not great)

[am11]

Perhaps something like this:

using BenchmarkDotNet.Attributes;

public class Bench
{
    [Params(16, 32, 64, 128)]
    public int Size;

    [Benchmark]
    public double[] AllocDoubleArray() => new double[Size];

    [Benchmark]
    public double[] AllocDoubleArrayFixed() => new double[32];
}

[EgorBo]

@am11 feel free to run? 🙂-pr num should not be necessary when you invoke it from here, it's going to take it from the title

[am11]

@EgorBot -windows_intel -use32bit -pr 115339

using BenchmarkDotNet.Attributes;

public class Bench
{
    [Params(16, 32, 64, 128)]
    public int Size;

    [Benchmark]
    public double[] AllocDoubleArray() => new double[Size];

    [Benchmark]
    public double[] AllocDoubleArrayFixed() => new double[32];
}

[am11]

-pr num should not be necessary when you invoke it from here, it's going to take it from the title

Ah good to know. I will remember it the next time. 🙂

[EgorBot]

Benchmark results on windows_azure_cascadelake

BenchmarkDotNet v0.15.0, Windows 11 (10.0.26100.2314/24H2/2024Update/HudsonValley) (Hyper-V)
azure_cascadelake
  Job-WHNAZQ : .NET 10.0.0 (42.42.42.42424), X86 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-YEDRZD : .NET 10.0.0 (42.42.42.42424), X86 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI

Method	Toolchain	size	Mean	Error	Ratio
AllocDoubleArrayFixed	\core_root_base\corerun.exe	?	13.05 ns	0.202 ns	1.00
AllocDoubleArrayFixed	\core_root_diff\corerun.exe	?	36.56 ns	0.115 ns	2.80
AllocDoubleArray	\core_root_base\corerun.exe	1024	1,970.75 ns	39.178 ns	1.00
AllocDoubleArray	\core_root_diff\corerun.exe	1024	1,925.88 ns	37.687 ns	0.98

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

[EgorBot]

Benchmark results on windows_azure_cascadelake

BenchmarkDotNet v0.15.0, Windows 11 (10.0.26100.2314/24H2/2024Update/HudsonValley) (Hyper-V)
azure_cascadelake
  Job-PQOLPV : .NET 10.0.0 (42.42.42.42424), X86 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-CFIXEK : .NET 10.0.0 (42.42.42.42424), X86 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI

Method	Toolchain	Size	Mean	Error	Ratio
AllocDoubleArray	\core_root_base\corerun.exe	16	7.738 ns	0.2308 ns	1.00
AllocDoubleArray	\core_root_diff\corerun.exe	16	31.040 ns	0.3803 ns	4.02
AllocDoubleArrayFixed	\core_root_base\corerun.exe	16	13.881 ns	0.3555 ns	1.00
AllocDoubleArrayFixed	\core_root_diff\corerun.exe	16	37.336 ns	0.3251 ns	2.69
AllocDoubleArray	\core_root_base\corerun.exe	32	14.265 ns	0.0715 ns	1.00
AllocDoubleArray	\core_root_diff\corerun.exe	32	37.066 ns	0.7109 ns	2.60
AllocDoubleArrayFixed	\core_root_base\corerun.exe	32	14.492 ns	0.1068 ns	1.00
AllocDoubleArrayFixed	\core_root_diff\corerun.exe	32	37.408 ns	0.2524 ns	2.58
AllocDoubleArray	\core_root_base\corerun.exe	64	26.053 ns	0.6050 ns	1.00
AllocDoubleArray	\core_root_diff\corerun.exe	64	49.363 ns	0.9478 ns	1.90
AllocDoubleArrayFixed	\core_root_base\corerun.exe	64	13.778 ns	0.3504 ns	1.00
AllocDoubleArrayFixed	\core_root_diff\corerun.exe	64	37.484 ns	0.2162 ns	2.72
AllocDoubleArray	\core_root_base\corerun.exe	128	51.420 ns	0.2833 ns	1.00
AllocDoubleArray	\core_root_diff\corerun.exe	128	74.450 ns	0.0498 ns	1.45
AllocDoubleArrayFixed	\core_root_base\corerun.exe	128	14.219 ns	0.0875 ns	1.00
AllocDoubleArrayFixed	\core_root_diff\corerun.exe	128	36.263 ns	0.0502 ns	2.55

BDN_Artifacts.zip

cc @am11 (agent_logs.txt). EgorBot manual: link.

[EgorBot]

Benchmark results on windows_azure_cascadelake

BenchmarkDotNet v0.15.0, Windows 11 (10.0.26100.2314/24H2/2024Update/HudsonValley) (Hyper-V)
azure_cascadelake
  Job-BHMWFV : .NET 10.0.0 (42.42.42.42424), X86 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-TBRJCP : .NET 10.0.0 (42.42.42.42424), X86 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI

Method	Toolchain	Size	Mean	Error	Ratio
AllocDoubleArray	\core_root_base\corerun.exe	16	7.259 ns	0.2242 ns	1.00
AllocDoubleArray	\core_root_diff\corerun.exe	16	30.722 ns	0.2652 ns	4.24
AllocDoubleArrayFixed	\core_root_base\corerun.exe	16	12.386 ns	0.0255 ns	1.00
AllocDoubleArrayFixed	\core_root_diff\corerun.exe	16	36.054 ns	0.4491 ns	2.91
AllocDoubleArray	\core_root_base\corerun.exe	32	12.666 ns	0.3263 ns	1.00
AllocDoubleArray	\core_root_diff\corerun.exe	32	36.061 ns	0.7371 ns	2.85
AllocDoubleArrayFixed	\core_root_base\corerun.exe	32	12.393 ns	0.0272 ns	1.00
AllocDoubleArrayFixed	\core_root_diff\corerun.exe	32	36.065 ns	0.4254 ns	2.91
AllocDoubleArray	\core_root_base\corerun.exe	64	24.077 ns	0.5567 ns	1.00
AllocDoubleArray	\core_root_diff\corerun.exe	64	47.568 ns	0.7320 ns	1.98
AllocDoubleArrayFixed	\core_root_base\corerun.exe	64	12.356 ns	0.0168 ns	1.00
AllocDoubleArrayFixed	\core_root_diff\corerun.exe	64	36.221 ns	0.4312 ns	2.93
AllocDoubleArray	\core_root_base\corerun.exe	128	46.046 ns	0.9742 ns	1.00
AllocDoubleArray	\core_root_diff\corerun.exe	128	69.811 ns	0.9754 ns	1.52
AllocDoubleArrayFixed	\core_root_base\corerun.exe	128	12.862 ns	0.2684 ns	1.00
AllocDoubleArrayFixed	\core_root_diff\corerun.exe	128	36.617 ns	0.6901 ns	2.85

BDN_Artifacts.zip

cc @am11 (agent_logs.txt). EgorBot manual: link.

[filipnavara]

It may be worth benchmarking other allocation scenarios and 64-bit platforms too. I expected the x86 new double[...] to be the obvious worst regression but we may generally have regressions elsewhere too. It would be useful to get some idea how bad they are, the slow path will also differ between Windows x64 and Linux x64 presumably due to number of saved registers.

[am11]

@EgorBot -arm -amd

using BenchmarkDotNet.Attributes;

public class Bench
{
    [Params(16, 32, 64)]
    public int Size;

    [Benchmark]
    public double[] AllocDoubleArray() => new double[Size];

    [Benchmark]
    public double[] AllocDoubleArrayFixed() => new double[32];
}

[EgorBot]

Benchmark results on linux_azure_cobalt100

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
azure_cobalt100
  Job-VMEGTQ : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
  Job-KETQPQ : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD

Method	Toolchain	Size	Mean	Error	Ratio
AllocDoubleArray	Main	16	8.714 ns	0.1617 ns	1.00
AllocDoubleArray	PR	16	8.034 ns	0.0128 ns	0.92
AllocDoubleArrayFixed	Main	16	13.090 ns	0.0189 ns	1.00
AllocDoubleArrayFixed	PR	16	13.171 ns	0.0234 ns	1.01
AllocDoubleArray	Main	32	12.616 ns	0.0184 ns	1.00
AllocDoubleArray	PR	32	12.496 ns	0.0103 ns	0.99
AllocDoubleArrayFixed	Main	32	13.512 ns	0.0104 ns	1.00
AllocDoubleArrayFixed	PR	32	13.152 ns	0.0182 ns	0.97
AllocDoubleArray	Main	64	21.725 ns	0.3645 ns	1.00
AllocDoubleArray	PR	64	20.878 ns	0.0199 ns	0.96
AllocDoubleArrayFixed	Main	64	13.476 ns	0.0153 ns	1.00
AllocDoubleArrayFixed	PR	64	12.260 ns	0.0197 ns	0.91

BDN_Artifacts.zip

cc @am11 (agent_logs.txt). EgorBot manual: link.

[EgorBot]

Benchmark results on linux_azure_genoa

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
AMD EPYC 9V74, 1 CPU, 8 logical and 4 physical cores
  Job-MHMYPZ : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-ORYIBM : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI

Method	Toolchain	Size	Mean	Error	Ratio
AllocDoubleArray	Main	16	8.519 ns	0.1532 ns	1.00
AllocDoubleArray	PR	16	8.215 ns	0.0753 ns	0.96
AllocDoubleArrayFixed	Main	16	11.521 ns	0.2384 ns	1.00
AllocDoubleArrayFixed	PR	16	11.022 ns	0.2083 ns	0.96
AllocDoubleArray	Main	32	11.387 ns	0.2474 ns	1.00
AllocDoubleArray	PR	32	11.939 ns	0.1280 ns	1.05
AllocDoubleArrayFixed	Main	32	11.721 ns	0.1698 ns	1.00
AllocDoubleArrayFixed	PR	32	12.154 ns	0.2682 ns	1.04
AllocDoubleArray	Main	64	16.311 ns	0.0541 ns	1.00
AllocDoubleArray	PR	64	16.438 ns	0.1446 ns	1.01
AllocDoubleArrayFixed	Main	64	10.937 ns	0.1696 ns	1.00
AllocDoubleArrayFixed	PR	64	11.154 ns	0.2144 ns	1.02

BDN_Artifacts.zip

cc @am11 (agent_logs.txt). EgorBot manual: link.