Skip to content

[Perf] Windows/x64: 79 Regressions on 3/6/2023 11:36:59 AM #13995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
performanceautofiler bot opened this issue Mar 14, 2023 · 5 comments
Closed

[Perf] Windows/x64: 79 Regressions on 3/6/2023 11:36:59 AM #13995

performanceautofiler bot opened this issue Mar 14, 2023 · 5 comments
Assignees
@AndyAyersMS
Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EncodeUtf8 - Duration of single invocation 79.49 ns 86.50 ns 1.09 0.01 False
EncodeUtf8 - Duration of single invocation 66.06 ns 74.62 ns 1.13 0.04 False
EncodeUtf8 - Duration of single invocation 57.32 ns 66.33 ns 1.16 0.04 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

Payloads

Baseline
Compare

Histogram

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,&lorem ipsum=dolor sit amet,512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.49511848181112 > 83.5538407638473.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -119.57934323858325 (T) = (0 -86.18799593706828) / Math.Sqrt((0.01989936205140299 / (24)) + (0.08040259258575326 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.08316597032593837 = (79.57044284832288 - 86.18799593706828) / 79.57044284832288 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: JavaScript,&Hello+<World>!,512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 74.6237372864477 > 69.41757445712778.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -116.76856758117555 (T) = (0 -74.6062875482607) / Math.Sqrt((0.010243949585289716 / (24)) + (0.17093653979911036 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.12687771958258215 = (66.20619633503478 - 74.6062875482607) / 66.20619633503478 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: UnsafeRelaxed,hello "there",512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 66.32828236958538 > 60.18437576911939.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -144.8997615076406 (T) = (0 -66.77881697920037) / Math.Sqrt((0.011856860513257208 / (24)) + (0.1346398819420697 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.16440283453627252 = (57.350270025575185 - 66.77881697920037) / 57.350270025575185 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfAnyTwoValues - Duration of single invocation 12.10 ns 14.07 ns 1.16 0.01 False
IndexOfAnyThreeValues - Duration of single invocation 5.50 ns 6.58 ns 1.20 0.01 False
Clear - Duration of single invocation 3.67 ns 4.93 ns 1.35 0.26 False
IndexOfAnyTwoValues - Duration of single invocation 3.55 ns 4.56 ns 1.28 0.01 False
IndexOfAnyThreeValues - Duration of single invocation 13.86 ns 15.91 ns 1.15 0.02 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Char>.IndexOfAnyTwoValues(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.074189979373168 > 12.731450378667962.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/17/2023 9:53:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -231.83331520873986 (T) = (0 -14.065880044275872) / Math.Sqrt((0.0015631750257780485 / (24)) + (0.000158012843443573 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.15948724186938523 = (12.131121013111049 - 14.065880044275872) / 12.131121013111049 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyThreeValues(Size: 33)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.5822732227539635 > 5.775606499312917.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -742.767446226959 (T) = (0 -6.57915016805052) / Math.Sqrt((6.636319589554143E-06 / (24)) + (6.409153022641278E-05 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.19603376949097623 = (5.500806361722179 - 6.57915016805052) / 5.500806361722179 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.Clear(Size: 33)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.9320827964007865 > 4.065434581487989.
IsChangePoint: Marked as a change because one of 1/7/2023 12:45:06 AM, 1/10/2023 1:28:02 AM, 2/17/2023 9:53:52 PM, 3/5/2023 11:54:18 PM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -10.113556971926728 (T) = (0 -4.738262217306085) / Math.Sqrt((0.04425202676076872 / (22)) + (0.17709462286183417 / (37))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (37) - 2, .025) and -0.21356612783079607 = (3.9044120535693847 - 4.738262217306085) / 3.9044120535693847 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyTwoValues(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.561214224648973 > 3.7283327037432197.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/17/2023 9:53:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -1307.2256261227444 (T) = (0 -4.559385808248598) / Math.Sqrt((2.648202526564375E-06 / (24)) + (1.695824428428527E-05 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.2839155343246184 = (3.5511571332821235 - 4.559385808248598) / 3.5511571332821235 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyThreeValues(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.90542924475584 > 14.547626528731172.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -168.06203308429758 (T) = (0 -16.249888418820845) / Math.Sqrt((2.316577996253899E-05 / (24)) + (0.007067520835814389 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.17276943213676807 = (13.855995879099437 - 16.249888418820845) / 13.855995879099437 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 125.20 ns 136.58 ns 1.09 0.11 False
ToString - Duration of single invocation 185.24 ns 199.66 ns 1.08 0.04 False
ToStringWithFormat - Duration of single invocation 167.71 ns 180.04 ns 1.07 0.04 False
ToString - Duration of single invocation 182.68 ns 197.84 ns 1.08 0.04 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.TryParse(value: "1.7976931348623157e+308")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 136.58422990731475 > 132.1264845186577.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 2/8/2023 3:55:57 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -12.347099224407398 (T) = (0 -144.32329930155305) / Math.Sqrt((33.28839344479712 / (24)) + (11.390719690484914 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.12606772630697854 = (128.16573633174966 - 144.32329930155305) / 128.16573633174966 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.ToString(value: -1.7976931348623157E+308)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 199.65683680795124 > 193.63268538069798.
IsChangePoint: Marked as a change because one of 1/19/2023 8:10:32 PM, 1/26/2023 7:55:58 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:36:59 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -10.942253093135342 (T) = (0 -200.52522225292913) / Math.Sqrt((24.846025989354953 / (25)) + (3.519626324509351 / (35))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (35) - 2, .025) and -0.06054159426487934 = (189.0781307751766 - 200.52522225292913) / 189.0781307751766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.ToStringWithFormat(value: 1.7976931348623157E+308, format: "G17")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 180.03961044996865 > 175.81341869506457.
IsChangePoint: Marked as a change because one of 1/13/2023 11:20:20 PM, 2/14/2023 11:21:41 AM, 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -16.742243180659532 (T) = (0 -181.1576209930889) / Math.Sqrt((4.760415377768425 / (24)) + (10.9727263039687 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07015382872748935 = (169.28185101061766 - 181.1576209930889) / 169.28185101061766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.ToString(value: 1.7976931348623157E+308)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 197.8402573512462 > 192.09549389273545.
IsChangePoint: Marked as a change because one of 1/9/2023 3:58:12 PM, 2/17/2023 5:12:50 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:36:59 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -12.561049854828598 (T) = (0 -199.00047558944723) / Math.Sqrt((16.35165569652949 / (25)) + (8.578536276383302 / (35))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (35) - 2, .025) and -0.06366440657833547 = (187.08953158412515 - 199.00047558944723) / 187.08953158412515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReplaceWords - Duration of single invocation 56.35 μs 60.10 μs 1.07 0.18 False
MatchesWord - Duration of single invocation 1.04 μs 1.18 μs 1.13 0.05 False
MatchesBoundary - Duration of single invocation 49.99 μs 62.35 μs 1.25 0.05 False
MatchesBoundary - Duration of single invocation 128.42 μs 137.79 μs 1.07 0.18 False
MatchesBoundary - Duration of single invocation 50.42 μs 63.28 μs 1.26 0.02 False
MatchesWord - Duration of single invocation 897.42 ns 952.33 ns 1.06 0.04 False
Uri_IsNotMatch - Duration of single invocation 90.57 ns 98.96 ns 1.09 0.09 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.099919250645996 > 59.12534628787353.
IsChangePoint: Marked as a change because one of 1/3/2023 9:28:36 AM, 1/20/2023 12:33:18 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -5.150253894798643 (T) = (0 -61492.338521293845) / Math.Sqrt((4929824.096610276 / (24)) + (19585413.171780925 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07817498476091901 = (57033.72772549488 - 61492.338521293845) / 57033.72772549488 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1767921299447848 > 1.1082952884529715.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -23.470873459607 (T) = (0 -1213.1073300192284) / Math.Sqrt((260.39742048214026 / (24)) + (1020.8398051689537 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.13784552285538618 = (1066.1441343768486 - 1213.1073300192284) / 1066.1441343768486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 62.35373861696072 > 52.47879842252397.
IsChangePoint: Marked as a change because one of 1/20/2023 8:02:54 AM, 2/9/2023 12:40:40 PM, 2/24/2023 9:01:57 AM, 3/3/2023 4:59:02 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -4.242580088948566 (T) = (0 -57757.51326160844) / Math.Sqrt((25627734.76320062 / (24)) + (3468110.623573912 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.08607724735036967 = (53179.931172037344 - 57757.51326160844) / 53179.931172037344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.79348621511232 > 134.64698924036156.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/1/2023 6:08:01 AM, 2/17/2023 9:53:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -15.42815884980818 (T) = (0 -137755.8522680183) / Math.Sqrt((10178935.297662929 / (24)) + (527925.319306239 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.08013234255888073 = (127536.08686660438 - 137755.8522680183) / 127536.08686660438 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.28360956477732 > 52.568392088043744.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -57.71819385049054 (T) = (0 -61285.16258625139) / Math.Sqrt((450311.7474320481 / (24)) + (657056.3589406938 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.22129041596132856 = (50180.66283440969 - 61285.16258625139) / 50180.66283440969 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 952.3261018717368 > 943.4432166812592.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 2/9/2023 4:27:10 AM, 2/28/2023 5:58:27 PM, 3/6/2023 11:36:59 AM, 3/13/2023 2:14:21 PM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -17.72720879187587 (T) = (0 -958.5371067781595) / Math.Sqrt((393.53269149773416 / (57)) + (42.8579794795631 / (3))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (57) + (3) - 2, .025) and -0.09305499611614286 = (876.9340153826165 - 958.5371067781595) / 876.9340153826165 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 98.95733409121391 > 95.37176299970194.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 6:24:13 PM, 1/27/2023 7:12:54 PM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -11.287470726246964 (T) = (0 -100.52911394437659) / Math.Sqrt((9.3495984907112 / (43)) + (1.7230851213099918 / (16))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (16) - 2, .025) and -0.06840134140125713 = (94.09302482953463 - 100.52911394437659) / 94.09302482953463 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in Span.IndexerBench

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CoveredIndex2 - Duration of single invocation 757.48 ns 868.67 ns 1.15 0.03 False
WriteViaIndexer2 - Duration of single invocation 638.48 ns 683.82 ns 1.07 0.02 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Span.IndexerBench*'

Payloads

Baseline
Compare

Histogram

Span.IndexerBench.CoveredIndex2(length: 1024)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 868.6694328219432 > 793.7614795127765.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -528.7165591326337 (T) = (0 -868.644605622546) / Math.Sqrt((1.0404024993999381 / (24)) + (0.07327696756848821 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.14909919548168005 = (755.9352656742807 - 868.644605622546) / 755.9352656742807 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Span.IndexerBench.WriteViaIndexer2(length: 1024)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 683.8237477690978 > 670.8679993054843.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -111.26261905677086 (T) = (0 -681.3638234450007) / Math.Sqrt((1.7666489030307402 / (24)) + (2.499640908718883 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.06582521674155604 = (639.2828887348604 - 681.3638234450007) / 639.2828887348604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.IterateFor<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableList - Duration of single invocation 5.57 μs 6.27 μs 1.12 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateFor&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateFor<Int32>.ImmutableList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.268370710303749 > 5.868966593351013.
IsChangePoint: Marked as a change because one of 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -48.430665834731016 (T) = (0 -6405.035245773167) / Math.Sqrt((2769.2766252571782 / (24)) + (5453.576731893705 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.14223318578705976 = (5607.467306563813 - 6405.035245773167) / 5607.467306563813 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WhereFirst_LastElementMatches - Duration of single invocation 644.05 ns 693.47 ns 1.08 0.08 False
SelectToArray - Duration of single invocation 246.26 ns 271.03 ns 1.10 0.05 False
Take_All - Duration of single invocation 939.09 ns 1.01 μs 1.07 0.05 False
Max - Duration of single invocation 441.16 ns 493.23 ns 1.12 0.20 False
AnyWithPredicate_LastElementMatches - Duration of single invocation 631.45 ns 685.60 ns 1.09 0.12 False
All_AllElementsMatch - Duration of single invocation 579.03 ns 676.61 ns 1.17 0.22 False
Min - Duration of single invocation 440.91 ns 467.90 ns 1.06 0.08 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.WhereFirst_LastElementMatches(input: IEnumerable)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 693.4651139886611 > 675.9479987304337.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/5/2023 12:24:13 PM, 2/9/2023 10:24:28 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -12.536533937486704 (T) = (0 -697.8407771813061) / Math.Sqrt((315.1901171123951 / (24)) + (58.02218870363946 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07409356139969396 = (649.7020392450003 - 697.8407771813061) / 649.7020392450003 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToArray(input: Range)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 271.0346927523187 > 260.73783736381836.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/19/2023 12:32:21 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -70.20717812817685 (T) = (0 -272.99747576188855) / Math.Sqrt((1.4959494902109423 / (24)) + (1.8923011410482706 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.09617777523383263 = (249.04489210580257 - 272.99747576188855) / 249.04489210580257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Take_All(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0071769871505132 > 998.2841471347064.
IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -12.077655006770149 (T) = (0 -1023.0244224924746) / Math.Sqrt((565.5999189969177 / (24)) + (304.87556268075343 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07160557813127097 = (954.6650776832321 - 1023.0244224924746) / 954.6650776832321 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 493.23469751493747 > 462.9029535765845.
IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 2/20/2023 10:31:48 AM, 3/2/2023 1:15:31 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -12.833829273906025 (T) = (0 -483.8651983670354) / Math.Sqrt((90.23583113172795 / (24)) + (202.89072094235854 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.08849581537826172 = (444.5264662766646 - 483.8651983670354) / 444.5264662766646 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 685.6043332191421 > 663.0727701616191.
IsChangePoint: Marked as a change because one of 1/20/2023 4:48:32 AM, 2/8/2023 3:55:57 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -9.537354743801718 (T) = (0 -686.1886229948103) / Math.Sqrt((290.62451693898345 / (24)) + (32.551488783076536 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.052841633510515136 = (651.7491341094055 - 686.1886229948103) / 651.7491341094055 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.All_AllElementsMatch(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 676.6077617843746 > 608.2227961285435.
IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -9.161031318152382 (T) = (0 -643.6015108428378) / Math.Sqrt((434.9913075273374 / (24)) + (357.56704304664134 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.08198918808704801 = (594.8317394748855 - 643.6015108428378) / 594.8317394748855 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 467.8982749640145 > 462.9126116267709.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -52.97530866075204 (T) = (0 -469.46929939289515) / Math.Sqrt((0.17972998054914996 / (24)) + (10.00121770709398 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.06413799861579107 = (441.1733252675604 - 469.46929939289515) / 441.1733252675604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Tests.Perf_Char

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Char_ToUpperInvariant - Duration of single invocation 11.12 ns 14.51 ns 1.30 0.04 False
Char_ToLowerInvariant - Duration of single invocation 11.90 ns 14.46 ns 1.21 0.04 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Char*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Char.Char_ToUpperInvariant(input: "Hello World!")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.511821391933893 > 11.642514206049684.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -85.12134684681399 (T) = (0 -14.534782204917526) / Math.Sqrt((0.016354624275273275 / (24)) + (0.032126299990208544 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.30264754848039926 = (11.15787783262867 - 14.534782204917526) / 11.15787783262867 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Char.Char_ToLowerInvariant(input: "Hello World!")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.46016719723473 > 12.50032363074448.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -193.72234477310772 (T) = (0 -14.43162005274996) / Math.Sqrt((8.319488520936595E-06 / (24)) + (0.006109697833762702 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.21219717084046194 = (11.905340484126006 - 14.43162005274996) / 11.905340484126006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ctor_seeded - Duration of single invocation 388.18 ns 440.40 ns 1.13 0.11 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Random*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Random.ctor_seeded

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 440.4047718801468 > 407.043742643342.
IsChangePoint: Marked as a change because one of 1/13/2023 4:18:46 AM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 4:10:34 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -54.00472364293441 (T) = (0 -440.74164555953723) / Math.Sqrt((11.669337985248504 / (24)) + (21.860523346246328 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.14695925278734961 = (384.2696630141335 - 440.74164555953723) / 384.2696630141335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CheckObjIsInterfaceNo - Duration of single invocation 27.85 μs 55.71 μs 2.00 0.03 False
FooObjIsFoo2 - Duration of single invocation 278.51 μs 306.27 μs 1.10 0.04 False
CheckObjIsInterfaceYes - Duration of single invocation 27.86 μs 55.71 μs 2.00 0.03 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.CheckObjIsInterfaceNo

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.7074621886121 > 29.25413981729055.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -5255.454895926883 (T) = (0 -55709.88208814238) / Math.Sqrt((97.25132621788408 / (24)) + (865.082465671652 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.9996486645884584 = (27859.835117389415 - 55709.88208814238) / 27859.835117389415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.FooObjIsFoo2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 306.27454481792716 > 292.43855382898363.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -1082.675924244439 (T) = (0 -306367.8655545552) / Math.Sqrt((10079.088358466619 / (24)) + (8442.103177183655 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.09995008342188964 = (278528.88069380395 - 306367.8655545552) / 278528.88069380395 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.CheckObjIsInterfaceYes

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.706696322657166 > 29.25506800185109.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -4433.40545854543 (T) = (0 -55712.77994871444) / Math.Sqrt((87.69267437522485 / (24)) + (1289.6022364049923 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.9999078568497678 = (27857.673421250805 - 55712.77994871444) / 27857.673421250805 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Get

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetUInt32 - Duration of single invocation 599.57 ns 654.98 ns 1.09 0.00 False
GetUInt16 - Duration of single invocation 600.37 ns 656.08 ns 1.09 0.01 False
GetUInt64 - Duration of single invocation 599.13 ns 654.33 ns 1.09 0.01 False
GetBoolean - Duration of single invocation 99.99 ns 129.17 ns 1.29 0.09 False
GetInt64 - Duration of single invocation 655.54 ns 737.50 ns 1.13 0.01 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Get.GetUInt32

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 654.9776992342001 > 629.8668908661182.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -211.2308704356864 (T) = (0 -655.1519602940394) / Math.Sqrt((1.5073618924931154 / (24)) + (0.15543740524848743 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.0912375769476891 = (600.375183309368 - 655.1519602940394) / 600.375183309368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Get.GetUInt16

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 656.082645092421 > 631.0407505821066.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -431.68722630569937 (T) = (0 -655.7681411628637) / Math.Sqrt((0.3110131282902992 / (24)) + (0.10814311390480276 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.09098215944131323 = (601.0805359994882 - 655.7681411628637) / 601.0805359994882 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Get.GetUInt64

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 654.3301919657872 > 629.071391395274.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -557.940097727762 (T) = (0 -654.216889975582) / Math.Sqrt((0.18843966269864995 / (24)) + (0.06694288367235997 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.09175435657849201 = (599.234512812816 - 654.216889975582) / 599.234512812816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Get.GetBoolean

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 129.17176207636714 > 104.4840917704814.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 4:10:34 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -57.87842166747247 (T) = (0 -129.5349602591003) / Math.Sqrt((1.605031369360693 / (24)) + (7.72046905981037 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.3150328621181741 = (98.50321158549099 - 129.5349602591003) / 98.50321158549099 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Get.GetInt64

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 737.5002565073953 > 688.1866634067509.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -55.43512693948012 (T) = (0 -711.7602486520336) / Math.Sqrt((0.08480886526649435 / (24)) + (35.9775964413613 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.08589468228719598 = (655.4597423323492 - 711.7602486520336) / 655.4597423323492 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Guid, Guid>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HeapSort - Duration of single invocation 212.48 ns 224.50 ns 1.06 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Guid, Guid&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<Guid, Guid>.HeapSort(Size: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 224.4988000872664 > 223.3073219484203.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 2/9/2023 12:40:40 PM, 2/13/2023 6:54:14 PM, 3/2/2023 1:15:31 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -44.091713020142514 (T) = (0 -225.67965042456737) / Math.Sqrt((0.2985161197010395 / (24)) + (2.4709472892975337 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.059657498373059226 = (212.97414567543163 - 225.67965042456737) / 212.97414567543163 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IntObj - Duration of single invocation 111.41 μs 167.11 μs 1.50 0.03 False
ScalarValueTypeObj - Duration of single invocation 167.13 μs 250.64 μs 1.50 0.03 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf2.CastingPerf.IntObj

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 167.11135068389058 > 116.96793162677307.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -5461.854370898857 (T) = (0 -167103.1675747051) / Math.Sqrt((1129.0427751181267 / (24)) + (2049.5911906712236 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.49990393307793546 = (111409.2468787615 - 167103.1675747051) / 111409.2468787615 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf2.CastingPerf.ScalarValueTypeObj

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 250.64416005291008 > 175.51000398936173.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -4832.23270729336 (T) = (0 -250675.10389008405) / Math.Sqrt((3791.7650726586935 / (24)) + (5075.439044197384 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.49995693208247155 = (167121.53431102732 - 250675.10389008405) / 167121.53431102732 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Memory.Slice<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MemorySpanStartLength - Duration of single invocation 7.48 ns 8.62 ns 1.15 0.05 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Slice&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Slice<String>.MemorySpanStartLength

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.61739751559472 > 7.881834117628658.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -44.930314848944505 (T) = (0 -8.61100773804193) / Math.Sqrt((0.0024830326077022852 / (24)) + (0.01756509183693931 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.14728134123704487 = (7.505576381776763 - 8.61100773804193) / 7.505576381776763 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.IndexerSetReverse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IList - Duration of single invocation 1.15 μs 1.29 μs 1.12 0.05 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSetReverse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IndexerSetReverse<Int32>.IList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2947472995429998 > 1.2070504509113706.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -30.59029936347741 (T) = (0 -1300.2562818799047) / Math.Sqrt((104.6196843576466 / (24)) + (694.7496701901334 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.12921676127610174 = (1151.4673944535814 - 1300.2562818799047) / 1151.4673944535814 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AppendLine_Strings - Duration of single invocation 188.38 ns 200.00 ns 1.06 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_StringBuilder.AppendLine_Strings

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 199.99845032559475 > 197.41476492477847.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/9/2023 12:40:40 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -38.042068349794704 (T) = (0 -199.9555566401687) / Math.Sqrt((1.0421794886192095 / (24)) + (1.7588568802869469 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.0613397319531206 = (188.39920019973468 - 199.9555566401687) / 188.39920019973468 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Drawing.Tests.Perf_Color

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetSaturation - Duration of single invocation 884.51 ns 956.95 ns 1.08 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Drawing.Tests.Perf_Color*'

Payloads

Baseline
Compare

Histogram

System.Drawing.Tests.Perf_Color.GetSaturation

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 956.9452579982509 > 928.9896584045281.
IsChangePoint: Marked as a change because one of 2/1/2023 8:39:12 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -60.382066879893706 (T) = (0 -955.5654347818087) / Math.Sqrt((26.271564168400868 / (24)) + (6.4097363578776925 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.0769238919763138 = (887.3100893213591 - 955.5654347818087) / 887.3100893213591 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 975.04 ms 1.18 secs 1.21 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "([A-Za-z]awyer|[A-Za-z]inn)\s", Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1822493 > 1023.7153395.
IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -34.43500223696184 (T) = (0 -1182025884.592491) / Math.Sqrt((811148483486539.2 / (24)) + (2402379659244.136 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.20413710677710675 = (981637288.5943222 - 1182025884.592491) / 981637288.5943222 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Equals_Object_Valid - Duration of single invocation 5.20 ns 6.54 ns 1.26 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.StringSegmentBenchmark.Equals_Object_Valid

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.543305343889672 > 5.46604514033074.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -8.60549442204807 (T) = (0 -5.621172786764096) / Math.Sqrt((3.3766718753087146E-06 / (24)) + (0.08183942797501821 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.07994888844693487 = (5.2050359483659046 - 5.621172786764096) / 5.2050359483659046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Net.Primitives.Tests.IPAddressPerformanceTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor_Span - Duration of single invocation 6.17 ns 7.63 ns 1.24 0.05 False
Ctor_Bytes - Duration of single invocation 6.18 ns 7.26 ns 1.17 0.04 False
Ctor_Span - Duration of single invocation 10.22 ns 12.35 ns 1.21 0.07 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*'

Payloads

Baseline
Compare

Histogram

System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [143, 24, 20, 36])

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.6298106406555934 > 6.485385944147932.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/6/2023 9:02:07 PM, 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -57.08224119120558 (T) = (0 -7.5681172276902675) / Math.Sqrt((0.001433738530773278 / (24)) + (0.0177712795775618 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.21903187132352298 = (6.208301362517649 - 7.5681172276902675) / 6.208301362517649 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Bytes(address: [143, 24, 20, 36])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.257514562322532 > 6.488531967417447.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/6/2023 9:02:07 PM, 2/21/2023 5:46:41 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -88.19523866077635 (T) = (0 -7.248081298124088) / Math.Sqrt((0.002773369088214514 / (24)) + (0.0008280804767274602 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.16717703928766497 = (6.209924505152736 - 7.248081298124088) / 6.209924505152736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [16, 32, 48, 64, 80, ...])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.346697144151522 > 10.706479236008898.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/4/2023 4:58:40 AM, 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -49.3383400669882 (T) = (0 -12.484178894789844) / Math.Sqrt((0.028006565239918898 / (24)) + (0.025564285876565566 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.20794313186013805 = (10.335071714481446 - 12.484178894789844) / 10.335071714481446 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IterateTryGetTenSegments - Duration of single invocation 39.12 ns 41.92 ns 1.07 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.ReadOnlySequenceTests<Byte>.IterateTryGetTenSegments

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.92000401031485 > 40.84618161483159.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -29.661054793374667 (T) = (0 -41.84737505383498) / Math.Sqrt((0.062208445175478355 / (24)) + (0.220647525757492 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07088375992015887 = (39.0774205567885 - 41.84737505383498) / 39.0774205567885 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringX - Duration of single invocation 65.62 ns 72.75 ns 1.11 0.08 False
Parse - Duration of single invocation 108.43 ns 124.64 ns 1.15 0.07 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BigInteger.ToStringX(numberString: -2147483648)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.74922285395104 > 69.24392348519096.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 1/20/2023 4:39:21 PM, 2/6/2023 9:02:07 PM, 2/9/2023 12:40:40 PM, 2/28/2023 10:21:25 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -14.458283891989469 (T) = (0 -73.35275247888345) / Math.Sqrt((1.069696380901767 / (24)) + (9.605540585176191 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.12527498029017575 = (65.18651330892286 - 73.35275247888345) / 65.18651330892286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BigInteger.Parse(numberString: 123)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 124.6415496302327 > 114.28986847787132.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 1/26/2023 7:55:58 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -18.31558353051251 (T) = (0 -123.94135494251732) / Math.Sqrt((12.80594017149637 / (24)) + (3.9983013331652795 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.13499757472963242 = (109.19966500548811 - 123.94135494251732) / 109.19966500548811 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in SciMark2.kernel

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
benchSparseMult - Duration of single invocation 437.71 ms 463.98 ms 1.06 0.02 False
benchMonteCarlo - Duration of single invocation 732.54 ms 776.57 ms 1.06 0.00 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SciMark2.kernel*'

Payloads

Baseline
Compare

Histogram

SciMark2.kernel.benchSparseMult

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 463.98416 > 459.8085275.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 2/9/2023 12:40:40 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -54.89514125554927 (T) = (0 -463054832.1768707) / Math.Sqrt((2516206418757.302 / (24)) + (4244682940739.6675 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.059740543611515054 = (436951133.9056776 - 463054832.1768707) / 436951133.9056776 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### SciMark2.kernel.benchMonteCarlo

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 776.5685285714285 > 769.2384315.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -259.0857833167326 (T) = (0 -776763624.8084774) / Math.Sqrt((261454074556.5269 / (24)) + (631816368965.6694 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.0601616431309801 = (732684142.8769842 - 776763624.8084774) / 732684142.8769842 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringL - Duration of single invocation 49.20 ns 57.68 ns 1.17 0.03 False
ToString4 - Duration of single invocation 28.99 ns 39.12 ns 1.35 0.03 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.ToStringL

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 57.679915332404335 > 51.682616507639466.
IsChangePoint: Marked as a change because one of 2/10/2023 2:43:15 PM, 2/13/2023 6:54:14 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -20.348149029850774 (T) = (0 -58.10342096114684) / Math.Sqrt((0.23551298943865678 / (24)) + (6.544432664004445 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.18104701117305605 = (49.19653528730964 - 58.10342096114684) / 49.19653528730964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString4

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.11568251020091 > 30.38238082916476.
IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -112.60356304204215 (T) = (0 -38.922777060617584) / Math.Sqrt((0.09795155426840374 / (24)) + (0.14531607503246968 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.35255223345260883 = (28.777282013915933 - 38.922777060617584) / 28.777282013915933 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParseInt64 - Duration of single invocation 19.50 ns 23.80 ns 1.22 0.06 False
TryParseInt64 - Duration of single invocation 20.61 ns 25.33 ns 1.23 0.10 False
TryParseUInt64 - Duration of single invocation 19.41 ns 25.56 ns 1.32 0.14 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8ParserTests.TryParseInt64(value: 9223372036854775807)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.80486677290403 > 20.699088790913788.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -48.198087957616295 (T) = (0 -24.192231087862478) / Math.Sqrt((0.031543599852378076 / (24)) + (0.2703670952321068 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.23024544383642606 = (19.664556539564057 - 24.192231087862478) / 19.664556539564057 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseInt64(value: -9223372036854775808)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.32686196242631 > 21.108287753802056.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -40.57810013384828 (T) = (0 -24.81174545668887) / Math.Sqrt((0.15669677860114528 / (24)) + (0.20533004779000258 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.22082484576786088 = (20.323755322233023 - 24.81174545668887) / 20.323755322233023 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64(value: 18446744073709551615)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.562552519023154 > 20.740062062728892.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 1/18/2023 3:20:52 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -39.99644269731091 (T) = (0 -24.974207876369416) / Math.Sqrt((0.20451384615580084 / (24)) + (0.2899350129054097 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.26200793808538403 = (19.78926369849801 - 24.974207876369416) / 19.78926369849801 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.MathBenchmarks.Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Hypot - Duration of single invocation 15.33 μs 16.72 μs 1.09 0.01 False
ExpM1 - Duration of single invocation 19.51 μs 20.89 μs 1.07 0.02 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Single.Hypot

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.718681818181818 > 16.096810753676472.
IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -1060.85532267612 (T) = (0 -16717.608166090366) / Math.Sqrt((26.04920982303215 / (24)) + (22.306348884983777 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.09085637988807066 = (15325.214642651406 - 16717.608166090366) / 15325.214642651406 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.MathBenchmarks.Single.ExpM1

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.88791666666667 > 20.47689197549782.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -1135.5052485125527 (T) = (0 -20895.758912290392) / Math.Sqrt((19.26602812847197 / (24)) + (25.392294921388753 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07150549513004822 = (19501.308212847085 - 20895.758912290392) / 19501.308212847085 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compare_Same - Duration of single invocation 39.52 ns 46.44 ns 1.17 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringEquality*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, Ordinal))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 46.4371603072408 > 41.59486040935448.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 2/23/2023 8:54:44 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -89.8561459057418 (T) = (0 -46.46975502242408) / Math.Sqrt((0.12265615288040231 / (24)) + (0.019404391632283812 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.1700579095608194 = (39.71577358924609 - 46.46975502242408) / 39.71577358924609 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.TryGetValueTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 19.49 μs 21.44 μs 1.10 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueTrue<Int32, Int32>.SortedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.43998577380347 > 21.038346696358627.
IsChangePoint: Marked as a change because one of 1/5/2023 11:35:29 PM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -19.16499252554446 (T) = (0 -21348.165838629804) / Math.Sqrt((115450.55361175518 / (24)) + (24987.636228635718 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07135810195253715 = (19926.26536330199 - 21348.165838629804) / 19926.26536330199 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.IndexerSetReverse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IList - Duration of single invocation 1.86 μs 2.01 μs 1.08 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSetReverse&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IndexerSetReverse<String>.IList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0056826902465166 > 1.9565865132488034.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -10.772541023423813 (T) = (0 -2005.6631154407785) / Math.Sqrt((3165.2542943420594 / (24)) + (0.5487327008751488 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.06574088938174015 = (1881.9425391516206 - 2005.6631154407785) / 1881.9425391516206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.IterateFor<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableSortedSet - Duration of single invocation 5.78 μs 6.63 μs 1.15 0.13 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateFor&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateFor<String>.ImmutableSortedSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.627363249178981 > 6.027546029580638.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 11:43:56 PM, 2/14/2023 11:21:41 AM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -40.511055601353625 (T) = (0 -6563.998978644639) / Math.Sqrt((6479.743660559641 / (24)) + (4994.368591398943 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.1425601029834236 = (5744.992286624479 - 6563.998978644639) / 5744.992286624479 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in PerfLabTests.DelegatePerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DelegateInvoke - Duration of single invocation 389.91 μs 445.62 μs 1.14 0.03 False
MulticastDelegateCombineInvoke - Duration of single invocation 192.62 ms 205.50 ms 1.07 0.06 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.DelegatePerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.DelegatePerf.DelegateInvoke

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 445.6245342548077 > 409.4387179878049.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -1807.5593568319034 (T) = (0 -445594.7000248898) / Math.Sqrt((15162.605657070915 / (24)) + (11436.667304482893 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.14284999086290576 = (389897.8025002605 - 445594.7000248898) / 389897.8025002605 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.DelegatePerf.MulticastDelegateCombineInvoke

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 205.49720666666667 > 202.23854350000005.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/18/2023 3:20:52 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -29.164140446734176 (T) = (0 -205129062.67195767) / Math.Sqrt((1363407883740.7798 / (24)) + (5342722665664.832 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.06884019492251228 = (191917429.42155063 - 205129062.67195767) / 191917429.42155063 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Threading.Tests.Perf_Timer

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ShortScheduleAndDispose - Duration of single invocation 132.71 ns 144.37 ns 1.09 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tests.Perf_Timer*'

Payloads

Baseline
Compare

Histogram

System.Threading.Tests.Perf_Timer.ShortScheduleAndDispose

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 144.36645249509712 > 136.49978461379308.
IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -13.961625866497947 (T) = (0 -142.14448956015445) / Math.Sqrt((10.866479986924533 / (24)) + (10.077392384938129 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.09179278854443235 = (130.1936512601994 - 142.14448956015445) / 130.1936512601994 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinkedList - Duration of single invocation 6.65 μs 7.22 μs 1.09 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<String>.LinkedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.217305078742751 > 6.9758025195817766.
IsChangePoint: Marked as a change because one of 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -19.001670207092996 (T) = (0 -7211.245029597267) / Math.Sqrt((11198.421052924998 / (24)) + (6133.873940738786 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07124165662490119 = (6731.669726434386 - 7211.245029597267) / 6731.669726434386 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GenericClassGenericStaticMethod - Duration of single invocation 176.85 μs 195.05 μs 1.10 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.04961419753087 > 186.0517691669524.
IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -16.231907855740833 (T) = (0 -194969.04020501836) / Math.Sqrt((24488525.27543127 / (24)) + (2228.0325125227487 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.09182159606462967 = (178572.25109648527 - 194969.04020501836) / 178572.25109648527 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.Tests.Perf_Dictionary

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ContainsValue - Duration of single invocation 2.98 ms 3.79 ms 1.27 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_Dictionary*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_Dictionary.ContainsValue(Items: 3000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.7883881818181817 > 3.1291555.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -385.25496613207656 (T) = (0 -3788976.37647846) / Math.Sqrt((104497445.31611405 / (24)) + (1291354.0587328929 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.27070790285482244 = (2981783.9079823117 - 3788976.37647846) / 2981783.9079823117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Span - Duration of single invocation 16.56 μs 20.06 μs 1.21 0.18 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<Int32>.Span(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.05772038346187 > 17.377346254372583.
IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -90.99002957850855 (T) = (0 -20106.25146087404) / Math.Sqrt((32624.427517600718 / (24)) + (4128.518940453487 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.2102823874896677 = (16612.859667054923 - 20106.25146087404) / 16612.859667054923 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Drawing.Tests.Perf_Graphics_Transforms

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TransformPoints - Duration of single invocation 249.89 ns 273.54 ns 1.09 0.13 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Drawing.Tests.Perf_Graphics_Transforms*'

Payloads

Baseline
Compare

Histogram

System.Drawing.Tests.Perf_Graphics_Transforms.TransformPoints

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 273.53608909988293 > 262.6760138617892.
IsChangePoint: Marked as a change because one of 2/11/2023 10:42:07 AM, 2/15/2023 3:19:23 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -13.720397984391475 (T) = (0 -275.3029762710684) / Math.Sqrt((43.614324716762056 / (24)) + (8.01850773883152 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07663731841995948 = (255.706328919654 - 275.3029762710684) / 255.706328919654 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 100.84 ns 107.91 ns 1.07 0.04 False
IsMatch - Duration of single invocation 77.19 ns 87.30 ns 1.13 0.12 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 107.91458858467264 > 106.1550183790564.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 2/14/2023 11:21:41 AM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -18.576151223822908 (T) = (0 -107.18689229476465) / Math.Sqrt((0.7351817209594569 / (24)) + (2.0250534827114874 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.05383395574142751 = (101.71136706195146 - 107.18689229476465) / 101.71136706195146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 87.303088193048 > 82.7540214383599.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/18/2023 11:30:09 AM, 1/27/2023 7:12:54 PM, 2/1/2023 8:39:12 AM, 2/14/2023 11:21:41 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -7.177970305704114 (T) = (0 -87.25036766766307) / Math.Sqrt((1.466483956620247 / (43)) + (23.01969221049446 / (16))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (16) - 2, .025) and -0.11091556546371538 = (78.53915309148023 - 87.25036766766307) / 78.53915309148023 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 6123cb00bbec8ae3500c970890f5b0efcf7950e8
Compare ffbcb8336ac67496b0565112f663f74fda6cac34
Diff Diff

Regressions in System.Collections.ContainsKeyTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 3.84 μs 4.14 μs 1.08 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<Int32, Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.13514742488325 > 4.045452933287963.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 2/13/2023 6:54:14 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -48.31624909526639 (T) = (0 -4139.381660256372) / Math.Sqrt((457.5765968534205 / (24)) + (566.401463721238 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07394769871068005 = (3854.3605663719713 - 4139.381660256372) / 3854.3605663719713 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@EgorBo
Copy link
Member

EgorBo commented Mar 14, 2023

We blame internal BDN's heuristics here, the only change in the commit range is dotnet/runtime#83002 and it only affects Tier0

@EgorBo EgorBo removed the untriaged label Mar 14, 2023
@EgorBo EgorBo closed this as completed Mar 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AndyAyersMS AndyAyersMS

Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@EgorBo @AndyAyersMS