Parse version > 9 in dotnet.py #4470

am11 · 2024-09-20T05:14:31Z

A job run in PR dotnet/runtime#106599 is failing like https://dev.azure.com/dnceng-public/public/_build/results?buildId=813428&view=logs&j=3ca38e2b-df6d-5512-8718-3c103788a0f9&t=0d6eb74e-f21d-5060-4e66-099a24a67e5e

[2024/09/19 22:41:48][INFO] $ pushd "/mnt/vss/_work/1/performance"
[2024/09/19 22:41:48][INFO] $ popd
[2024/09/19 22:41:48][ERROR] Unexpected error: <class 'RuntimeError'>
[2024/09/19 22:41:48][ERROR] Traceback (most recent call last):
  File "/mnt/vss/_work/1/performance/scripts/run_performance_job.py", line 1187, in main
    run_performance_job(RunPerformanceJobArgs(**args))
  File "/mnt/vss/_work/1/performance/scripts/run_performance_job.py", line 768, in run_performance_job
    ci_setup.main(ci_setup_arguments)
  File "/mnt/vss/_work/1/performance/scripts/ci_setup.py", line 447, in main
    dotnet_version = dotnet.get_dotnet_version_precise(target_framework_moniker, args.cli) if args.dotnet_versions == [] else args.dotnet_versions[0]
  File "/mnt/vss/_work/1/performance/scripts/performance/tracer.py", line 90, in wrapper
    return func(*args, **kwargs)
  File "/mnt/vss/_work/1/performance/scripts/dotnet.py", line 614, in get_dotnet_version_precise
    sdk = get_dotnet_version_from_path(framework, dotnet_path, sdk_path)
  File "/mnt/vss/_work/1/performance/scripts/performance/tracer.py", line 90, in wrapper
    return func(*args, **kwargs)
  File "/mnt/vss/_work/1/performance/scripts/dotnet.py", line 601, in get_dotnet_version_from_path
    raise RuntimeError(
RuntimeError: Unable to determine the .NET SDK used for net9.0

am11 · 2024-09-20T06:46:21Z

cc @ilonatommy

am11 · 2024-09-20T08:21:03Z

Apparently, printf wasn't showing up in CI logs, so I moved the diag to exception message. Please retry.

am11 · 2024-09-20T08:29:28Z

Ah, so it is cloning the wrong repo :(

git clone https://github.com/dotnet/performance.git --depth 1 -b main --single-branch

am11 · 2024-09-20T12:16:53Z

Thanks, sorry I was in my just before calling it a weekend meeting with my manager 🤣

ilonatommy · 2024-09-20T12:19:54Z

That's fine. From some reason we're ignoring net10 that we parsed and we fall back to net9, see

performance/scripts/performance/common.py

Line 214 in f6d0cff

getLogger().info("Writing pipeline variable %s with value %s" % (name, value))

that logs

[2024/09/20 11:47:56][INFO] Writing pipeline variable PERFLAB_Framework with value net9.0

in https://dev.azure.com/dnceng-public/public/_build/results?buildId=813887&view=logs&j=63c48da7-769c-5a69-ecd6-caa9e1c7e293&t=8b4a1f4f-91f7-5091-3359-c47019cbfacd&l=86

am11 · 2024-09-20T12:21:18Z

noticed this one as well: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106599-merge-9f744b5760d349c987/x64.micro.net10.0.Partition0/1/console.a0f73dde.log?helixlogtype=result

[2024/09/20 12:00:47][INFO] //    * The required .NET Core SDK version 9.0 for runtime moniker WasmNet90 is not installed.
[2024/09/20 12:00:47][INFO] 
[2024/09/20 12:00:47][INFO] // Benchmark Perf_Type.GetType_FullyQualifiedNames: Job-ANOAPY(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [input=System.Int32[,]]
[2024/09/20 12:00:47][INFO] //    * The required .NET Core SDK version 9.0 for runtime moniker WasmNet90 is not installed.
[2024/09/20 12:00:47][INFO] 
[2024/09/20 12:00:47][INFO] // Benchmark Perf_UInt16.ToString: Job-LAURTV(OutlierMode=DontRemove, PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MemoryRandomization=True, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [value=0]
[2024/09/20 12:00:47][INFO] //    * The required .NET Core SDK version 9.0 for runtime moniker WasmNet90 is not installed.
[2024/09/20 12:00:47][INFO] 
[2024/09/20 12:00:47][INFO] // Benchmark Perf_UInt32.TryParse: Job-LAURTV(OutlierMode=DontRemove, PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MemoryRandomization=True, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [value=0]
[2024/09/20 12:00:47][INFO] //    * The required .NET Core SDK version 9.0 for runtime moniker WasmNet90 is not installed.
[2024/09/20 12:00:47][INFO] 
[2024/09/20 12:00:47][INFO] // Benchmark Perf_UInt64.Parse: Job-ANOAPY(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [value=0]
[2024/09/20 12:00:47][INFO] //    * The required .NET Core SDK version 9.0 for runtime moniker WasmNet90 is not installed.
[2024/09/20 12:00:47][INFO] 
[2024/09/20 12:00:47][INFO] // Benchmark Perf_Uri.CombineAbsoluteRelative: Job-ANOAPY(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [input=/new/path]
[2024/09/20 12:00:47][INFO] //    * The required .NET Core SDK version 9.0 for runtime moniker WasmNet90 is not installed.
[2024/09/20 12:00:47][INFO] 
[2024/09/20 12:00:47][INFO] // Benchmark Perf_Uri.CtorIdnHostPathAndQuery: Job-ANOAPY(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [input=http://host/p(...)s?key=ünicode [50]]
[2024/09/20 12:00:47][INFO] //    * The required .NET Core SDK version 9.0 for runtime moniker WasmNet90 is not installed.
[2024/09/20 12:00:47][INFO] 
[2024/09/20 12:00:47][INFO] // Benchmark Perf_Version.Parse4: Job-LAURTV(OutlierMode=DontRemove, PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MemoryRandomization=True, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0)
[2024/09/20 12:00:47][INFO] //    * The required .NET Core SDK version 9.0 for runtime moniker WasmNet90 is not installed.
[2024/09/20 12:00:47][INFO]

ilonatommy · 2024-09-20T12:22:33Z

noticed this one as well: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106599-merge-9f744b5760d349c987/x64.micro.net10.0.Partition0/1/console.a0f73dde.log?helixlogtype=result

I think this is understandable, considering that sdk 10 is expected to be used.

ilonatommy · 2024-09-20T12:31:58Z

Don't we need another case here:

performance/scripts/dotnet.py

Line 81 in f6d0cff

def get_target_framework_moniker(framework: str) -> str:

?

We're using this method to set framework that we then use to set the env variable that is logged as net9:

performance/scripts/ci_setup.py

Line 511 in f6d0cff

set_environment_variable('PERFLAB_Framework', framework)

This it not the full fix for sure, though. We have to figure out how args.channel became nativeaot9.0

Edit: aha, like this:

performance/azure-pipelines.yml

Line 149 in f6d0cff

- nativeaot9.0

scripts/dotnet.py

ilonatommy · 2024-09-20T12:42:03Z

I re-run before the final touches were done, so in case it won't pass, we'll make another round.

ilonatommy · 2024-09-20T13:10:38Z

Waiting for the 2nd run to finish:
I'm not sure if we are even using it in this CI run but this place looks like a candidate for changes as well:
https://github.com/dotnet/performance/blob/f6d0cff8bfb8bc0c3eec23b4b07c837e4c5f8abb/src/tools/ResultsComparer/Data.cs#L131C31-L131C41

am11 · 2024-09-20T13:18:14Z

BenchmarkDotNet apparently also has a lot of hardcoded monikers https://github.com/search?q=repo%3Adotnet%2FBenchmarkDotNet%20net9&type=code

Maybe we need to update BDN as well?

am11 · 2024-09-20T13:33:25Z

Maybe this will help? At least for PERFLAB_Framework 😅

am11 · 2024-09-20T15:52:08Z

RuntimeError: Unable to determine the .NET SDK used for net10.0. SDKs found in /mnt/vss/_work/1/performance/CorrelationStaging/payload/dotnet/sdk: ['10.0.100-alpha.1.24468.12']. Major version: 0

Looks like regex required non-greedy when there is more than one digit in first match. Some sort of python gotcha, but this works (tested on godbolt with net9.0 and net10.0)

am11 · 2024-09-20T17:14:35Z

Next up:

[2024/09/20 16:59:55][INFO] /datadisks/disk1/work/A2B508ED/p/dotnet/sdk/10.0.100-alpha.1.24468.12/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(166,5): error NETSDK1045: The current .NET SDK does not support targeting .NET 10.0. Either target .NET 9.0 or lower, or use a version of the .NET SDK that supports .NET 10.0. Download the .NET SDK from https://aka.ms/dotnet/download [/datadisks/disk1/work/A2B508ED/w/B0B10936/e/performance/src/benchmarks/micro/MicroBenchmarks.csproj::TargetFramework=net10.0]

so it is correctly passing 10.0, but 10.0 SDK is only supporting .NET 9.. see dotnet/runtime#106599 (comment)

caaavik-msft · 2024-09-20T17:58:04Z

We might need to get the BDN change to support targeting .NET 10 out and in this as well, similar to when we made this change last year:

caaavik-msft · 2024-09-20T18:23:44Z

scripts/channel_map.py

-            'tfm': 'net9.0',
-            'branch': '9.0',
+            'tfm': 'net10.0',
+            'branch': '10.0',


We should do this in a separate change as this shouldn't be a dependency to get dotnet/runtime#106599 working. This also means we can spend some time to get the BDN changes in as well to support wasm on .NET 10.

@am11 Can you remove the change to channel_map.py then we can merge this in?

azure-pipelines.yml

caaavik-msft

LGTM

am11 · 2024-09-23T21:21:11Z

@caaavik-msft thanks. Just curious, are those unsupported TMFs like (netcoreapp3) and desktop CLR ones (net4XX) channels still relevant / working? If not, maybe we should clean that up and keep it to lowest supported LTS.

caaavik-msft · 2024-09-23T21:27:01Z

Perhaps a good idea, I suppose there isn't much need for the older target frameworks. @DrewScoggins not sure if you had any other thoughts on if that might be something worth cleaning up.

am11 · 2024-09-23T21:31:46Z

Judging by

performance/src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Configurations/Common.cs

Line 52 in d384536

    
           throw new ArgumentException($"{prefix}: COMPlus Environment variables are disallowed. Please replace it with it's DOTNET equivalent.");

if we are actively disallowing COMPlus_ in favor of DOTNET_ environment vars, and those vars are essential for perf scenarios execution, then it means we are only supporting .NET 6+ (since DOTNET_ config functionality was added back then and COMPlus_ was the only supported prefix in previous versions).

DrewScoggins · 2024-09-23T21:32:05Z

Yeah, I was thinking about this. I am trying to come up with a realistic scenario that would involve us needing to go back and do performance testing on anything that old, and can't really get there. If the need did somehow arise, we can always just add it back in.

I think removing the obsolete TFMs seems like a good idea.

DrewScoggins · 2024-09-23T21:34:04Z

Judging by

performance/src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Configurations/Common.cs

Line 52 in d384536

throw new ArgumentException($"{prefix}: COMPlus Environment variables are disallowed. Please replace it with it's DOTNET equivalent.");

if we are actively disallowing COMPlus_ in favor of DOTNET_ environment vars, and those vars are essential for perf scenarios execution, then it means we are only supporting .NET 6+ (since DOTNET_ config functionality was added back then and COMPlus_ was the only supported prefix in previous versions).

That is for the GC perf tests and infrastructure, which is different than the existing microbenchmarks. However, I get your point.

Parse version > 9 in dotnet.py

e9ab42e

am11 mentioned this pull request Sep 20, 2024

Target .NET 10 / net10.0 dotnet/runtime#106599

Merged

am11 added 4 commits September 20, 2024 10:53

More fixes

36cb485

Print sdk dirs found in the CI

a143577

Update scripts/dotnet.py

3f308ed

Move diag to exception message

949627f

am11 and others added 3 commits September 20, 2024 11:41

Delete special condition (since we were not using the fork earlier)

14ce411

Reapply the workaround (needed after all=

8bc04ef

Allow more digits on the 1st position of version.

a0df24f

am11 commented Sep 20, 2024

View reviewed changes

scripts/dotnet.py Show resolved Hide resolved

am11 commented Sep 20, 2024

View reviewed changes

scripts/dotnet.py Show resolved Hide resolved

am11 added 2 commits September 20, 2024 15:37

return 10.0 for nativeaot9.0

4d8fe13

Update azure-pipelines.yml

6132bfe

am11 commented Sep 20, 2024

View reviewed changes

scripts/dotnet.py Outdated Show resolved Hide resolved

revert 9.0->10.0

12260ec

Update channel_map.py

16a797f

am11 added 2 commits September 20, 2024 16:50

Update Data.cs

de23481

make regex non-greedy

42cdb4e

lewing requested a review from DrewScoggins September 20, 2024 16:48

lewing requested a review from caaavik-msft September 20, 2024 17:39

caaavik-msft reviewed Sep 20, 2024

View reviewed changes

am11 mentioned this pull request Sep 23, 2024

Change .NET SDK Validator to account for backwards compatibility dotnet/BenchmarkDotNet#2645

Merged

am11 closed this Sep 23, 2024

am11 reopened this Sep 23, 2024

caaavik-msft reviewed Sep 23, 2024

View reviewed changes

azure-pipelines.yml Outdated Show resolved Hide resolved

Address feedback

d96daa7

caaavik-msft approved these changes Sep 23, 2024

View reviewed changes

caaavik-msft merged commit d384536 into dotnet:main Sep 23, 2024
2 of 56 checks passed

am11 deleted the patch-1 branch September 23, 2024 21:20

This was referenced Sep 23, 2024

Checkout failure: "Git fetch failed with exit code 128" dotnet/arcade#9009

Open

Unable to checkout dotnet/dnceng#4115

Open

Parse version > 9 in dotnet.py #4470

Parse version > 9 in dotnet.py #4470

Uh oh!

Conversation

am11 commented Sep 20, 2024

Uh oh!

am11 commented Sep 20, 2024

Uh oh!

am11 commented Sep 20, 2024

Uh oh!

am11 commented Sep 20, 2024

Uh oh!

am11 commented Sep 20, 2024

Uh oh!

ilonatommy commented Sep 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

am11 commented Sep 20, 2024

Uh oh!

ilonatommy commented Sep 20, 2024

Uh oh!

ilonatommy commented Sep 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ilonatommy commented Sep 20, 2024

Uh oh!

ilonatommy commented Sep 20, 2024

Uh oh!

am11 commented Sep 20, 2024

Uh oh!

am11 commented Sep 20, 2024

Uh oh!

am11 commented Sep 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

am11 commented Sep 20, 2024

Uh oh!

caaavik-msft commented Sep 20, 2024

Uh oh!

caaavik-msft Sep 20, 2024

Choose a reason for hiding this comment

Uh oh!

caaavik-msft Sep 23, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

caaavik-msft left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

am11 commented Sep 23, 2024

Uh oh!

caaavik-msft commented Sep 23, 2024

Uh oh!

am11 commented Sep 23, 2024

Uh oh!

DrewScoggins commented Sep 23, 2024

Uh oh!

DrewScoggins commented Sep 23, 2024

Uh oh!

Uh oh!

ilonatommy commented Sep 20, 2024 •

edited

Loading

ilonatommy commented Sep 20, 2024 •

edited

Loading

am11 commented Sep 20, 2024 •

edited

Loading