The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2736712	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736714	dotnet-roslyn	Checkout dotnet-roslyn@main-vs-deps to s	Log
2736440	dotnet-dotnet	🛡 Guardian: CredScan (Binary)	Log
2736549	dotnet-runtime	Send job to Helix (Windows)	Log
2736626	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736612	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736608	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736574	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736573	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736548	dotnet-runtime	Build product	Log
2736525	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736492	dotnet-roslyn	Validate TODO/PROTOTYPE comments are not present	Log
2736422	dotnet-runtime	Build product	Log
1076566	dotnet/runtime	Send to Helix	Log
2736399	dotnet-runtime	Build product	Log
2736391	dotnet-runtime	Zip AOT Mono Artifacts	Log
2736389	dotnet-runtime	Build HelloAndroid sample app RUNTIME_FLAVOR=Mono AOT=true	Log
2736398	dotnet-runtime	Build product	Log
2736395	dotnet-runtime	Build product	Log
2736400	dotnet-runtime	Build product	Log
2736397	dotnet-runtime	Build product	Log
2736396	dotnet-runtime	Build product	Log
2736390	dotnet-runtime	Build product	Log
2736392	dotnet-runtime	Build product	Log
2736387	dotnet-runtime	Build product	Log
2736386	dotnet-runtime	Build product	Log
2736385	dotnet-runtime	Build product	Log
2736393	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736394	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736388	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736301	dotnet-aspire	Build	Log
2736218	dotnet-runtime	Build product	Log
2736248	dotnet-aspire	🛡 Component Governance	Log
2736196	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2736189	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
2736177	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
2736160	dotnet-performance	Send job to Helix (Windows)	Log
2736085	dotnet-runtime	Zip Build Assets	Log
2736113	dotnet-aspire	Build	Log
2736011	dotnet-runtime	Send job to Helix (Windows)	Log
1075630	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116874
2736094	dotnet-aspire	Checkout dotnet-aspire@ankj/cli-installers to s	Log
1076043	dotnet/runtime	Send to Helix	Log
2736029	dotnet-roslyn	Checkout dotnet-roslyn@main-vs-deps to s	Log
2735988	dotnet-runtime	Build product	Log
2736000	dotnet-roslyn	Checkout dotnet-roslyn@main to s	Log
2735987	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2735970	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2735957	dotnet-aspnetcore	Build ARM64	Log
1075685	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116903
2735935	dotnet-runtime	Build product	Log
1075701	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116918
2735939	dotnet-aspnetcore	Build ARM64	Log
1075627	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116874
1075570	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116732
2735895	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2735847	dotnet-aspnetcore	Checkout dotnet-aspnetcore@main to s	Log
2735789	dotnet-dotnet	Build (centOSStreamContainer)	Log
2735685	dotnet-dotnet	🛡 Component Governance	Log
1075141	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79048
1075072	dotnet/runtime	Send to Helix	Log
1075039	dotnet/runtime	Send job to Helix (Unix)	Log
1075019	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116899
1075016	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116354
2735292	dotnet-runtime	Run performance job script	Log
1074973	dotnet/runtime	AOT compile CoreCLR tests	Log
2735179	dotnet-runtime	Send job to Helix (Unix)	Log
1074860	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116893
1074839	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116795
2734989	dotnet-runtime	Send job to Helix (Windows)	Log
2734780	dotnet-runtime	Component Detection	Log
2734775	dotnet-runtime	Publish Build Assets	Log
2734781	dotnet-runtime	Install workload using artifacts	Log
2734771	dotnet-runtime	Publish Build Assets	Log
2734779	dotnet-runtime	Publish Build Assets	Log
2734774	dotnet-runtime	Zip Build Assets	Log
2734770	dotnet-runtime	Zip Build Assets	Log
2734772	dotnet-runtime	Publish Build Assets	Log
2734768	dotnet-runtime	Publish Build Assets	Log
2734776	dotnet-runtime	Build product	Log
1074672	dotnet/runtime	AOT compile CoreCLR tests	Log
1074187	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116879
1074641	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116795
2734628	dotnet-runtime	Send job to Helix (Unix)	Log
2734615	dotnet-runtime	Zip Build Assets	Log
2734579	dotnet-runtime	Publish Build Assets	Log
2734556	dotnet-runtime	Download BrowserWasm	Log
1074294	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79087
2734446	dotnet-dotnet	Build (centOSStreamContainer)	Log
2734468	dotnet-dotnet	Secure Supply Chain Analysis (auto-injected by policy)	Log
2734441	dotnet-aspnetcore	Checkout dotnet-aspnetcore@wtgodbe/TransportSymbols to s	Log
2734433	dotnet-aspnetcore	Validate Hosted Pool Information (1ES PT)	Log
2734379	dotnet-dotnet-monitor	Build	Log
2734346	dotnet-sdk	Run Tests in Helix	Log	#51089
2734347	dotnet-sdk	Run Tests in Helix	Log	#51090
2734334	dotnet-dotnet	Validate Signing	Log
2734277	dotnet-dotnet	Download Vertical Build Artifacts	Log
2734265	dotnet-dotnet	Build	Log
2734264	dotnet-dotnet	Download Vertical Build Artifacts	Log
1073774	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116860
Displaying 100 of 890 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
43	162	890

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)