-
Notifications
You must be signed in to change notification settings - Fork 5.1k
[infra][Apple] Fix Apple mobile runtime test reporting #114862
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[infra][Apple] Fix Apple mobile runtime test reporting #114862
Conversation
These tests are disabled in |
Good catch. I suppose the new merged runner might not be respecting these correctly (cc: @kotlarmilos) . I'll disable them directly in the test file using ActiveIssue attribute. Edit: disabling directly in the test file looks to be the only way now #112246 |
I'm pretty sure we have a ton of non-working disabled tests in |
/azp run runtime-extra-platforms |
Azure Pipelines successfully started running 1 pipeline(s). |
I believe you're right. I can confirm that the issue is with TCP tunnel by invoking mlaunch directly with and without TCP tunnel option. With TCP:
Without TCP:
I'm a bit confused by the merged runner system, do you know where that change could be made? |
Let's test this with XHarness once dotnet/xharness#1416 is merged. In the meantime, try to disable the project build in the merged test runner (
|
/azp run runtime-extra-platforms |
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR fixes Apple mobile CI test reporting by switching from apple run
to apple test
to capture exit codes correctly, removes base64 encoding of XML results, disables problematic tests on Apple mobile and non-Windows Mono, and adds a MacCatalyst arm64/x64 pipeline.
- Disable building managed tests on non-Windows Mono
- Annotate and disable specific JIT tests on Apple mobile via
ActiveIssue
- Turn off base64 XML encoding in
GeneratedTestRunner
and add MacCatalyst pipeline
Reviewed Changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.
Show a summary per file
File | Description |
---|---|
src/tests/managed/Managed.csproj | Disable managed test build for Mono on non-Windows platforms |
src/tests/JIT/opt/Structs/MemsetMemcpyNullref.* | Add TestLibrary reference and ActiveIssue for Apple mobile |
src/tests/JIT/Regression/**/GitHub_*.{csproj,cs,il} | Add TestLibrary reference and ActiveIssue in each test |
src/tests/Directory.Build.targets | Remove AppleMobile test exclusion |
src/tests/Common/XHarnessRunnerLibrary/RunnerEntryPoint.cs | Disable base64 encoding of XML results |
src/mono/msbuild/apple/build/AppleBuild.targets | Enable UseMonoRuntime for Mono builds |
eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml | Add MacCatalyst arm64/x64 runtime test pipeline |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
/ba-g Failure on extra-platforms are unrelated to this PR and are visible on main rolling builds |
Runtime test results reporting is broken on Apple mobile CI, see e.g., https://dev.azure.com/dnceng-public/public/_build/results?buildId=1020751&view=results and the log below.
This PR switches to using
apple test
instead ofapple run
(introduced in #107118) to correctly detect exit codes. Also removing the base64 encoding forAppleEntryPoint
XML results dump (introduced in #67393, not sure why?)Additional changes
Directed_1
on ios/iossimulators jobs is passing yet the logs contain a SIGSEGV error message.