Skip to content

[infra][Apple] Fix Apple mobile runtime test reporting #114862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

matouskozak
Copy link
Member

@matouskozak matouskozak commented Apr 21, 2025

Runtime test results reporting is broken on Apple mobile CI, see e.g., https://dev.azure.com/dnceng-public/public/_build/results?buildId=1020751&view=results and the log below.

This PR switches to using apple test instead of apple run (introduced in #107118) to correctly detect exit codes. Also removing the base64 encoding for AppleEntryPoint XML results dump (introduced in #67393, not sure why?)

Additional changes

  • New merged runner is not correctly detecting running platform as such some tests are currently failing even though they should be disabled. This PR disables them directly in the test source files.
  • Disabling managed test suit on non-Windows Mono configurations due to issue with TCP (see [infra][Apple] Fix Apple mobile runtime test reporting #114862 (comment)).
  • Adding MacCatalyst arm64/x64 runtime test pipeline

Directed_1 on ios/iossimulators jobs is passing yet the logs contain a SIGSEGV error message.

[17:45:44] dbug: 
[17:45:44] dbug: =================================================================
[17:45:44] dbug: Native Crash Reporting
[17:45:44] dbug: =================================================================
[17:45:44] dbug: Got a SIGSEGV while executing native code. This usually indicates
[17:45:44] dbug: a fatal error in the mono runtime or one of the native libraries
[17:45:44] dbug: used by your application.
[17:45:44] dbug: =================================================================
[17:45:44] dbug: 
[17:45:44] dbug: =================================================================
[17:45:44] dbug: Native stacktrace:
[17:45:44] dbug: =================================================================
[17:45:44] dbug: 0x10b0d650f - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_dump_native_crash_info
[17:45:44] dbug: 0x10b0752ce - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_handle_native_crash
[17:45:44] dbug: 0x10afcec8f - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_sigsegv_signal_handler_debug
[17:45:44] dbug: 0x10b8e2e1d - /usr/lib/system/libsystem_platform.dylib : _sigtramp
[17:45:44] dbug: 0x0 - Unknown
[17:45:44] dbug: 0x10b0d8077 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : interp_run_filter
[17:45:44] dbug: 0x10b073ada - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_handle_exception_internal
[17:45:44] dbug: 0x10b072c24 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_handle_exception
[17:45:44] dbug: 0x10b0eb1f9 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : interp_throw
[17:45:44] dbug: 0x10b0e931d - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : interp_throw_ex_general
[17:45:44] dbug: 0x10b0de878 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_interp_exec_method
[17:45:44] dbug: 0x10b0d7a44 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : interp_runtime_invoke
[17:45:44] dbug: 0x10afd1ac0 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_jit_runtime_invoke
[17:45:44] dbug: 0x10af641b7 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_runtime_invoke_checked
[17:45:44] dbug: 0x10af6b495 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_runtime_exec_main_checked
[17:45:44] dbug: 0x10b030bd6 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_jit_exec
[17:45:44] dbug: 0x10b1ac699 - /Users/helix-runner/Library/Developer/CoreSimulator/Devices/E509338A-0A88-4D1A-B48F-F98F161992E6/data/Containers/Bundle/Application/E500201A-7840-41DA-A970-363EBC48CF2A/Directed_1.app/Directed_1 : mono_ios_runtime_init
[17:45:44] dbug: 0x7ff80017e1e0 - /Library/Developer/CoreSimulator/Volumes/iOS_22B81/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 18.1.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdispatch.dylib : _dispatch_call_block_and_release
[17:45:44] dbug: 0x7ff80017f563 - /Library/Developer/CoreSimulator/Volumes/iOS_22B81/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 18.1.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdispatch.dylib : _dispatch_client_callout
[17:45:44] dbug: 0x7ff8001825f6 - /Library/Developer/CoreSimulator/Volumes/iOS_22B81/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 18.1.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdispatch.dylib : _dispatch_queue_override_invoke
[17:45:44] dbug: 0x7ff800191cdd - /Library/Developer/CoreSimulator/Volumes/iOS_22B81/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 18.1.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdispatch.dylib : _dispatch_root_queue_drain
[17:45:44] dbug: 0x7ff800192740 - /Library/Developer/CoreSimulator/Volumes/iOS_22B81/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 18.1.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdispatch.dylib : _dispatch_worker_thread2
[17:45:44] dbug: 0x10b888c3e - /usr/lib/system/libsystem_pthread.dylib : _pthread_wqthread
[17:45:44] dbug: 0x10b887bdb - /usr/lib/system/libsystem_pthread.dylib : start_wqthread
[17:45:44] dbug: 
[17:45:44] dbug: =================================================================
[17:45:44] dbug: External Debugger Dump:
[17:45:44] dbug: =================================================================
[17:45:44] dbug: 
[17:45:44] dbug: =================================================================
[17:45:44] dbug: Basic Fault Address Reporting
[17:45:44] dbug: =================================================================
[17:45:44] dbug: Memory around native instruction pointer (0x10b0e373b):0x10b0e372b  b7 56 04 48 8b 14 10 48 85 d2 0f 84 d3 db ff ff  .V.H...H........
[17:45:44] dbug: 0x10b0e373b  8b 52 10 41 0f b7 76 02 89 14 30 49 83 c6 06 41  .R.A..v...0I...A
[17:45:44] dbug: 0x10b0e374b  0f b7 16 48 8d 35 2b cd 38 00 49 89 db 4d 89 fc  ...H.5+.8.I..M..
[17:45:44] dbug: 0x10b0e375b  49 89 c7 41 89 d5 ff 24 d6 66 66 66 2e 0f 1f 84  I..A...$.fff....
[17:45:44] dbug: 
[17:45:44] dbug: =================================================================
[17:45:44] dbug: Managed Stacktrace:
[17:45:44] dbug: =================================================================
[17:45:44] dbug: at <unknown> <0xffffffff>
[17:45:44] dbug: at <Module>:MoveBetweenFilterAndHandler <0x00032>
[17:45:44] dbug: at FilterToHandler:Main <0x00008>
[17:45:44] dbug: at <>c__DisplayClass0_0:<<Main>$>g__TestExecutor34|35 <0x000b4>
[17:45:44] dbug: at <>c__DisplayClass0_0:<<Main>$>g__RunTests|1 <0x004a8>
[17:45:44] dbug: at XHarnessRunnerLibrary.GeneratedTestRunner:Run <0x0005c>
[17:45:44] dbug: at <InternalRunAsync>d__38:MoveNext <0x00486>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
[17:45:44] dbug: at Microsoft.DotNet.XHarness.TestRunners.Common.ApplicationEntryPoint:InternalRunAsync <0x00082>
[17:45:44] dbug: at <InternalRunAsync>d__39:MoveNext <0x00148>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
[17:45:44] dbug: at Microsoft.DotNet.XHarness.TestRunners.Common.ApplicationEntryPoint:InternalRunAsync <0x000ae>
[17:45:44] dbug: at <RunAsync>d__0:MoveNext <0x001fa>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
[17:45:44] dbug: at Microsoft.DotNet.XHarness.TestRunners.Common.iOSApplicationEntryPointBase:RunAsync <0x0006c>
[17:45:44] dbug: at <RunTests>d__0:MoveNext <0x002b4>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
[17:45:44] dbug: at XHarnessRunnerLibrary.RunnerEntryPoint:RunTests <0x000ae>
[17:45:44] dbug: at <<Main>$>d__0:MoveNext <0x00126>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
[17:45:44] dbug: at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
[17:45:44] dbug: at Program:<Main>$ <0x0006c>
[17:45:44] dbug: at Program:<Main> <0x00014>
[17:45:44] dbug: at <Module>:runtime_invoke_direct_int_string[] <0x0004a>
[17:45:44] dbug: at <unknown> <0x00000>
[17:45:44] dbug: =================================================================
[17:45:44] dbug: Xamarin.Hosting: Simulated process has exited (could not get process info: Process with an Id of 30393 is not running.).
[17:45:45] dbug: Process mlaunch exited with 0
[17:45:45] dbug: Process simctl exited with 137
[17:45:45] dbug: Failed to determine the exit code from /tmp/helix/working/AC0D0941/w/AC630917/uploads/iPhone Xs (iOS 18.1) - created by XHarness.log
[17:45:45] dbug: Failed to determine the exit code from /tmp/helix/working/AC0D0941/w/AC630917/uploads/Directed_1.log
[17:45:45] info: App run ended, no abnormal exit code detected (0 assumed)

@BrzVlad
Copy link
Member

BrzVlad commented Apr 22, 2025

These tests are disabled in issues.targets globally for mono targets. They shouldn't be running in the first place.

@matouskozak
Copy link
Member Author

matouskozak commented Apr 22, 2025

These tests are disabled in issues.targets globally for mono targets. They shouldn't be running in the first place.

Good catch. I suppose the new merged runner might not be respecting these correctly (cc: @kotlarmilos) . I'll disable them directly in the test file using ActiveIssue attribute.

Edit: disabling directly in the test file looks to be the only way now #112246

@BrzVlad
Copy link
Member

BrzVlad commented Apr 22, 2025

I'm pretty sure we have a ton of non-working disabled tests in issues.targets. There should be more failing if there is a general problem with parsing those.

@dotnet dotnet deleted a comment from azure-pipelines bot Apr 23, 2025
@dotnet dotnet deleted a comment from azure-pipelines bot Apr 23, 2025
@dotnet dotnet deleted a comment from azure-pipelines bot Apr 23, 2025
@matouskozak
Copy link
Member Author

/azp run runtime-extra-platforms

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@matouskozak
Copy link
Member Author

The apple run doesn't use TCP tunnel for communication. I think the problem is that when app contains only unsupported tests (not skipped), the TestCompleted events doesn't fire, or similar event that is required to close the tunnel.

I believe you're right.

I can confirm that the issue is with TCP tunnel by invoking mlaunch directly with and without TCP tunnel option. With TCP:

        Using verbose logging.
        2025-06-01 19:42:08.113+0200  Acquired tunnel connection to device.
        2025-06-01 19:42:08.178+0200  Enabling developer disk image services.
        2025-06-01 19:42:08.233+0200  Acquired usage assertion.
        Launched application with net.dot.Managed bundle identifier.
        Waiting for the application to terminate…
        The app terminated with the exit code 1.

Without TCP:

2025-06-01 19:42:31.799+0200  Acquired tunnel connection to device.
2025-06-01 19:42:31.838+0200  Enabling developer disk image services.
2025-06-01 19:42:31.894+0200  Acquired usage assertion.
Launched application with net.dot.Managed bundle identifier.
Waiting for the application to terminate…
2025-06-01 19:42:32.085 Managed[722:68324] Failed to initialize TCP writer. Continuing on console.
System.ArgumentNullException: Value cannot be null. (Parameter 'hostName')
   at Microsoft.DotNet.XHarness.TestRunners.Common.TcpTextWriter.InitializeWithDirectConnection(String hostName, Int32 port)
2025-06-01 19:42:32.085 Managed[722:68324]    at Microsoft.DotNet.XHarness.TestRunners.Common.iOSApplicationEntryPointBase.RunAsync()
2025-06-01 19:42:32.085 Managed[722:68324] [Runner executing:   Run everything]
2025-06-01 19:42:32.085 Managed[722:68324] [:    v]
2025-06-01 19:42:32.085 Managed[722:68324] [Device Name:        ]
2025-06-01 19:42:32.085 Managed[722:68324] [Device UDID:        ]
2025-06-01 19:42:32.085 Managed[722:68324] [Device Locale:      ]
2025-06-01 19:42:32.100 Managed[722:68324] [Device Date/Time:   01.06.2025 19:42:32]
2025-06-01 19:42:32.100 Managed[722:68324] [Bundle:     net.dot.Managed]
2025-06-01 19:42:32.100 Managed[722:68324] [Runner executing:   Run everything]
2025-06-01 19:42:32.100 Managed[722:68324] [:    v]
2025-06-01 19:42:32.100 Managed[722:68324] [Device Name:        ]
2025-06-01 19:42:32.100 Managed[722:68324] [Device UDID:        ]
2025-06-01 19:42:32.100 Managed[722:68324] [Device Locale:      ]
2025-06-01 19:42:32.100 Managed[722:68324] [Device Date/Time:   01.06.2025 19:42:32]
2025-06-01 19:42:32.100 Managed[722:68324] [Bundle:     net.dot.Managed]
2025-06-01 19:42:37.156 Managed[722:68324] <assemblies>
2025-06-01 19:42:37.156 Managed[722:68324] <assembly
2025-06-01 19:42:37.156 Managed[722:68324]     name="Managed"
2025-06-01 19:42:37.157 Managed[722:68324]     test-framework="XUnitWrapperGenerator-generated-runner"
2025-06-01 19:42:37.158 Managed[722:68324]     run-date="2025-06-01"
2025-06-01 19:42:37.158 Managed[722:68324]     run-time="19:42:32"
2025-06-01 19:42:37.158 Managed[722:68324]     time="5,044204"
2025-06-01 19:42:37.158 Managed[722:68324]     total="0"
2025-06-01 19:42:37.161 Managed[722:68324]     passed="0"
2025-06-01 19:42:37.161 Managed[722:68324]     failed="0"
2025-06-01 19:42:37.161 Managed[722:68324]     skipped="0"
2025-06-01 19:42:37.161 Managed[722:68324]     errors="0">
2025-06-01 19:42:37.163 Managed[722:68324] <collection
2025-06-01 19:42:37.163 Managed[722:68324]     name="Collection"
2025-06-01 19:42:37.163 Managed[722:68324]     time="5,044204"
2025-06-01 19:42:37.163 Managed[722:68324]     total="0"
2025-06-01 19:42:37.164 Managed[722:68324]     passed="0"
2025-06-01 19:42:37.164 Managed[722:68324]     failed="0"
2025-06-01 19:42:37.164 Managed[722:68324]     skipped="0"
2025-06-01 19:42:37.164 Managed[722:68324]     errors="0"
2025-06-01 19:42:37.164 Managed[722:68324] >
2025-06-01 19:42:37.164 Managed[722:68324] </collection>
2025-06-01 19:42:37.164 Managed[722:68324] </assembly>
2025-06-01 19:42:37.165 Managed[722:68324] </assemblies>
2025-06-01 19:42:37.165 Managed[722:68324] Xml file was written to the provided writer.
2025-06-01 19:42:37.167 Managed[722:68324] 
=== TEST EXECUTION SUMMARY ===
Tests run: 0 Passed: 0 Inconclusive: 0 Failed: 0 Ignored: 0 Skipped: 0
The app terminated with the exit code 0.

Could you try to avoid running an empty app instead?

I'm a bit confused by the merged runner system, do you know where that change could be made?

@kotlarmilos
Copy link
Member

Let's test this with XHarness once dotnet/xharness#1416 is merged. In the meantime, try to disable the project build in the merged test runner (managed.csproj) by adding:

<DisableProjectBuild Condition="'$(RuntimeFlavor)' == 'mono'">true</DisableProjectBuild>

@matouskozak
Copy link
Member Author

/azp run runtime-extra-platforms

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes Apple mobile CI test reporting by switching from apple run to apple test to capture exit codes correctly, removes base64 encoding of XML results, disables problematic tests on Apple mobile and non-Windows Mono, and adds a MacCatalyst arm64/x64 pipeline.

  • Disable building managed tests on non-Windows Mono
  • Annotate and disable specific JIT tests on Apple mobile via ActiveIssue
  • Turn off base64 XML encoding in GeneratedTestRunner and add MacCatalyst pipeline

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/tests/managed/Managed.csproj Disable managed test build for Mono on non-Windows platforms
src/tests/JIT/opt/Structs/MemsetMemcpyNullref.* Add TestLibrary reference and ActiveIssue for Apple mobile
src/tests/JIT/Regression/**/GitHub_*.{csproj,cs,il} Add TestLibrary reference and ActiveIssue in each test
src/tests/Directory.Build.targets Remove AppleMobile test exclusion
src/tests/Common/XHarnessRunnerLibrary/RunnerEntryPoint.cs Disable base64 encoding of XML results
src/mono/msbuild/apple/build/AppleBuild.targets Enable UseMonoRuntime for Mono builds
eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml Add MacCatalyst arm64/x64 runtime test pipeline

@matouskozak matouskozak requested a review from akoeplinger June 9, 2025 10:18
@kotlarmilos kotlarmilos self-requested a review June 10, 2025 08:05
Copy link
Member

@kotlarmilos kotlarmilos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@matouskozak matouskozak enabled auto-merge (squash) June 10, 2025 10:47
@matouskozak
Copy link
Member Author

/ba-g Failure on extra-platforms are unrelated to this PR and are visible on main rolling builds

@matouskozak matouskozak merged commit 2ddab7b into dotnet:main Jun 10, 2025
203 of 206 checks passed
@matouskozak matouskozak deleted the fix-apple-runtime-test-xharness-reporting branch June 10, 2025 10:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants