Skip to content

[tests] Check for emulator issues and error early #660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 22, 2017

Conversation

jonpryor
Copy link
Member

The Android emulator is a "wonderful" beast. Case in point:
PR Build #1074, which hung during emulator startup, for
5 hours:

09:13:49 Hax is enabled
09:13:49 Hax ram_size 0x0
09:13:49 HAX is working and emulator runs in fast virt mode.
...
09:13:56 		adb I 06-21 09:13:56 47148 21294922 adb_io.cpp:75] readx: fd=3 wanted=4
...
# 5.5 hours later...
14:47:17 		Tool /Users/builder/android-toolchain/sdk/platform-tools/adb execution finished.

It "finished" because I manually killed the Job, as it had take over
6 hours, cumulative, to do nothing.

(Arguably we shouldn't have a 10 hour timeout on Jenkins jobs, but
Mono bumps can take upwards of 6 hours to build, so a 6 hour time
isn't necessarily unusual...)

This also isn't our first attempt at improving emulator reliability.
See also commits c089267, bc6440b, 3fa9e9e, b54f8cd, 3b893cd,
7450efc, and 6358a64. (Is that enough? Likely not. Is that all of
them? Probably not.)

Take yet another stab at improving things: in this case, improving
error checking, so that when an error occurs we can fail early, as
opposed to waiting for hours on end for a Jenkins timeout to occur
or for me to get annoyed at general Jenkins build slowness enough to
track down the hung job and manually kill it...

Update the emulator process launch so that we have event handlers
for stdout and stderr messages, so that we can look for currently
known error conditions, which fall into two categories:

  1. HAXM needs reinstallation, or
  2. Another VM is in use.

HAXM appears to need reinstallation when no memory is allocated:

Hax ram_size 0x0

I don't know why HAXM would require re-installation in these
circumstances, but that generally appears to fix the problem.

When another VM is in use, e.g. when I'm running a Veertu VM on macOS,
emulator also fails to launch, writing the following to stderr:

Failed to sync vcpu reg
Failed to sync HAX vcpu contextInternal error: Initial hax sync failed

If either of these conditions occur, the <StartAndroidEmulator/>
task should fail, with a (hopefully) relevant error message.

With these changes in place, remove the <Sleep/> task invocations
from the AcquireAndroidTarget target, as the new 20 second timeout
in <StartAndroidEmulator/> removes the need for the first Sleep, and
the 2nd sleep doesn't appear to do any good.

Followup to c089267!

in case the added sleep doesn't help, we might at least get some
more useful information

We do get more information. It's not entirely useful: adb is
waiting for adbd to provide it information, and that information
never arrives:

adb I 06-21 09:13:56 47148 21294922 adb_io.cpp:75] readx: fd=3 wanted=4

...and there it waits, seemingly forever, until Job timeout or Job
death occurs.

The Android emulator is a "wonderful" beast. Case in point:
[PR Build dotnet#1074][0], which *hung* during emulator startup, for
*5 hours*:

	09:13:49 Hax is enabled
	09:13:49 Hax ram_size 0x0
	09:13:49 HAX is working and emulator runs in fast virt mode.
	...
	09:13:56 		adb I 06-21 09:13:56 47148 21294922 adb_io.cpp:75] readx: fd=3 wanted=4
	...
	# 5.5 hours later...
	14:47:17 		Tool /Users/builder/android-toolchain/sdk/platform-tools/adb execution finished.

It "finished" because I manually killed the Job, as it had take over
6 *hours*, cumulative, to do nothing.

(*Arguably* we shouldn't have a 10 hour timeout on Jenkins jobs, but
Mono bumps can take upwards of 6 hours to build, so a 6 hour time
isn't necessarily unusual...)

This also isn't our first attempt at improving emulator reliability.
See also commits c089267, bc6440b, 3fa9e9e, b54f8cd, 3b893cd,
7450efc, and 6358a64. (Is that enough? Likely not. Is that all of
them? Probably not.)

Take yet another stab at improving things: in this case, improving
error checking, so that *when* an error occurs we can *fail early*, as
opposed to waiting for hours on end for a Jenkins timeout to occur
*or* for me to get annoyed at general Jenkins build slowness enough to
track down the hung job and manually kill it...

Update the `emulator` process launch so that we have event handlers
for stdout and stderr messages, so that we can look for currently
known error conditions, which fall into two categories:

 1. HAXM needs reinstallation, or
 2. Another VM is in use.

HAXM appears to need reinstallation when no memory is allocated:

	Hax ram_size 0x0

I don't know *why* HAXM would require *re*-installation in these
circumstances, but that generally appears to fix the problem.

When another VM is in use, e.g. when I'm running a Veertu VM on macOS,
`emulator` also fails to launch, writing the following to stderr:

	Failed to sync vcpu reg
	Failed to sync HAX vcpu contextInternal error: Initial hax sync failed

If either of these conditions occur, the `<StartAndroidEmulator/>`
task should *fail*, with a (hopefully) relevant error message.

With these changes in place, remove the `<Sleep/>` task invocations
from the `AcquireAndroidTarget` target, as the new 20 *second* timeout
in `<StartAndroidEmulator/>` removes the need for the first Sleep, and
the 2nd sleep doesn't appear to do any good.

Followup to c089267!

> in case the added sleep doesn't help, we might at least get some
> more useful information

We do get more information. It's not entirely *useful*: `adb` is
waiting for `adbd` to provide it information, and that information
never arrives:

	adb I 06-21 09:13:56 47148 21294922 adb_io.cpp:75] readx: fd=3 wanted=4

...and there it waits, seemingly forever, until Job timeout or Job
death occurs.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/1074/
@radekdoulik
Copy link
Member

The build took very long time. It looks like Azure related though, so hopefully next build will be OK.

03:30:03 	 0 Warning(s)
03:30:03 	 0 Error(s)
03:30:03 
03:30:03 Time Elapsed 00:01:02.7685530
03:30:03 Upload artifacts to Microsoft Azure Blob storage is waiting for a checkpoint on xamarin-android-pr-builder #1078
07:46:06 MicrosoftAzureStorage - Uploading files to Microsoft Azure```

@radekdoulik radekdoulik merged commit db668ba into dotnet:master Jun 22, 2017
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Jun 15, 2020
Changes: dotnet/java-interop@b00e644...1de5501

  * dotnet/java-interop@1de5501: [CI] Add an "macOS .NET Core" build (dotnet#655)
  * dotnet/java-interop@9a56465: [generator] Use proper syntax for nested classes for DIM invokers (dotnet#662)
  * dotnet/java-interop@267c3f3: [generator] Support XML defined enums with no JNI info (dotnet#659)
  * dotnet/java-interop@5dcf896: [generator] Don't invalidate interface if static method is invalidated (dotnet#660)
  * dotnet/java-interop@1b59dcc: [java-interop] Update to SDK style project (dotnet#657)
  * dotnet/java-interop@b136ac9: [param-name-importer] Bump to Microsoft.Xml.SgmlReader 1.8.16 (dotnet#656)
  * dotnet/java-interop@eb39a3a: Bump to xamarin/xamarin-android-tools/master@3974fc38 (dotnet#658)
  * dotnet/java-interop@a99b451: [logcat-parse] Update to Mono.Terminal 5.4.0 (dotnet#654)
  * dotnet/java-interop@425f79d: [Java.Interop] Fix C# warnings (dotnet#652)
  * dotnet/java-interop@2e0f55d: [Java.Interop.Tools.Generator] Specify $(OutputPath) (dotnet#650)
  * dotnet/java-interop@4b266fa: [Xamarin.Android.Tools.Bytecode] Support @jvmoverloads (dotnet#651)
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Jun 15, 2020
Changes: dotnet/java-interop@76d1ac7...1f3388a

  * dotnet/java-interop@1f3388a: [generator] Use proper syntax for nested classes for DIM invokers (dotnet#662)
  * dotnet/java-interop@5e23163: [generator] Support XML defined enums with no JNI info (dotnet#659)
  * dotnet/java-interop@5c4581d: [generator] Don't invalidate interface if static method is invalidated (dotnet#660)
  * dotnet/java-interop@9687bb5: Bump to xamarin/xamarin-android-tools/d16-7@017078f2 (dotnet#658)
  * dotnet/java-interop@abfade5: [Java.Interop] Fix C# warnings (dotnet#652)
  * dotnet/java-interop@fb6d5f9: [Java.Interop.Tools.Generator] Specify $(OutputPath) (dotnet#650)
  * dotnet/java-interop@a3f148c: [Xamarin.Android.Tools.Bytecode] Support @jvmoverloads (dotnet#651)
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Jun 15, 2020
Changes: dotnet/java-interop@76d1ac7...1f3388a

  * dotnet/java-interop@1f3388a: [generator] Use proper syntax for nested classes for DIM invokers (dotnet#662)
  * dotnet/java-interop@5e23163: [generator] Support XML defined enums with no JNI info (dotnet#659)
  * dotnet/java-interop@5c4581d: [generator] Don't invalidate interface if static method is invalidated (dotnet#660)
  * dotnet/java-interop@9687bb5: Bump to xamarin/xamarin-android-tools/d16-7@017078f2 (dotnet#658)
  * dotnet/java-interop@abfade5: [Java.Interop] Fix C# warnings (dotnet#652)
  * dotnet/java-interop@fb6d5f9: [Java.Interop.Tools.Generator] Specify $(OutputPath) (dotnet#650)
  * dotnet/java-interop@a3f148c: [Xamarin.Android.Tools.Bytecode] Support @jvmoverloads (dotnet#651)
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Jun 15, 2020
Changes: dotnet/java-interop@b00e644...1de5501

  * dotnet/java-interop@1de5501: [CI] Add an "macOS .NET Core" build (dotnet#655)
  * dotnet/java-interop@9a56465: [generator] Use proper syntax for nested classes for DIM invokers (dotnet#662)
  * dotnet/java-interop@267c3f3: [generator] Support XML defined enums with no JNI info (dotnet#659)
  * dotnet/java-interop@5dcf896: [generator] Don't invalidate interface if static method is invalidated (dotnet#660)
  * dotnet/java-interop@1b59dcc: [java-interop] Update to SDK style project (dotnet#657)
  * dotnet/java-interop@b136ac9: [param-name-importer] Bump to Microsoft.Xml.SgmlReader 1.8.16 (dotnet#656)
  * dotnet/java-interop@eb39a3a: Bump to xamarin/xamarin-android-tools/master@3974fc38 (dotnet#658)
  * dotnet/java-interop@a99b451: [logcat-parse] Update to Mono.Terminal 5.4.0 (dotnet#654)
  * dotnet/java-interop@425f79d: [Java.Interop] Fix C# warnings (dotnet#652)
  * dotnet/java-interop@2e0f55d: [Java.Interop.Tools.Generator] Specify $(OutputPath) (dotnet#650)
  * dotnet/java-interop@4b266fa: [Xamarin.Android.Tools.Bytecode] Support @jvmoverloads (dotnet#651)
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Jun 15, 2020
Changes: dotnet/java-interop@b00e644...1de5501

  * dotnet/java-interop@1de5501: [CI] Add an "macOS .NET Core" build (dotnet#655)
  * dotnet/java-interop@9a56465: [generator] Use proper syntax for nested classes for DIM invokers (dotnet#662)
  * dotnet/java-interop@267c3f3: [generator] Support XML defined enums with no JNI info (dotnet#659)
  * dotnet/java-interop@5dcf896: [generator] Don't invalidate interface if static method is invalidated (dotnet#660)
  * dotnet/java-interop@1b59dcc: [java-interop] Update to SDK style project (dotnet#657)
  * dotnet/java-interop@b136ac9: [param-name-importer] Bump to Microsoft.Xml.SgmlReader 1.8.16 (dotnet#656)
  * dotnet/java-interop@eb39a3a: Bump to xamarin/xamarin-android-tools/master@3974fc38 (dotnet#658)
  * dotnet/java-interop@a99b451: [logcat-parse] Update to Mono.Terminal 5.4.0 (dotnet#654)
  * dotnet/java-interop@425f79d: [Java.Interop] Fix C# warnings (dotnet#652)
  * dotnet/java-interop@2e0f55d: [Java.Interop.Tools.Generator] Specify $(OutputPath) (dotnet#650)
  * dotnet/java-interop@4b266fa: [Xamarin.Android.Tools.Bytecode] Support @jvmoverloads (dotnet#651)
jonpryor added a commit that referenced this pull request Jun 15, 2020
Changes: dotnet/java-interop@76d1ac7...1f3388a

  * dotnet/java-interop@1f3388a: [generator] Use proper syntax for nested classes for DIM invokers (#662)
  * dotnet/java-interop@5e23163: [generator] Support XML defined enums with no JNI info (#659)
  * dotnet/java-interop@5c4581d: [generator] Don't invalidate interface if static method is invalidated (#660)
  * dotnet/java-interop@9687bb5: Bump to xamarin/xamarin-android-tools/d16-7@017078f2 (#658)
  * dotnet/java-interop@abfade5: [Java.Interop] Fix C# warnings (#652)
  * dotnet/java-interop@fb6d5f9: [Java.Interop.Tools.Generator] Specify $(OutputPath) (#650)
  * dotnet/java-interop@a3f148c: [Xamarin.Android.Tools.Bytecode] Support @jvmoverloads (#651)
jonpryor added a commit that referenced this pull request Jun 16, 2020
Changes: dotnet/java-interop@b00e644...1de5501

  * dotnet/java-interop@1de5501: [CI] Add an "macOS .NET Core" build (#655)
  * dotnet/java-interop@9a56465: [generator] Use proper syntax for nested classes for DIM invokers (#662)
  * dotnet/java-interop@267c3f3: [generator] Support XML defined enums with no JNI info (#659)
  * dotnet/java-interop@5dcf896: [generator] Don't invalidate interface if static method is invalidated (#660)
  * dotnet/java-interop@1b59dcc: [java-interop] Update to SDK style project (#657)
  * dotnet/java-interop@b136ac9: [param-name-importer] Bump to Microsoft.Xml.SgmlReader 1.8.16 (#656)
  * dotnet/java-interop@eb39a3a: Bump to xamarin/xamarin-android-tools/master@3974fc38 (#658)
  * dotnet/java-interop@a99b451: [logcat-parse] Update to Mono.Terminal 5.4.0 (#654)
  * dotnet/java-interop@425f79d: [Java.Interop] Fix C# warnings (#652)
  * dotnet/java-interop@2e0f55d: [Java.Interop.Tools.Generator] Specify $(OutputPath) (#650)
  * dotnet/java-interop@4b266fa: [Xamarin.Android.Tools.Bytecode] Support @jvmoverloads (#651)
@github-actions github-actions bot locked and limited conversation to collaborators Feb 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants