runtime: tests failing to terminate on ios-arm64-corellium

[bcmills]

https://build.golang.org/log/5fd56793a86a5e6392ab13c645ab0f9b24fa3268:

Installed Go for ios/arm64 in /tmp/workdir-host-ios-arm64-corellium-ios/go
Installed commands in /tmp/workdir-host-ios-arm64-corellium-ios/go/bin
Test "go_test:archive/tar" ran over 20m0s limit (20m0.000566363s); saw output:

XXXBANNERXXX:Test execution environment.
# GOARCH: arm64
# CPU: 
# GOOS: ios
# OS Version: Darwin 20.6.0 Darwin Kernel Version 20.6.0: Tue Aug 24 21:10:43 PDT 2021; root:xnu-7195.140.44~1/RELEASE_ARM64_T8010 iPhone9,1

XXXBANNERXXX:Testing packages.

The hanging test command is for archive/tar, but I don't think the specific package is relevant — that just happens to be the very first test in the run.

[bcmills]

This may have started on or around CL 454836, but since this builder is slow and heavily backlogged we don't have a clean bisection in the dashboard runs.

It appears that something in either the toolchain or the test process itself is failing to terminate properly. However, due to #53201 I can't even tell which process is hung.

@golang/release, can we prioritize a fix for #53201 to unblock investigation of this issue?

(CC @golang/runtime @golang/ios)

[heschi]

Since ios-arm64 is not a first class port, I'd like the iOS maintainers to spend at least some time investigating before we discuss prioritizing #53201.

[changkun]

Could you maybe clarify what needs to be investigated? It is not entirely clear after a large number of CLs are landed.

[heschi]

Essentially, see why the tests are hanging. archive/tar is the first test to run, so I suspect the port is completely broken.

[cherrymui]

It is not always the first test timed out (although it seems the case in a few most recent runs). https://build.golang.org/log/fe7dc38047bccedcf67640847ff602864fabc40b includes a few successful tests before hanging.

[cherrymui]

For ones who have access to the builder, would it be possible to see which process(es) hang, and would it be possible to send it a SIGQUIT and get a stack dump? Thanks.

[changkun]

Based on what I saw from the Corellium dashboard, the Go team has admin access to these builders (although I don't know how it is access from the Go team side)

@cherrymui, could you help me understand exactly what you expect me to do in which steps? Do you mean to kill the builtlet process and share the core dump file?

[cherrymui]

I'm not familiar with the builder infrastructure, so I don't know exactly. If you can log in to the builder machine/VM that is running the build, maybe run command like ps or top to see which processes are running, and send a SIGQUIT to a long-running one?

But before we spend more time on this, I'd like to hear some answer about the meta-question: if this is really due to cgo not enabled, do we really want to support non-cgo programs on iOS? If the answer is no, cmd/dist shouldn't disable cgo. If the answer is yes, maybe we want to support internal linking, so it is actually pure Go.

[bcmills]

would it be possible to see which process(es) hang, and would it be possible to send it a SIGQUIT and get a stack dump?

That is exactly why #53201 is relevant. The buildlet should be sending SIGQUIT itself in case of timeouts (on all platforms that support it, not just ios), but today does not seem to be doing so.

[gopherbot]

Change https://go.dev/cl/463739 mentions this issue: cmd/dist: leave cgo enabled if external linking is required

[dr2chase]

Is this fixed?

[bcmills]

This is indeed fixed by CL 463739, which re-enables cgo when building cmd on this platform.

I suspect that it was related to some iffy behavior in runtime/cgo on the ios/arm64 platform (filed as #58225), although there may be other issues involved as well.