runtime: Race detector causes exit status 0xc0000139 on Windows 11 with 1.21rc2 running gcc 9.2.0

[jameshartig]

What version of Go are you using (go version)?

$ go version
go version go1.21rc2 windows/amd64

Does this issue reproduce with the latest release?

Reproducible with 1.21rc2 but not reproducible with 1.20.5. I'm not sure how to download 1.21rc1 to test that.

What operating system and processor architecture are you using (go env)?

Windows 11 Pro Version 10.0.22621 Build 22621

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\james\AppData\Local\go-build
set GOENV=C:\Users\james\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\james\go\pkg\mod
set GONOPROXY=gerrit.levenlabs.com,gitlab.com/levenlabs
set GONOSUMDB=gerrit.levenlabs.com,gitlab.com/levenlabs
set GOOS=windows
set GOPATH=C:\Users\james\go;
set GOPRIVATE=gerrit.levenlabs.com,gitlab.com/levenlabs
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLCHAIN=auto
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.21rc2
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\james\Dropbox\aftermath\backend\go-llib\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=C:\Users\james\AppData\Local\Temp\go-build2611928154=/tmp/go-build -gno-record-gcc-switches

I had previously installed tdm-gcc 9.2.0 which causes the error:

gcc -v Output

> gcc -v
Using built-in specs.
COLLECT_GCC=C:\TDM-GCC-64\bin\gcc.exe
COLLECT_LTO_WRAPPER=C:/TDM-GCC-64/bin/../libexec/gcc/x86_64-w64-mingw32/9.2.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../../../src/gcc-git-9.2.0/configure --build=x86_64-w64-mingw32 --enable-targets=all --enable-languages=ada,c,c++,fortran,lto,objc,obj-c++ --enable-libgomp --enable-lto --enable-graphite --enable-cxx-flags=-DWINPTHREAD_STATIC --disable-build-with-cxx --disable-build-poststage1-with-cxx --enable-libstdcxx-debug --enable-threads=posix --enable-version-specific-runtime-libs --enable-fully-dynamic-string --enable-libstdcxx-threads --enable-libstdcxx-time --with-gnu-ld --disable-werror --disable-nls --disable-win32-registry --enable-large-address-aware --disable-rpath --disable-symvers --prefix=/mingw64tdm --with-local-prefix=/mingw64tdm --with-pkgversion=tdm64-1 --with-bugurl=http://tdm-gcc.tdragon.net/bugs
Thread model: posix
gcc version 9.2.0 (tdm64-1) 

But upon upgrading to tdm-gcc 10.3.0 the error no longer occurs:

gcc -v Output

> gcc -v
Using built-in specs.
COLLECT_GCC=C:\TDM-GCC-64\bin\gcc.exe
COLLECT_LTO_WRAPPER=C:/TDM-GCC-64/bin/../libexec/gcc/x86_64-w64-mingw32/10.3.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../../../src/gcc-git-10.3.0/configure --build=x86_64-w64-mingw32 --enable-targets=all --enable-languages=ada,c,c++,fortran,jit,lto,objc,obj-c++ --enable-libgomp --enable-lto --enable-graphite --enable-cxx-flags=-DWINPTHREAD_STATIC --disable-build-with-cxx --disable-build-poststage1-with-cxx --enable-libstdcxx-debug --enable-threads=posix --enable-version-specific-runtime-libs --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --disable-libstdcxx-pch --enable-libstdcxx-threads --enable-libstdcxx-time=yes --enable-mingw-wildcard --with-gnu-ld --disable-werror --enable-nls --disable-win32-registry --enable-large-address-aware --disable-rpath --disable-symvers --prefix=/mingw64tdm --with-local-prefix=/mingw64tdm --with-pkgversion=tdm64-1 --with-bugurl=https://github.com/jmeubank/tdm-gcc/issues
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.3.0 (tdm64-1)

What did you do?

Trying to run any tests with -race and gcc 9.2.0 ends up failing with exit status 0xc0000139.
https://go.dev/play/p/RFjBrSPywGg

> go.exe test ./... -v -race
exit status 0xc0000139

> go.exe test ./... -v
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS

But upon upgrading to gcc 10.3.0:

> go.exe test ./... -v -race
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS

What did you expect to see?

I expected for the race detector not to break between versions without something in the release notes pointing out an expectation of breakage. All I noticed in the release notes for 1.21 that might be related is:

On Windows AMD64, the linker (with help from the compiler) now emits SEH unwinding data by default, which improves the integration of Go applications with Windows debuggers and other tools.

But it's not obvious that this would require an upgraded gcc. It wasn't until after several hours of Googling and debugging that I ended up attempting to update gcc.

What did you see instead?

Instead I saw a bizarre exit status that led me down a deep rabbit hole on Google to try and figure out why my go tests suddenly don't work.

[jameshartig]

After reading through #35006 it seems like the recommendation is to use llvm. Downloading the latest ucrt from https://github.com/mstorsjo/llvm-mingw/releases and putting that in my path seems to also be working.

[bcmills]

Given that the binary works with a newer GCC, this sounds like a GCC bug rather than a Go bug.

That said, if we know of specific compilers that do or do not work with the race detector, or with Go in general, then probably we should add that detail to https://go.dev/wiki/MinimumRequirements and/or https://go.dev/doc/articles/race_detector#Requirements.

(CC @golang/windows, @golang/runtime)

[qmuntal]

GCC 9.2 is still widely used, we should understand what happened before the final go1.22 release, I'll investigate if SEH or the frame pointer changes have anything to do with this issue.

[qmuntal]

Did some archeology and found the culprit: the program fails to load the WakeByAddressSingle, WakeByAddressAll and WaitOnAddress symbols from kernel32.dll. These symbols are required by the new race_windows_amd64.syso library, updated in go1.21 as part of #49761, specifically in CL 420197.

The issue is that the missing symbols are not part of kernel32.dll since Windows 10, they are exported in libsynchronization.a instead. Mingw-w64 fixed this loading issue in v8.0.0 (mingw-w64/mingw-w64@68d0e7d), but tdm-gcc (which repackage mingw-w64) didn't port that change till v10 (jmeubank/mingw-w64@68d0e7d). Therefore, tdm-gcc v9 can't be used with go1.21.

That said, if we know of specific compilers that do or do not work with the race detector, or with Go in general, then probably we should add that detail to https://go.dev/wiki/MinimumRequirements and/or https://go.dev/doc/articles/race_detector#Requirements.

This was discussed and rejected in #35006. This comment made by @thanm gives a good reasoning: #35006 (comment).

[jameshartig]

@qmuntal thank you for the in-depth analysis and explanation! Does it make sense to mention this in the release notes at all if GCC 9 is still fairly popular?

[qmuntal]

Does it make sense to mention this in the release notes at all if GCC 9 is still fairly popular?

This is not a problem with GCC or Mingw-w64 9, they are both fine, but with tdm-gcc 9. I don't know how popular is tdm-gcc and if it worth adding this issue to the release notes, as add this kind of things in there.

[bcmills]

This was discussed and rejected in #35006. This comment made by @thanm gives a good reasoning: #35006 (comment).

I agree with that reasoning, but https://go.dev/wiki/MinimumRequirements#cgo is currently pretty unambiguous that “gcc 4.6 or newer is required”, implying that every version higher than that one is supported. I don't think it's accurate at this point.

I think what we really mean is “cmd/go will pass C compiler flags compatible with gcc 4.6 or newer”, which is a much weaker statement. I'm not sure exactly how to update the wiki page to reflect that.

[gopherbot]

Change https://go.dev/cl/508461 mentions this issue: _content/doc/articles/race_detector: document windows C compiler version requirements

[JAicewizard]

I am currently facing this on github CI with gcc 12.2 from mingw64, any tips on debugging this? @qmuntal do you remember how you found out what the issue was?