cmd/go: TestScripts fail when glibc is included with FLAGS

[nemith]

What version of Go are you using (go version)?

1.16.13

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOOS="linux"

What did you do?

Building go from source in a hermedic build environment where glibc, binutils, and gcc paths is provided with environment variables (PATH, C_INCLUDE_PATH, CFLAGS, LDFLAGS, etc)

PATH=/private/build/gcc/9.x/centos7-native/3bed279/bin:/private/build/binutils/2.32/centos7-native/da39a3e/bin:/bin:/usr/bin:/sbin:/usr/sbin
LIBRARY_PATH=/private/build/glibc/2.30/platform009/f259413/lib:/private/build/libgcc/9.x/platform009/9202ce7/lib:/private/build/glibc/2.30/platform009/f259413/lib
CMAKE_LIBRARY_PATH=/private/build/glibc/2.30/platform009/f259413/lib:/private/build/libgcc/9.x/platform009/9202ce7/lib
LDFLAGS='-L/private/build/glibc/2.30/platform009/f259413/lib -Wl,-rpath-link,/private/build/glibc/2.30/platform009/f259413/lib -Wl,--dynamic-linker,/usr/local/platforms/platform009/lib/ld.so -L/private/build/libgcc/9.x/platform009/9202ce7/lib -Wl,-rpath-link,/private/build/libgcc/9.x/platform009/9202ce7/lib -Wl,--hash-style,gnu'

What did you expect to see?

All tests pass

What did you see instead?

cmd/go TestScript fails for list_compiled_imports, ldflag, link_syso_issue33139, which utilize cgo.

--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/list_compiled_imports (0.60s)
        script_test.go:214: 
            # go list should report import "C" (0.122s)
            # go list -compiled should report imports in compiled files as well,
            # adding "runtime/cgo", "unsafe", and "syscall" but not dropping "C". (0.458s)
            > go list -compiled -f '{{.Imports}}'
            [stdout]
            [C unicode unicode/utf16 unsafe runtime/cgo syscall]
            [stderr]
            # x
            In file included from /usr/include/features.h:399,
                             from /usr/include/stdlib.h:24,
                             from _cgo_export.c:3:
            /usr/include/gnu/stubs.h:10:11: fatal error: gnu/stubs-64.h: No such file or directory
               10 | # include 
                  |           ^~~~~~~~~~~~~~~~
            compilation terminated.
            [exit status 2]
            FAIL: testdata/script/list_compiled_imports.txt:17: unexpected command failure
            
    --- FAIL: TestScript/ldflag (0.58s)
        script_test.go:214: 
            # Issue #42565 (0.000s)
            # We can't build package bad, which uses #cgo LDFLAGS. (0.295s)
            # We can build package ok with the same flags in CGO_LDFLAGS. (0.282s)
            > env CGO_LDFLAGS=-Wno-such-warning -Wno-unknown-warning-option
            -Wno-unknown-warning-option=
            > cd ../ok
            $WORK/gopath/src/ok
            > go build
            [stderr]
            # runtime/cgo
            In file included from /usr/include/features.h:399,
                             from /usr/include/stdlib.h:24,
                             from _cgo_export.c:3:
            /usr/include/gnu/stubs.h:10:11: fatal error: gnu/stubs-64.h: No such file or directory
               10 | # include 
                  |           ^~~~~~~~~~~~~~~~
            compilation terminated.
            [exit status 2]
            FAIL: testdata/script/ldflag.txt:13: unexpected command failure
            
    --- FAIL: TestScript/link_syso_issue33139 (1.02s)
        script_test.go:214: 
            # Test that we can use the external linker with a host syso file that is
            # embedded in a package, that is referenced by a Go assembly function.
            # See issue 33139. (0.000s)
            # External linking is not supported on linux/ppc64.
            # See: https://github.com/golang/go/issues/8912 (1.018s)
            > [linux] [ppc64] skip
            > cc -c -o syso/objTestImpl.syso syso/src/objTestImpl.c
            > go build -ldflags='-linkmode=external' ./cmd/main.go
            [stderr]
            # command-line-arguments
            /private/build/go/1.16.13/src/build-platform009/build/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
            /private/build/binutils/2.32/centos7-native/da39a3e/bin/ld: cannot find crt1.o: No such file or directory
            /private/build/binutils/2.32/centos7-native/da39a3e/bin/ld: cannot find crti.o: No such file or directory
            /private/build/binutils/2.32/centos7-native/da39a3e/bin/ld: cannot find -lpthread
            collect2: error: ld returned 1 exit status
            
            [exit status 2]
            FAIL: testdata/script/link_syso_issue33139.txt:12: unexpected command failure

Adding C_INCLUDE_PATH to extraEnvKeys fixes the import errors but adding LDFLAGS or CGO_LDFLAGS doesn't seem to be able to fix the linker problems.

These tests probably shouldn't assume that it can link against glibc without any flags.

[nemith]

Adding LIBRARY_PATH to the list of extraEnvKeys fixes ld errors. Thoughts on allowing C_INCLUDE_PATH and LIBRARY_PATH to that list?

[ianlancetaylor]

Seems reasonable. Thanks.

[gopherbot]

Change https://golang.org/cl/383334 mentions this issue: cmd/go: preserve LIBRARY_PATH and C_INCLUDE_PATH for script tests

[nemith]

Testing with a newer and incompatible glibc to the system glibc it seems I need at least CGO_LDFLAGS as well (and CGO_CFLAGS may be needed as well). I can add these as well, but wanted to see at how "pure" these tests need to be.

Ideally we want to run these when we update our glibc versions.