Cgo and --compiler=gcc

[jayconrod]

Copied from bazel-go-discuss. cc @nathantchan

---

One of our verification builds uses gcc, and I currently cannot get past the installation portion of the stdlib rule. It's throwing errors like the following:

not-int-const: In function '__cgo_f_1_3':
not-int-const:1:60: error: invalid type argument of unary '*' (have 'int')
not-num-const: In function '__cgo_f_1_4':
not-num-const:1:73: error: expected expression before ';' token
not-str-lit: In function '__cgo_f_1_5':
not-str-lit:1:73: error: expected expression before ';' token
not-type: In function '__cgo_f_2_2':
not-type:2:39: error: '__cgo_undefined__2' undeclared (first use in this function)
not-type:2:39: note: each undeclared identifier is reported only once for each function it appears in
not-type:2:38: error: invalid operands to binary * (have 'int (*)(const struct sockaddr * __restrict__,  socklen_t,  char * __restrict__,  socklen_t,  char * __restrict__,  socklen_t,  int)' and 'const char *')
not-int-const: In function '__cgo_f_2_3':
not-int-const:2:67: error: invalid operands to binary * (have 'int (*)(const struct sockaddr * __restrict__,  socklen_t,  char * __restrict__,  socklen_t,  char * __restrict__,  socklen_t,  int)' and 'int')
not-num-const: In function '__cgo_f_2_4':
not-num-const:2:67: error: incompatible types when initializing type 'double' using type 'int (*)(const struct sockaddr * __restrict__,  socklen_t,  char * __restrict__,  socklen_t,  char * __restrict__,  socklen_t,  int)'
not-str-lit: In function '__cgo_f_2_5':
not-str-lit:2:1: error: invalid initializer
not-int-const: In function '__cgo_f_3_3':
not-int-const:3:65: error: invalid type argument of unary '*' (have 'int')
not-int-const:3:33: error: enumerator value for '__cgo_undefined__3' is not an integer constant
not-num-const: In function '__cgo_f_3_4':
not-num-const:3:78: error: expected expression before ';' token
not-num-const:3:78: error: initializer element is not constant
not-str-lit: In function '__cgo_f_3_5':
not-str-lit:3:78: error: expected expression before ';' token
not-str-lit:3:78: error: invalid initializer
completed: At top level:
completed:1:16: error: '__cgo__2' undeclared here (not in a function)
completed:1:1: error: initializer element is not constant

It doesn't seem like setting --compiler=gcc works right now in 0.9.0 of the go rules. Any ideas?

$ bazel test --compiler=gcc tests/...
...................
ERROR: No toolchain found for --cpu='k8' --compiler='gcc'. Valid toolchains are: [
  --cpu='armeabi-v7a' --compiler='compiler' --glibc='armeabi-v7a',
  --cpu='ios_x86_64' --compiler='compiler' --glibc='ios',
  --cpu='k8' --compiler='compiler' --glibc='local',
  --cpu='x64_windows' --compiler='cl' --glibc='msvcrt140',
]

[jayconrod]

Cgo determines the types of symbols by attempting to compile a specially crafted C file, then checking whether errors occur in specific places. It's possible it's expecting the compiler to behave a certain way, but a different compiler is actually being used?

Could you tell me a bit more about your environment? OS, compiler version, Bazel version? What is the command that produces the top error message? Does anything different happen if the environment variable CC=/usr/bin/gcc is set?

[nathantchan]

OS: debian jessie
compiler version:
Using built-in specs.
COLLECT_GCC=/usr/bin/x86_64-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.9.2-10+deb8u1' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10+deb8u1)
bazel version: 0.9.0

Here's the exact subcommand being run:
'export BAZEL_OUTPUT_ROOT="$(pwd)/" GOROOT="$(pwd)/bazel-out/k8-fastbuild/bin/external/go_stdlib_linux_amd64_cgo" GOROOT_FINAL="GOROOT" GOOS="linux" GOARCH="amd64" CGO_ENABLED="1" CC="$(pwd)/tools/cpp/debian_amd64_compilers/x86_64-linux-gnu-gcc" CXX="$(pwd)/tools/cpp/debian_amd64_compilers/x86_64-linux-gnu-gcc" COMPILER_PATH="tools/cpp/debian_amd64_compilers" CGO_CPPFLAGS="-nostdinc -isystem $BAZEL_OUTPUT_ROOT/external/amd64_compilers_repo/usr/lib/gcc/x86_64-linux-gnu/4.9/include -isystem $BAZEL_OUTPUT_ROOT/external/amd64_compilers_repo/usr/local/include -isystem $BAZEL_OUTPUT_ROOT/external/amd64_compilers_repo/usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed -isystem $BAZEL_OUTPUT_ROOT/external/amd64_compilers_repo/usr/include/x86_64-linux-gnu -isystem $BAZEL_OUTPUT_ROOT/external/amd64_compilers_repo/usr/include -L$BAZEL_OUTPUT_ROOT/external/amd64_compilers_repo/lib/x86_64-linux-gnu -fmax-errors=20 -pipe -ggdb3 -pthread -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -fstack-protector -fPIE -fdiagnostics-color -fno-omit-frame-pointer -fno-strict-aliasing" CGO_LDFLAGS="-pipe -lstdc++ -B$BAZEL_OUTPUT_ROOT/external/amd64_compilers_repo/usr/bin -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes -Wl,--build-id=md5 -Wl,--hash-style=gnu -fuse-ld=gold -Wl,--warn-execstack -Wl,--detect-odr-violations" && export PATH=$PATH:$(cd "$COMPILER_PATH" && pwd) && mkdir -p bazel-out/k8-fastbuild/bin/external/go_stdlib_linux_amd64_cgo/src && mkdir -p bazel-out/k8-fastbuild/bin/external/go_stdlib_linux_amd64_cgo/pkg && cp -rf external/go_sdk/src/* bazel-out/k8-fastbuild/bin/external/go_stdlib_linux_amd64_cgo/src/ && cp -rf external/go_sdk/pkg/tool bazel-out/k8-fastbuild/bin/external/go_stdlib_linux_amd64_cgo/pkg/ && cp -rf external/go_sdk/pkg/include bazel-out/k8-fastbuild/bin/external/go_stdlib_linux_amd64_cgo/pkg/ && external/go_sdk/bin/go install -asmflags "-trimpath $(pwd)" std && external/go_sdk/bin/go install -asmflags "-trimpath $(pwd)" runtime/cgo'

The environment variable BAZEL_OUTPUT_ROOT is set to the execution root of the sandbox. CC is currently pointing to a shell script that converts the full paths to relative paths. Within that shell script if I change the compiler to point to /usr/bin/gcc instead, I get a similar set of errors.

If I don't try to set a full path for all of the -isystem options, then the install stops at an earlier stage where it cannot find the basic header files like stddef.h

[nathantchan]

Actually if I remove the environment variables COMPILER_PATH and CGO_CPP_FLAGS, everything now succeeds. This is true is I set CC to either /usr/bin/gcc, or the sandboxed compiler we have in external/amd64_compilers. Maybe it's one of the flags that is throwing things off?

[jayconrod]

Thanks for the info. I'll try and reproduce this, but I probably won't be able to get to it until tomorrow or Friday.

Something to try before then: what happens if you remove the -fdiagnostics-color flag from CGO_CPPFLAGS? We have blacklisted -fcolor-diagnostics because cgo doesn't understand color codes emitted by the compiler. Maybe this is a different name for the same thing.

Do you see the same error using a newer version of rules_go? We rewrote the stdlib builder where the error is happening, I think after 0.9.0. Might fix this.

Also, just to confirm: are you using the Bazel auto-configuration (detects compiler in PATH or CC) or do you have a custom CROSSTOOL file?

[nathantchan]

Removing the -fdiagnostics-color flag does not solve the problem.

I just tried upgrading to a 0.10.1. It's now failing out at an earlier stage because it cannot find the compiler we placed in external/amd64_compilers_repo/usr/bin/x86_64-linux-gnu-gcc.

We are using a custom CROSSTOOL file.

[jayconrod]

Ok, let's proceed with 0.10.1 if possible.

I'm only vaguely familiar with custom CROSSTOOL files. I'll help as much as I can, but I'm coming up to speed on this. :)

Could you paste the error you're getting about the compiler being missing? I'm wondering if this is a Bazel error (some dependency missing) or an execution error (bad path on the command line or something).

If you're able to get to the part where it builds the stdlib, could you paste the command line? I'm interested in both the command line for stdlib builder (Bazel will print this with the -s flag) and the command line the stdlib builder passes to go install. There's no simple way to get this. You'll need to check out a local copy of rules_go, apply the patch below, then build with --override_repository=io_bazel_rules_go=/path/to/local/copy.

diff --git a/go/tools/builders/stdlib.go b/go/tools/builders/stdlib.go
index 4f9264e..86126a1 100644
--- a/go/tools/builders/stdlib.go
+++ b/go/tools/builders/stdlib.go
@@ -21,6 +21,7 @@ import (
 	"log"
 	"os"
 	"os/exec"
+	"strings"
 )
 
 func install_stdlib(goenv *GoEnv, target string, args []string) error {
@@ -32,6 +33,7 @@ func install_stdlib(goenv *GoEnv, target string, args []string) error {
 	cmd.Stdout = os.Stdout
 	cmd.Stderr = os.Stderr
 	cmd.Env = goenv.Env()
+	fmt.Fprintf(os.Stderr, ">> stdlib:\n%s \\\n%s %s\n", strings.Join(cmd.Env, " \\\n"), cmd.Path, strings.Join(cmd.Args, " "))
 	if err := cmd.Run(); err != nil {
 		return fmt.Errorf("error running go install %s: %v", target, err)
 	}

[nathantchan]

I suspect it's a bad path. Here's the error I'm currently blocked on:

ERROR: /home/nathan/.cache/bazel/_bazel_nathan/8a7aba3b76b230775497a5ea4d4123bb/external/io_bazel_rules_go/BUILD.bazel:9:1: GoStdlib external/io_bazel_rules_go/linux_amd64/stdlib~/pkg failed (Exit 1)
>> stdlib:
BAZEL_OUTPUT_ROOT=$(pwd)/ \
TMPDIR=/dev/shm/bazel-sandbox.d71478fcc543f4fd657affe6afd3ad99/8504803830990673641/execroot/com_peloton_tech/tmp \
GOROOT=/dev/shm/bazel-sandbox.d71478fcc543f4fd657affe6afd3ad99/8504803830990673641/execroot/com_peloton_tech/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64/stdlib~ \
GOROOT_FINAL=GOROOT \
GOOS=linux \
GOARCH=amd64 \
CGO_ENABLED=1 \
PATH=/dev/shm/bazel-sandbox.d71478fcc543f4fd657affe6afd3ad99/8504803830990673641/execroot/com_peloton_tech/tools/cpp/debian_amd64_compilers \
COMPILER_PATH=/dev/shm/bazel-sandbox.d71478fcc543f4fd657affe6afd3ad99/8504803830990673641/execroot/com_peloton_tech/tools/cpp/debian_amd64_compilers \
CC=/dev/shm/bazel-sandbox.d71478fcc543f4fd657affe6afd3ad99/8504803830990673641/execroot/com_peloton_tech/tools/cpp/debian_amd64_compilers/x86_64-linux-gnu-gcc \
CXX=/dev/shm/bazel-sandbox.d71478fcc543f4fd657affe6afd3ad99/8504803830990673641/execroot/com_peloton_tech/tools/cpp/debian_amd64_compilers/x86_64-linux-gnu-gcc \
CGO_CPPFLAGS=-nostdinc -isystem external/amd64_compilers_repo/usr/lib/gcc/x86_64-linux-gnu/4.9/include -isystem external/amd64_compilers_repo/usr/local/include -isystem external/amd64_compilers_repo/usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed -isystem external/amd64_compilers_repo/usr/include/x86_64-linux-gnu -isystem external/amd64_compilers_repo/usr/include -Lexternal/amd64_compilers_repo/lib/x86_64-linux-gnu -Wextra -fmax-errors=20 -pipe -ggdb3 -pthread -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -fstack-protector -fPIE -fdiagnostics-color -fno-omit-frame-pointer -fno-strict-aliasing \
CGO_LDFLAGS=-pipe -lstdc++ -Bexternal/amd64_compilers_repo/usr/bin -pie -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes -Wl,--build-id=md5 -Wl,--hash-style=gnu -fuse-ld=gold -Wl,--warn-execstack -Wl,--detect-odr-violations \
external/go_sdk/bin/go external/go_sdk/bin/go install -asmflags -trimpath /dev/shm/bazel-sandbox.d71478fcc543f4fd657affe6afd3ad99/8504803830990673641/execroot/com_peloton_tech std
# runtime/cgo
/dev/shm/bazel-sandbox.d71478fcc543f4fd657affe6afd3ad99/8504803830990673641/execroot/com_peloton_tech/tools/cpp/debian_amd64_compilers/x86_64-linux-gnu-gcc: line 56: $(pwd)/external/amd64_compilers_repo/usr/bin/x86_64-linux-gnu-gcc: No such file or directory

Do you know where the commands are being executed?

[nathantchan]

Forgot to paste in the subcommand:

SUBCOMMAND: # @io_bazel_rules_go//:stdlib [action 'GoStdlib external/io_bazel_rules_go/linux_amd64/stdlib~/pkg']
(cd /home/nathan/.cache/bazel/_bazel_nathan/8a7aba3b76b230775497a5ea4d4123bb/execroot/com_peloton_tech &&
exec env -
BAZEL_OUTPUT_ROOT='$(pwd)/'
bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/stdlib -go external/go_sdk/bin/go -root_file external/go_sdk/packages.txt -goos linux -goarch amd64 '-cgo=1' -compiler_path tools/cpp/debian_amd64_compilers/../../../ -cc tools/cpp/debian_amd64_compilers/x86_64-linux-gnu-gcc -cpp_flag -nostdinc -cpp_flag -isystem -cpp_flag external/amd64_compilers_repo/usr/lib/gcc/x86_64-linux-gnu/4.9/include -cpp_flag -isystem -cpp_flag external/amd64_compilers_repo/usr/local/include -cpp_flag -isystem -cpp_flag external/amd64_compilers_repo/usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed -cpp_flag -isystem -cpp_flag external/amd64_compilers_repo/usr/include/x86_64-linux-gnu -cpp_flag -isystem -cpp_flag external/amd64_compilers_repo/usr/include -cpp_flag -Lexternal/amd64_compilers_repo/lib/x86_64-linux-gnu -cpp_flag -Wextra -cpp_flag '-fmax-errors=20' -cpp_flag -pipe -cpp_flag -ggdb3 -cpp_flag -pthread -cpp_flag -U_FORTIFY_SOURCE -cpp_flag '-D_FORTIFY_SOURCE=1' -cpp_flag -fstack-protector -cpp_flag -fPIE -cpp_flag -fdiagnostics-color -cpp_flag -fno-omit-frame-pointer -cpp_flag -fno-strict-aliasing -ld_flag -pipe -ld_flag -lstdc++ -ld_flag -Bexternal/amd64_compilers_repo/usr/bin -ld_flag -pie -ld_flag -Wl,-z,relro,-z,now -ld_flag -no-canonical-prefixes -ld_flag -pass-exit-codes -ld_flag '-Wl,--build-id=md5' -ld_flag '-Wl,--hash-style=gnu' -ld_flag '-fuse-ld=gold' -ld_flag -Wl,--warn-execstack -ld_flag -Wl,--detect-odr-violations -out 'bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64/stdlib~')

[jayconrod]

The commands will be executed in the execroot. Bazel prints it before each command. In this case,

/home/nathan/.cache/bazel/_bazel_nathan/8a7aba3b76b230775497a5ea4d4123bb/execroot/com_peloton_tech

I don't see anything odd, other than the relative paths for the includes. They might not be in the sandbox. Does this work if you build with --spawn_strategy=standalone (disabling the sandbox)?

The way I usually debug is to disable sandboxing, then execute commands manually in the execroot until I find something that works. Maybe try removing some of the flags and see what affects the build?

[AustinSchuh]

I'm going to put together a reproduction repo to send you. I'm collecting up a fully hermetic compiler that I can share. I'm hoping doing that will make it easier to reproduce issues. I'm assuming if I were to get you a repo for Linux, that would be helpful.

[AustinSchuh]

@jayconrod , https://github.com/AustinSchuh/golang_demo is a working demo. It's setup for Debian Jessie (Debian 8) from our robotics repo (4 target architectures). If you edit tools/bazel.rc and remove the crosstool_top flags, everything works. If you add them back, you get cgo errors. I included a trivial C++ application as an example to show that the CROSSTOOL file works.

@nathantchan @bsilver8192 and I have some modifications we've had to make to get rules_go working with a similar crosstool setup. It would be nice to figure out how to get this setup to work out of the box to make future upgrades less painful. Once this works, we can figure out how to get a repo which has both GCC and Clang in it to show you that as well.

As a side note, that should be a fully reproducible and hermetic compiler. I did some basic md5sum tests on that repo.

[jayconrod]

@AustinSchuh Thanks for posting that! I was able to reproduce the failure.

/usr/local/google/home/jayconrod/.cache/bazel/_bazel_jayconrod/abc4891756130f6096412cdbce30b66a/execroot/__main__/tools/cpp/clang_3p6/x86_64-linux-gnu-clang-3.6: line 13: bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/src/runtime/cgo/external/clang_3p6_repo/usr/bin/clang-3.6: No such file or directory

The script tools/cpp/clang_3p6/x86_64-linux-gnu-clang-3.6 sets LD_LIBRARY_PATH and execs the compiler. However, it expands ${BAZEL_OUTPUT_ROOT}, which is not normally set for these actions. Maybe it's set for C/C++ compile actions, but not for actions created by Skylark rules. So that script needs to be rewritten or generated in a way that would give it the complete path. I bypassed this by hardcoding the absolute path.

The next error I ran into was a missing <stddef.h>. It looks like --sysroot and most other include-related flags passed through CGO_CPPFLAGS (and other variables) use relative paths. go install (which we use to build the stdlib) may invoke the C compiler in different directories. It generally creates a temporary directory for each package it builds and invokes the C compiler there. Each directory passed to the C compiler should really be an absolute path. Is it possible to rewrite your CROSSTOOL so that all includes are absolute paths? If not, some wrapper might be needed to process the flags.

[AustinSchuh]

@jayconrod Wohoo! Progress.

The challenge here is that if we convert anything to absolute paths in the clang/gcc invocation, they get baked into all our binaries. If we have a generated shell script with an absolute path to the compiler, that won't work with remote-ex. Bazel propper works really hard to not expose absolute paths to the action. The only way to be reproducible is to do relative paths. The toolchain lives at a relative path (or a path findable with $(location ...)). Locally, we've added "BAZEL_OUTPUT_ROOT": "$(pwd)/", to the environment. Is there a way to get a similar mechanism upstreamed? Or do you have another suggestion on how to detect this?