go1.20: Suddenly linking static binaries

[zecke]

What version of rules_go are you using?

0.39.0

What version of gazelle are you using?

0.30.0

What version of Bazel are you using?

6.2.1

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Linux. Reproduces on x86_64 and aarch64

Any other potentially useful information about your toolchain?

It needs to have cgo selected

What did you do?

$ cat .bazelrc
build --features=fully_static_link --dynamic_mode=off

$ cat WORKSPACE
workspace(name = "rules_go_static")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_go",
    sha256 = "6b65cb7917b4d1709f9410ffe00ecf3e160edf674b78c54a894471320862184f",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.39.0/rules_go-v0.39.0.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.39.0/rules_go-v0.39.0.zip",
    ],
)

http_archive(
    name = "bazel_gazelle",
    sha256 = "727f3e4edd96ea20c29e8c2ca9e8d2af724d8c7778e7923a854b2c80952bc405",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.30.0/bazel-gazelle-v0.30.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.30.0/bazel-gazelle-v0.30.0.tar.gz",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")

go_rules_dependencies()

#go_register_toolchains(version = "1.19.3")
go_register_toolchains(version = "1.20.3")

gazelle_dependencies()

$ cat BUILD.bazel
load("@io_bazel_rules_go//go:def.bzl", "go_binary")

go_binary(
    name = "hello",
    srcs = ["hello.go"],
    cgo = True,
)

$ cat hello.go
package main

import (
	"fmt"
	"os/user"
)

func main() {
	u, _ := user.Current()
	print(u)
	fmt.Println("vim-go")
}


$ bazel build //:hello && file ./bazel-bin/hello_/hello

What did you expect to see?

For Go1.19.3 I see this:

./bazel-bin/hello_/hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, Go BuildID=redacted, with debug_info, not stripped

What did you see instead?

For Go1.20 I see this:

/tmp/go-link-3680801931/000001.o:cgo_lookup_cgo.cgo2.c:function mygetpwuid_r: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-3680801931/000001.o:cgo_lookup_cgo.cgo2.c:function mygetpwnam_r: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-3680801931/000001.o:cgo_lookup_cgo.cgo2.c:function mygetgrgid_r: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-3680801931/000001.o:cgo_lookup_cgo.cgo2.c:function mygetgrnam_r: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-3680801931/000002.o:getgrouplist_unix.cgo2.c:function mygetgrouplist: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
...
./bazel-bin/hello_/hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, Go BuildID=redacted, with debug_info, not stripped

With Go1.19 the binary was linked dynamically against glibc besides the features selection. In Go 1.20 this suddenly becomes statically linked. It is a surprising behavior change, especially as the go_binary rule has selection for pure/cgo/static.

I noticed this because a grpc protoc compiler plugin started to crash (when the log library invoked the app as the glibc version of the host and link didn't match...)

[zecke]

This seems to be somehow related to us building the Go stdlib on Go1.20 while we previously used the pre-compiled one. I have not fully understood what leads to this behavior.

When building I am printing the extldflags in go/private/actions/link.bzl. This appears to be the same for Go1.19 and Go1.20.

io_bazel_rules_go/go/private/actions/link.bzl:82:10: ["-fuse-ld=gold", "-Wl,-no-as-needed", "-Wl,-z,relro,-z,now", "-B/usr/local/bin", "-pass-exit-codes", "-lm", "-static"]

Filtering out -static (in addition to -lstdc++) makes no difference. However. Filtering out -static in go/private/actions/stdlib.bzl does make a difference. Apparently the ldflags make it into the final linking.

Any chance we can filter out -static in these two places to keep the go1.19 behavior?

[zecke]

Okay. So somehow when using cgo and compiling the stdlib the CGO_LDFLAGS will be stored as directive (JSON including an array of cgo_ldlfag) in the .a file. These directives will be used when linking the final binary.

See:
(https://github.com/golang/go/blob/master/src/cmd/link/internal/ld/go.go#L278)

[fmeum]

Is the -static coming from your toolchain or is rules_go adding this somewhere? I could see valid use cases for static linking, e.g. when using a custom cgo dependency but not the cgo parts of the stdlib.

[zecke]

Is the -static coming from your toolchain or is rules_go adding this somewhere? I could see valid use cases for static linking, e.g. when using a custom cgo dependency but not the cgo parts of the stdlib.

It comes from the C/C++ toolchain and is selected --features=fully_static_link in the .bazelrc. Up to Go 1.19 this didn't lead to go binaries being linked statically (and in our case doing this for the host utility is a bit of an issue). One can argue about this being a bug in rules_go but it is somewhat desirable as static linking with GNU libc is a tough one and go_binary has a static attribute.

[fmeum]

As long as the static attribute on go_binary build still allows for a static build, I would be fine with stripping -static from the external linker command line by default. Do you want to send a PR?

@linzhp as this may be relevant for you.

[zecke]

-static is passed explicitly here https://github.com/bazelbuild/rules_go/blob/master/go/private/actions/link.bzl#L122-L123.

I ended up tracking one more behavior change. With Go 1.20 the external linker is actually used when before we only passed -extld and -extldflags. It appears that the linkmode is set to auto and with the new version of Go and compiling the standard library we require an external linker:

The relevant linker output in Go1.20 is:

external linking is forced by: some packages could not be built to support internal linking ([os/user runtime/cgo])
external linking prefer list is [os/user runtime/cgo]

This change is a bit scary as I would not have expected to use the C/C++ toolchains linker for something like os/user (with cgo).

The builder is passing -fdebug-prefix-map when building the stdlib which leads to the cgo being marked with preferlinkext.