cmd/link: relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'

[saschagrunert]

What version of Go are you using (go version)?

$ go version
go version go1.20 linux/amd64

Does this issue reproduce with the latest release?

Yes, by cross compiling kubernetes/kubernetes@b7ad179 to the linux/arm platform.

What operating system and processor architecture are you using (go env)?

We build in a custom container image registry.k8s.io/build-image/kube-cross

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/go/src/k8s.io/kubernetes/.cache/go-build"
GOENV="/go/src/k8s.io/kubernetes/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp.k8s/go-build942910395=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Running make release in the root of the Kubernetes repository. Everything gets built successfully, except the k8s.io/kubernetes/test/e2e_node.test, which fails with:

/usr/local/go/pkg/tool/linux_amd64/link: running arm-linux-gnueabihf-gcc failed: exit status 1
/tmp.k8s/go-link-865684912/go.o: in function `k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/service/ec2.(*ModifyCapacityReservationInput).GoString':
go.go:(.text+0x207ca88): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
go.go:(.text+0x207ca94): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
collect2: error: ld returned 1 exit status

A small patch can be applied to make it fail faster for reproduction:

diff --git a/hack/make-rules/cross.sh b/hack/make-rules/cross.sh
index f8a6d0dbf5e..1a7a19ff307 100755
--- a/hack/make-rules/cross.sh
+++ b/hack/make-rules/cross.sh
@@ -27,12 +27,12 @@ source "${KUBE_ROOT}/hack/lib/init.sh"
 # NOTE: Using "${array[*]}" here is correct.  [@] becomes distinct words (in
 # bash parlance).
 
-make all WHAT="${KUBE_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_SERVER_PLATFORMS[*]}"
+#make all WHAT="${KUBE_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_SERVER_PLATFORMS[*]}"
 
-make all WHAT="${KUBE_NODE_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_NODE_PLATFORMS[*]}"
+#make all WHAT="${KUBE_NODE_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_NODE_PLATFORMS[*]}"
 
-make all WHAT="${KUBE_CLIENT_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_CLIENT_PLATFORMS[*]}"
+#make all WHAT="${KUBE_CLIENT_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_CLIENT_PLATFORMS[*]}"
 
-make all WHAT="${KUBE_TEST_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_PLATFORMS[*]}"
+#make all WHAT="${KUBE_TEST_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_PLATFORMS[*]}"
 
-make all WHAT="${KUBE_TEST_SERVER_TARGETS[*]}" KUBE_BUILD_PLATFORMS="${KUBE_TEST_SERVER_PLATFORMS[*]}"
+make all WHAT="test/e2e_node/e2e_node.test" KUBE_BUILD_PLATFORMS="linux/arm"

Setting CGO_CFLAGS=-mlong-calls does not help, disabling CGO does provide a workaround.

Source code changes seem to have impact on the issue, because the build works with the next commit kubernetes/kubernetes@b0171f7

Ref: kubernetes/kubernetes#115613, kubernetes/kubernetes#115605

What did you expect to see?

That the test binary compiles correctly with CGO enabled.

What did you see instead?

The link failure.

[saschagrunert]

Copying from kubernetes/kubernetes#115605 (comment)

Reproducer with a Dockerfile, on kubernetes/kubernetes@b7ad179:

FROM debian:latest

RUN apt-get update && \
    apt-get install -y \
        gcc-arm-linux-gnueabihf \
        wget

RUN wget https://go.dev/dl/go1.20.linux-amd64.tar.gz && \
    tar xf *.tar.gz && \
    mv go /usr/local

ENV PATH=$PATH:/usr/local/go/bin
ENV GOARCH=arm
ENV CGO_ENABLED=1
ENV CC=arm-linux-gnueabihf-gcc

COPY . /work
WORKDIR /work
RUN go test -o e2e_node.test k8s.io/kubernetes/test/e2e_node

• It breaks with go 1.20 but works with go 1.19.5 on debian:latest (bullseye-20230208, arm-linux-gnueabihf-gcc 10.2.1)
• It breaks with go 1.20 but works with go 1.19.5 on debian:testing (testing-20230208, arm-linux-gnueabihf-gcc 12.2.0)

[tpaschalis]

We're also having this issue when cross-compiling the Grafana Agent on ARMv6/ARMv7. I've posted some details on #58428

[dr2chase]

@golang/compiler

[thanm]

I'll take a look.

[liggitt]

I bisected to 833367e:

git bisect start
# good: [e99f53fed98b0378c147588789b8c56b0305469b] doc: move Go 1.19 release notes to x/website
git bisect good e99f53fed98b0378c147588789b8c56b0305469b
# bad: [de4748c47c67392a57f250714509f590f68ad395] [release-branch.go1.20] go1.20
git bisect bad de4748c47c67392a57f250714509f590f68ad395
# bad: [1e4989c33674bf9570c99250a081a448c3cded65] cmd: relocate search.MatchPattern to cmd/internal/pkgpattern
git bisect bad 1e4989c33674bf9570c99250a081a448c3cded65
# bad: [00bee6d9a4c3ed6168350fc6551043ff7a1895f2] cmd/compile/internal/typebits: relax alignment check
git bisect bad 00bee6d9a4c3ed6168350fc6551043ff7a1895f2
# bad: [72a76ca1f9c195ed39e929cf768d5df5421eada1] cmd/compile: restore test/nested.go test cases
git bisect bad 72a76ca1f9c195ed39e929cf768d5df5421eada1
# good: [9e4638ad20181ba4c3b1985da48c56be121901fb] net: discard unrooted 254 byte names, not rooted ones
git bisect good 9e4638ad20181ba4c3b1985da48c56be121901fb
# good: [90466e1ddf0e4305bc56f6eac61a690704e6fab8] crypto/internal/subtle: rename to crypto/internal/alias
git bisect good 90466e1ddf0e4305bc56f6eac61a690704e6fab8
# bad: [d654117075d1dd0686aa6833b22f28275623c759] cmd/compile: add intrinsic for MulUintptr on arm64
git bisect bad d654117075d1dd0686aa6833b22f28275623c759
# good: [c82bbc0e8edbbebe47e92729e8f3f1b60d380b5b] runtime: convert timer0When/timerModifiedEarliest to atomic.Int64
git bisect good c82bbc0e8edbbebe47e92729e8f3f1b60d380b5b
# good: [9485d4c1bd871be792d03c29b7902f6ac284ed27] cmd/compile/internal/syntax: handle missing index like in go/parser
git bisect good 9485d4c1bd871be792d03c29b7902f6ac284ed27
# bad: [55ecc3a88670c14bbbfc9d6c96cf65891cc608f5] net: allow single dot in resolv.conf search statement
git bisect bad 55ecc3a88670c14bbbfc9d6c96cf65891cc608f5
# bad: [833367e98af838a2511ee7e4e19dc8f1da7b8ed7] internal/buildcfg: enable unified IR by default
git bisect bad 833367e98af838a2511ee7e4e19dc8f1da7b8ed7
# good: [17211c355f015862b635f8cbd48ffc65a1081b92] go/types, types2: consolidate testdata/spec test files
git bisect good 17211c355f015862b635f8cbd48ffc65a1081b92
# good: [0a4a57de4d0764a4291e4d8654e88b44e65d8059] cmd/compile/internal/types2: use go/types/testdata/spec tests
git bisect good 0a4a57de4d0764a4291e4d8654e88b44e65d8059
# first bad commit: [833367e98af838a2511ee7e4e19dc8f1da7b8ed7] internal/buildcfg: enable unified IR by default

Notes:

1. I had to cherry-pick 00bee6d onto each candidate commit prior to that fix before building to get around the alignment bug to make it to the failed linking step
2. linux/arm failed, linux/arm64 seemed to work

Reproducer:

# checkout of https://github.com/golang/go
pushd go
  git checkout 833367e98af838a2511ee7e4e19dc8f1da7b8ed7
  git cherry-pick 00bee6d9a4c3ed6168350fc6551043ff7a1895f2
  pushd src
    ./make.bash
  popd
popd

export GOROOT=$PWD/go
export GOPATH=$PWD/emptydir
export PATH=$PWD/go/bin:$PATH

# checkout of https://github.com/kubernetes/kubernetes/
pushd kubernetes
  git checkout b7ad17978eaba508c560f81bab2fb9d0b256e469
  GOARCH=arm CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc go test -c k8s.io/kubernetes/test/e2e_node
popd

Output is:

# k8s.io/kubernetes/test/e2e_node.test
/usr/local/google/home/liggitt/tmp/arm/go/pkg/tool/linux_amd64/link: running arm-linux-gnueabihf-gcc failed: exit status 1
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*DescribeScheduledInstancesInput).GoString':
./<autogenerated>:1:(.text+0x207e440): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*DescribeScheduledInstancesInput).GoString':
/usr/local/google/home/liggitt/tmp/arm/kubernetes/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:94642:(.text+0x207e44c): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*DescribeScheduledInstancesInput).String':
./<autogenerated>:1:(.text+0x207e4f0): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*ScheduledInstance).GoString':
./<autogenerated>:1:(.text+0x207e598): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
/tmp/go-link-3447838833/go.o: in function `github.com/aws/aws-sdk-go/service/ec2.(*ScheduledInstance).GoString':
/usr/local/google/home/liggitt/tmp/arm/kubernetes/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:153021:(.text+0x207e5a4): relocation truncated to fit: R_ARM_CALL against `runtime.duffcopy'
collect2: error: ld returned 1 exit status

That commit defaulted GOEXPERIMENT=Unified on. Starting to run the bisection again with that experiment on in the invocation

[liggitt]

Building go1.19.5 with GOEXPERIMENT=Unified reproduces the relocation truncated to fit: R_ARM_CALL against 'runtime.duffcopy' issue, so it looks like the unified build regresses on arm

[liggitt]

cc @mdempsky