armv5tel "Illegal instruction" since golang 1.21.0

[cah-jad-baz]

What version of Go are you using (go version)?

Compiling with:

$ go version
go1.21.0 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64 build environment:

go env Output

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/jad/.cache/go-build'
GOENV='/home/jad/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/jad/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/jad/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3952936469=/tmp/go-build -gno-record-gcc-switches'

armv5 target environment

~# uname -a
Linux SDcard 2.6.34 #17 PREEMPT Fri Nov 11 15:06:02 MST 2011 armv5tel GNU/Linux

Issue

armv5tel is no longer supported as a runtime target (not sure if all armv5). Since the golang toolchain is not available on armv5, I'm building on amd64 and copying to armv5. I tried a simple hello world. It works with 1.20.7 but fails with 1.21.0

Full reproducer

On build machine (amd64)

vi helloworld.go

package main
import "fmt"

func main() {
    fmt.Println("Hello, World!")
}

Then, compile it with golang 1.20.7 and 1.21 for linux and arm

env GOOS=linux GOARCH=amd64 /usr/local/go1.20.7/bin/go build helloworld.go; mv helloworld helloworld.arm.go1.20.7
env GOOS=linux GOARCH=amd64 /usr/local/go1.21.0/bin/go build helloworld.go; mv helloworld helloworld.arm.go1.21.0

Then copy both to armv5 machine

scp helloworld.arm.go1.20.7 user@IP:
scp helloworld.arm.go1.21.0 user@IP:

On target machine (armv5)

Run both

# ./helloworld.arm.go1.20.7
Hello, World!

# ./helloworld.arm.go1.21.0
Illegal instruction 

The one built with golang 1.20.7 succeeds whereas the one built with 1.21.0 fails

[randall77]

Can you tell us what the illegal instruction is? Run the code under a debugger and disassemble around the faulting PC.

[cah-jad-baz]

@randall77, I don't think it's possible for me to run a debugger on armv5. As mentioned above, the toolchain is not available for armv5 so I need to build it on amd64 and then copy it over to armv5.

That being said, this is not an issue with the program I wrote as the program is tiny and it's also happening in VictoriaMetrics/VictoriaMetrics#4965

In any case, what other info can I provide that might aid in debugging this?

[randall77]

I built hello world with both 1.20 and 1.21. Disassembled both and looked for an instruction in the 1.21 binary that was not in the 1.20 binary. The only one I found is mrc, which should be fine for armv5.

That said, did you set GOARM=5 when building? 6 is the default. https://github.com/golang/go/wiki/MinimumRequirements#arm

Not sure what would have changed from 1.20 to 1.21.

[cah-jad-baz]

env GOOS=linux GOARCH=arm GOARM=5 /usr/local/go1.21.0/bin/go build helloworld.go; mv helloworld helloworld.arm.go1.21.0.armv5

Now it works

@randall77 do you know if prior to 1.21, default GOARM was also 6?

[randall77]

My memory is hazy on that front.
Maybe the default for hosted builds was 6, but the default for cross-compiled builds was 5? That latter part might be what changed due to our push to make builds reproducible independent of host.

@golang/arm

[cah-jad-baz]

Alright, that's great. Closing this issue. Thanks @randall77!

[randall77]

This did indeed change for 1.21:

% GOOS=linux GOARCH=arm ~/go1.20.6/bin/go env GOARM
5
% GOOS=linux GOARCH=arm ~/go1.21.0/bin/go env GOARM
7

The issue for reproducible builds is #58884 .