crypto/*: Go programs fails to start after upgrading go toolchain to 1.21.6

[sAnjAy060897]

Go version

go version go1.21.6 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/go/.cache'
GOENV='/home/skumar12/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/skumar12/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/skumar12/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/recipe-sysroot-native/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTMPDIR='/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/build-tmp'
GOTOOLCHAIN=''
GOTOOLDIR='/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/recipe-sysroot-native/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.6'
GCCGO='gccgo'
GOAMD64='v1'
AR='x86_64-ion-linux-gcc-ar'
CC='x86_64-ion-linux-gcc  -m64 -march=nehalem -mtune=generic -mfpmath=sse -msse4.2 --sysroot=/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/recipe-sysroot'
CXX='x86_64-ion-linux-g++  -m64 -march=nehalem -mtune=generic -mfpmath=sse -msse4.2 --sysroot=/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/recipe-sysroot'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/recipe-sysroot= -fdebug-prefix-map=/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/recipe-sysroot-native='
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/recipe-sysroot= -fdebug-prefix-map=/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/recipe-sysroot-native= -fvisibility-inlines-hidden'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -no-pie'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-mtune=generic -mfpmath=sse -msse4.2 --sysroot=/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/recipe-sysroot -I . -fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/ws2/skumar12/element_code/ap/build/tmp/work/corei7-64-ion-linux/go/1.21.6-r0/build-tmp/go-build3064235111=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Upgraded go toolchain in our product from go 1.18.9 to go1.21.6 and ran the image in the target. We use yocto build system to compile our product modules. go1.21.6 is built from source.

What did you see happen?

All go programs failed to start with index out of range in one of the sha1, md5 and flag packages.

panic: runtime error: index out of range [3] with length 0

goroutine 1 [running]:
crypto.RegisterHash(...)
/usr/lib/go/src/crypto/crypto.go:149
crypto/sha1.init.0()
/usr/lib/go/src/crypto/sha1/sha1.go:19 +0x65

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
flag.init()
/usr/lib/go/src/flag/flag.go:1199 +0xf9

What did you expect to see?

Go programs to start normally.

[randall77]

Strange. This looks like somehow your program is not being initialized correctly (or at all).
I suspect a corrupt installation. Maybe you installed 1.21.6 overtop 1.18.9? Nevertheless, try blowing away your 1.21.6 and reinstalling fresh.

[sAnjAy060897]

Thanks for your response @randall77 .

We tried 1.20.13 and that works fine and all go programs in our product came up fine. When we tried 1.21.0, it failed with the above mentioned error.

Is there a way to verify the installation after the toolchain is upgraded? The go env seems ok. Any other way to check?

[randall77]

Build a hello world that prints os.Args[0] and see if that works. That's one of the expressions your program crashed on that should never fail.

[sAnjAy060897]

@randall77 Thanks for your suggestion. Tried the same and the programs seems to run fine on the target.

./hello-world 
Hello World printing args[0]
1 [./hello-world]

We see the programs (our most programs do) importing sha1, md5 and flag packages are exiting with the above mentioned index out of range error.

[randall77]

Try adding import _ "flag" to your hello world program?

Frankly, I'm at a loss for what this could be. Clearly something is getting screwed up very early in initialization.
Can you share a program on which this occurs?

Are you doing anything nonstandard, like using a Go program called from a C library?
A bunch of your environment variables look odd to me, maybe try unsetting them to see if that helps? GOROOT, CGO_CFLAGS, CGO_LDFLAGS, etc.
Might also try go clean -cache.

[sAnjAy060897]

After importing flag package, program did not start with the same error.

No. We are not using go program from C library.
Unsetting env variables and cleaning cache did not help either.

[randall77]

So this program

package main

import (
	_ "flag"
	"fmt"
)

func main() {
	fmt.Println("hello world")
}

Builds ok but crashes when you try to run it?

[sAnjAy060897]

That's correct.

And FYI - we used the yocto patches available at https://patchwork.yoctoproject.org/project/oe-core/patch/[email protected]/ to build go 1.21 toolchain

[randall77]

Try without those patches.

[sAnjAy060897]

We were not able to get rid of the patches (it would take a lot of effort to redefine the build instruction for go programs) as the go programs in our product depend on yocto build system to compile and go toolchain is built differently in yocto (https://blog.gopheracademy.com/advent-2015/go-in-a-yocto-project/).

We are able to build the go 1.21.6 toolchain in yocto build system and the programs are running fine on the target after using the patches provided by yocto contributions here https://patchwork.yoctoproject.org/project/oe-core/list/?series=22831.

I guess yocto will soon align with upstream go release (1.22.x) soon and these run time panic may not be seen with the official recipes and patches.

[randall77]

I'm going to close this as I don't think there's anything here for the Go project to do.