ARM64 : Minio server crashes randomly

[kattakarthik]

Minio server is terminating randomly on ARM64 platform. Built the latest code on ARM64 platform and used for testing.The deployment type is "distributed mode".
In a week testing on 6 nodes, 3 servers crashed.
The load was moderate when these crashes happened.

Log messages ::

goroutine 338370336 [IO wait]:
runtime.gopark(0x1821768, 0xfffe25850a98, 0x4001841b02, 0x5)
/usr/local/go/src/runtime/proc.go:301 +0xf0 fp=0x40042fc9c0 sp=0x40042fc9a0 pc=0x43cf50
runtime.netpollblock(0xfffe25850a70, 0x72, 0x409266a000)
/usr/local/go/src/runtime/netpoll.go:389 +0xa4 fp=0x40042fca00 sp=0x40042fc9c0 pc=0x4385f4
internal/poll.runtime_pollWait(0xfffe25850a70, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:182 +0x48 fp=0x40042fca30 sp=0x40042fca00 pc=0x437b48
internal/poll.(*pollDesc).wait(0x408b030818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xa0 fp=0x40042fca60 sp=0x40042fca30 pc=0x493fc0
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0x408b030800, 0x409266a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x170 fp=0x40042fcac0 sp=0x40042fca60 pc=0x494e40
net.(*netFD).Read(0x408b030800, 0x409266a000, 0x1000, 0x1000, 0x40042fcb78, 0x413008, 0x40042fcb60)
/usr/local/go/src/net/fd_unix.go:202 +0x44 fp=0x40042fcb20 sp=0x40042fcac0 pc=0x598c14
net.(*conn).Read(0x4031f50158, 0x409266a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:177 +0x5c fp=0x40042fcb80 sp=0x40042fcb20 pc=0x5ab42c
net/http.(*persistConn).Read(0x408dc807e0, 0x409266a000, 0x1000, 0x1000, 0x40042fcc68, 0x6c2f30, 0x405fa8c840)
/usr/local/go/src/net/http/transport.go:1524 +0x60 fp=0x40042fcc00 sp=0x40042fcb80 pc=0x6c1da0
bufio.(*Reader).fill(0x4004105a40)
/usr/local/go/src/bufio/bufio.go:100 +0x100 fp=0x40042fcc50 sp=0x40042fcc00 pc=0x4b0000
bufio.(*Reader).Peek(0x4004105a40, 0x1, 0x0, 0x0, 0x1, 0x400d6c5600, 0x0)
/usr/local/go/src/bufio/bufio.go:138 +0x40 fp=0x40042fcc70 sp=0x40042fcc50 pc=0x4b0180
net/http.(*persistConn).readLoop(0x408dc807e0)
/usr/local/go/src/net/http/transport.go:1677 +0x16c fp=0x40042fcfd0 sp=0x40042fcc70 pc=0x6c282c
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1128 +0x4 fp=0x40042fcfd0 sp=0x40042fcfd0 pc=0x46aa74
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport

Expected Behavior

Minio server shouldn't terminate.

Current Behavior

Minio server terminates.

Your Environment

• Version used (minio version):
  Minio (built from source on ARM64 platform)
  Version: DEVELOPMENT.GOGET
  Release-Tag: DEVELOPMENT.GOGET
  Commit-ID: DEVELOPMENT.GOGET

• Operating System and version (uname -a):
  aarch64 aarch64 aarch64 GNU/Linux

[vadmeste]

@kattakarthik, the stack trace that you pasted is not the one which caused the crash, can you paste all the logs that you can find ?

[kattakarthik]

minio_log.txt

Here is the complete log.. Hope it helps to root cause the issue @kannappanr

[vadmeste]

Most likely, a C dependency is causing the issue in ARM64

[kattakarthik]

Do you have any suggestions for us to try ?

[vadmeste]

@kattakarthik did you enable compression or encryption in your Minio setup ?

[kattakarthik]

@vadmeste We just built the binary from source and ran it on arm64. Didn't change any default config.

[kattakarthik]

The binary was built on one machine and same is used across all machiens. The other machines don't have Go installed on them. Is it required to have Go installed on all machienes ??

In a week testing 3 out of 6 server instances terminated with this error..

[harshavardhana]

@kattakarthik can you build MinIO binary with CGO_ENABLED=0 ?

[kattakarthik]

GOPROXY=https://proxy.golang.org GO111MODULE=on
CGO_ENABLED=0
go get github.com/minio/minio

You mean add the above flag in this command ??

Will there be any impact to the already synced data if we restart minio after building the way you suggested ??

[harshavardhana]

You mean add the above flag in this command ??

yes it should be one line

GOPROXY=https://proxy.golang.org GO111MODULE=on CGO_ENABLED=0 go get github.com/minio/minio

[kattakarthik]

Got it.
Will there be any impact to the already synced data if we restart minio after building the way you suggested ??