fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

[dFionov]

What version of Go are you using (go version)?

1.14

Does this issue reproduce with the latest release?

Yes, and only with it

What operating system and processor architecture are you using (go env)?

Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

What did you do?

Rebuilded application using go 1.14 instead of using 1.13

What did you expect to see?

Working app

What did you see instead?

runtime: pointer 0xc0015e1d40 to unused region of span span.base()=0xc0015de000 span.limit=0xc0015e1c80 span.state=1
runtime: found in object at *(0xc001dccbb0+0x28)
object=0xc001dccbb0 s.base()=0xc001dcc000 s.limit=0xc001dcdfa0 s.spanclass=24 s.elemsize=176 s.state=mSpanInUse
*(object+0) = 0x1010000000060
*(object+8) = 0xc0015e1778
*(object+16) = 0x924079
*(object+24) = 0x0
*(object+32) = 0x0
*(object+40) = 0xc0015e1d40 <==
*(object+48) = 0x1504a60
*(object+56) = 0xc0015e1948
*(object+64) = 0x923bb7
*(object+72) = 0x0
*(object+80) = 0x0
*(object+88) = 0x0
*(object+96) = 0x0
*(object+104) = 0x0
*(object+112) = 0x0
*(object+120) = 0x0
*(object+128) = 0x0
*(object+136) = 0x0
*(object+144) = 0x0
*(object+152) = 0x0
*(object+160) = 0x0
*(object+168) = 0x0
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x141ed7f, 0x3e)
/usr/local/go/src/runtime/panic.go:1112 +0x72 fp=0x7fb7095c0870 sp=0x7fb7095c0840 pc=0x43c0b2
runtime.badPointer(0x7fb7092abb30, 0xc0015e1d40, 0xc001dccbb0, 0x28)
/usr/local/go/src/runtime/mbitmap.go:380 +0x230 fp=0x7fb7095c08b8 sp=0x7fb7095c0870 pc=0x41c390
runtime.findObject(0xc0015e1d40, 0xc001dccbb0, 0x28, 0x7fb7303a48f0, 0xc000058698, 0x2)
/usr/local/go/src/runtime/mbitmap.go:416 +0x9b fp=0x7fb7095c08f0 sp=0x7fb7095c08b8 pc=0x41c43b
runtime.scanobject(0xc001dccbb0, 0xc000058698)
/usr/local/go/src/runtime/mgcmark.go:1274 +0x235 fp=0x7fb7095c0980 sp=0x7fb7095c08f0 pc=0x4274d5
runtime.gcDrain(0xc000058698, 0x7)
/usr/local/go/src/runtime/mgcmark.go:1032 +0x1fb fp=0x7fb7095c09d8 sp=0x7fb7095c0980 pc=0x426cdb
runtime.gcBgMarkWorker.func2()
/usr/local/go/src/runtime/mgc.go:1955 +0x169 fp=0x7fb7095c0a18 sp=0x7fb7095c09d8 pc=0x4692f9
runtime.systemstack(0x40)
/usr/local/go/src/runtime/asm_amd64.s:370 +0x66 fp=0x7fb7095c0a20 sp=0x7fb7095c0a18 pc=0x46c0c6
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1041 fp=0x7fb7095c0a28 sp=0x7fb7095c0a20 pc=0x440e90

[networkimprov]

Can you point to what application you built? Did it fail during build or run/test?

Please run go version and go env and paste the outputs into the issue text between ``` lines.

[dFionov]

We are building backend service written in go.
I can't run go version because it's builded in CI. Our CI was configured to use 1.14 and our service started to crash every few minutes under low load and even every few seconds under high. We spend a day to figure out what happened, but finally found that root cause was, probably, in raising go version. Switching back to 1.13 helped - no crashes anymore.

build with crashes: GoVersion:go1.14 Os:linux Arch:amd64
build without them: GoVersion:go1.13.8 Os:linux Arch:amd64
I'm sorry that I can't add more details.

PS. We reviewed carefully all changes between this two builds(actually it was ~100 loc) no unsafe or cgo usages were added

[ALTree]

PS. We reviewed carefully all changes between this two builds(actually it was ~100 loc) no unsafe or cgo usages were added

Does the application in general use unsafe or cgo?

[dFionov]

@ALTree Not by itself, but we use thirdparties that use cgo. govendor list:
v github.com/Microsoft/go-winio
v github.com/StackExchange/wmi
v github.com/beorn7/perks/quantile
v github.com/codahale/hdrhistogram
v github.com/davecgh/go-spew/spew
v github.com/denisenkom/go-mssqldb
v github.com/denisenkom/go-mssqldb/internal/cp
v github.com/dgrijalva/jwt-go
v github.com/go-ole/go-ole
v github.com/go-ole/go-ole/oleutil
v github.com/go-sql-driver/mysql
v github.com/go-stack/stack
vu github.com/go-swagger/go-swagger
v github.com/gobuffalo/envy
v github.com/gobuffalo/here
v github.com/gobuffalo/packd
v github.com/gobuffalo/packd/internal/takeon/github.com/markbates/errx
v github.com/gobuffalo/packr
v github.com/gofrs/uuid
v github.com/golang/protobuf/proto
vu github.com/golang/snappy
v github.com/gomodule/redigo/redis
vu github.com/gorilla/context
v github.com/gorilla/mux
v github.com/gorilla/websocket
v github.com/hashicorp/consul/api
v github.com/hashicorp/go-cleanhttp
v github.com/hashicorp/go-rootcerts
vu github.com/hashicorp/go-version
v github.com/hashicorp/serf/coordinate
v github.com/jackc/pgx
v github.com/jackc/pgx/chunkreader
v github.com/jackc/pgx/internal/sanitize
v github.com/jackc/pgx/pgio
v github.com/jackc/pgx/pgproto3
v github.com/jackc/pgx/pgtype
v github.com/jackc/pgx/stdlib
vu github.com/jessevdk/go-flags
v github.com/jinzhu/gorm
v github.com/jinzhu/gorm/dialects/mssql
v github.com/jinzhu/inflection
v github.com/jmoiron/sqlx
v github.com/jmoiron/sqlx/reflectx
v github.com/joho/godotenv
vu github.com/lib/pq/oid
v github.com/lxn/win
v github.com/martingallagher/go-jsonmp
v github.com/mattn/go-sqlite3
v github.com/matttproud/golang_protobuf_extensions/pbutil
v github.com/mcuadros/go-version
v github.com/mendsley/gojwk
v github.com/mitchellh/go-homedir
v github.com/mna/redisc
vu github.com/mohae/deepcopy
vu github.com/nsqio/go-nsq
v github.com/opentracing/opentracing-go
v github.com/opentracing/opentracing-go/ext
v github.com/opentracing/opentracing-go/log
v github.com/pkg/errors
v github.com/pmezard/go-difflib/difflib
v github.com/prometheus/client_golang/prometheus
v github.com/prometheus/client_golang/prometheus/internal
v github.com/prometheus/client_golang/prometheus/promauto
v github.com/prometheus/client_golang/prometheus/promhttp
v github.com/prometheus/client_model/go
v github.com/prometheus/common/expfmt
v github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg
v github.com/prometheus/common/model
v github.com/prometheus/procfs
v github.com/prometheus/procfs/internal/fs
v github.com/prometheus/procfs/internal/util
vu github.com/prometheus/procfs/nfs
vu github.com/prometheus/procfs/xfs
v github.com/rubenv/sql-migrate
v github.com/rubenv/sql-migrate/sqlparse
v github.com/shirou/gopsutil/cpu
v github.com/shirou/gopsutil/host
v github.com/shirou/gopsutil/internal/common
v github.com/shirou/gopsutil/mem
v github.com/shirou/gopsutil/net
v github.com/shirou/gopsutil/process
v github.com/shirou/w32
v github.com/spacemonkeygo/spacelog
v github.com/ssgreg/jwtopenssl
v github.com/ssgreg/openssl
v github.com/ssgreg/openssl/utils
v github.com/streadway/amqp
v github.com/stretchr/objx
v github.com/stretchr/testify/assert
v github.com/stretchr/testify/mock
v github.com/stretchr/testify/require
vu github.com/stretchr/testify/suite
v github.com/uber/jaeger-client-go
v github.com/uber/jaeger-client-go/config
v github.com/uber/jaeger-client-go/internal/baggage
v github.com/uber/jaeger-client-go/internal/baggage/remote
v github.com/uber/jaeger-client-go/internal/spanlog
v github.com/uber/jaeger-client-go/internal/throttler
v github.com/uber/jaeger-client-go/internal/throttler/remote
v github.com/uber/jaeger-client-go/log
v github.com/uber/jaeger-client-go/rpcmetrics
v github.com/uber/jaeger-client-go/thrift
v github.com/uber/jaeger-client-go/thrift-gen/agent
v github.com/uber/jaeger-client-go/thrift-gen/baggage
v github.com/uber/jaeger-client-go/thrift-gen/jaeger
v github.com/uber/jaeger-client-go/thrift-gen/sampling
v github.com/uber/jaeger-client-go/thrift-gen/zipkincore
v github.com/uber/jaeger-client-go/utils
v github.com/uber/jaeger-lib/metrics
v github.com/vmihailenco/msgpack
v github.com/vmihailenco/msgpack/codes
v github.com/vmihailenco/tagparser
v github.com/vmihailenco/tagparser/internal
v github.com/vmihailenco/tagparser/internal/parser
v github.com/yhirose/go-peg
v go.uber.org/atomic
v go.uber.org/multierr
v go.uber.org/zap
v go.uber.org/zap/buffer
v go.uber.org/zap/internal/bufferpool
v go.uber.org/zap/internal/color
v go.uber.org/zap/internal/exit
v go.uber.org/zap/zapcore
v golang.org/x/crypto/md4
v golang.org/x/crypto/pbkdf2
v golang.org/x/net/context
v golang.org/x/sys/windows
v golang.org/x/sys/windows/registry
v golang.org/x/text/cases
v golang.org/x/text/internal
v golang.org/x/text/internal/language
v golang.org/x/text/internal/language/compact
v golang.org/x/text/internal/tag
v golang.org/x/text/language
v golang.org/x/text/runes
v golang.org/x/text/secure/bidirule
v golang.org/x/text/secure/precis
v golang.org/x/text/transform
v golang.org/x/text/unicode/bidi
v golang.org/x/text/unicode/norm
v golang.org/x/text/width
v gopkg.in/gorp.v1
v gopkg.in/natefinch/lumberjack.v2
vu gopkg.in/square/go-jose.v2
vu gopkg.in/square/go-jose.v2/cipher
v gopkg.in/square/go-jose.v2/json
v gopkg.in/testfixtures.v2
v gopkg.in/yaml.v2

[ALTree]

@dFionov thanks. This will be hard to investigate without a (smallish) reproducer. The most likely explanation for the crash is incorrect unsafe or cgo usage in one of your dependencies, or a data race in your application.

I assume that running it with the race detector enabled (go [run/test/build] -race ...) does not print any warning?

[networkimprov]

How is the page size of this builder?

@mengzhuo could you clarify what do you mean? Are you talking about memory page size on build host? Do you need to compare it with page size on runtime?

@dFionov Yes, I mean system memory page size.
Go runtime has set the runtime page size to 8KB.
It's just my guess this issue is related to page size.

I think that Q belongs in this thread...

[dFionov]

Sorry for delay
@ALTree race detector didn't print any warning
@mengzhuo All hosts for build and run has 4096 bytes page sizes

[dmitshur]

This looks the same as #37688. I'll close this issue in favor of that one, so discussion can happen in one place.

@dFionov, if you believe there is something different about this issue, please let us know and we can re-open and track it separately from #37688. Otherwise, feel free to use #37688 for further discussion.