runtime: weird panic while dumping non-pointer struct with wrapped error

[powerman]

What version of Go are you using (go version)?

$ go version
go version go1.14.2 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/powerman/.cache/go-build"
GOENV="/home/powerman/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/powerman/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="x86_64-pc-linux-gnu-gcc"
CXX="x86_64-pc-linux-gnu-g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build368985502=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I was running go test on my package, which in turn called https://godoc.org/github.com/powerman/check#C.Nil at value returned by https://godoc.org/github.com/powerman/rpc-codec/jsonrpc2#WrapError.

The value itself was implemented this way https://github.com/powerman/rpc-codec/blob/v1.2.0/jsonrpc2/errors.go#L96-L116 and contains valid non-nil wrapped error inside.

Line https://github.com/powerman/rpc-codec/blob/v1.2.0/jsonrpc2/errors.go#L111 generates different panics at different go test runs, like these:

• first example:

unexpected fault address 0xffffffffffff80a7
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xffffffffffff80a7 pc=0x7d0474]

• second example:

fatal error: bulkBarrierPreWrite: unaligned arguments

Complete panic one

=== RUN   TestGetStr
unexpected fault address 0xffffffffffff80a7
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xffffffffffff80a7 pc=0x7d0474]
goroutine 37 [running]:

runtime.throw(0x9b6e95, 0x5)

/usr/lib/go/src/runtime/panic.go:1116 +0x72 fp=0xc000135580 sp=0xc000135550 pc=0x436762

runtime.sigpanic()

/usr/lib/go/src/runtime/signal_unix.go:702 +0x3cc fp=0xc0001355b0 sp=0xc000135580 pc=0x44ccdc

github.com/powerman/rpc-codec/jsonrpc2.wrapError.Error(...)

/home/powerman/go/pkg/mod/github.com/powerman/rpc-codec@v1.2.0/jsonrpc2/errors.go:111

github.com/powerman/rpc-codec/jsonrpc2.(*wrapError).Error(0xc000316150, 0x925800, 0xc000316150)

:1 +0x44 fp=0xc000135628 sp=0xc0001355b0 pc=0x7d0474

github.com/davecgh/go-spew/spew.handleMethods(0xe0da80, 0xa78160, 0xc0003161b0, 0x94ff20, 0xc000316150, 0x19, 0x0)

/home/powerman/go/pkg/mod/github.com/davecgh/go-spew@v1.1.1/spew/common.go:125 +0x316 fp=0xc000135750 sp=0xc000135628 pc=0x88c646

github.com/davecgh/go-spew/spew.(*dumpState).dump(0xc000135960, 0x94ff20, 0xc000316150, 0x19)

/home/powerman/go/pkg/mod/github.com/davecgh/go-spew@v1.1.1/spew/dump.go:306 +0x1399 fp=0xc0001358f8 sp=0xc000135750 pc=0x8917f9

github.com/davecgh/go-spew/spew.fdump(0xe0da80, 0xa78160, 0xc0003161b0, 0xc000135b48, 0x1, 0x1)

/home/powerman/go/pkg/mod/github.com/davecgh/go-spew@v1.1.1/spew/dump.go:465 +0x159 fp=0xc0001359a0 sp=0xc0001358f8 pc=0x891e79

github.com/davecgh/go-spew/spew.(*ConfigState).Sdump(0xe0da80, 0xc000135b48, 0x1, 0x1, 0xc000135aa8, 0x89ae79)

/home/powerman/go/pkg/mod/github.com/davecgh/go-spew@v1.1.1/spew/config.go:281 +0x78 fp=0xc0001359e8 sp=0xc0001359a0 pc=0x88ed98

github.com/powerman/check.newDump(0x94ff20, 0xc000316150, 0x1, 0xc0003201e0, 0x0, 0xdba748)

/home/powerman/go/pkg/mod/github.com/powerman/check@v1.2.1/dump.go:70 +0x8c fp=0xc000135c08 sp=0xc0001359e8 pc=0x8a602c

github.com/powerman/check.(*C).report(0xc00019c3f0, 0xdba700, 0x0, 0x0, 0x0, 0xdba767, 0x3, 0xc000135dd8, 0x1, 0x1, ...)

/home/powerman/go/pkg/mod/github.com/powerman/check@v1.2.1/check.go:136 +0x12a fp=0xc000135d58 sp=0xc000135c08 pc=0x89af5a

github.com/powerman/check.(*C).report1(0xc00019c3f0, 0x94ff20, 0xc000316150, 0x0, 0x0, 0x0, 0xc000328100, 0x2d)

/home/powerman/go/pkg/mod/github.com/powerman/check@v1.2.1/check.go:204 +0x12c fp=0xc000135df8 sp=0xc000135d58 pc=0x89bfbc

github.com/powerman/check.(*C).Nil(0xc00019c3f0, 0x94ff20, 0xc000316150, 0x0, 0x0, 0x0, 0xc000316150)

/home/powerman/go/pkg/mod/github.com/powerman/check@v1.2.1/check.go:346 +0x97 fp=0xc000135e48 sp=0xc000135df8 pc=0x89cba7

github.com/dentalcpdpro/mono/ms/id/internal/svcid.TestGetStr(0xc0001c4240)

/home/powerman/go/src/github.com/dentalcpdpro/mono/ms/id/internal/svcid/methods_test.go:27 +0x33a fp=0xc000135f80 sp=0xc000135e48 pc=0x8ab57a

testing.tRunner(0xc0001c4240, 0x9e0348)

/usr/lib/go/src/testing/testing.go:991 +0xdc fp=0xc000135fd0 sp=0xc000135f80 pc=0x503b9c

runtime.goexit()

/usr/lib/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000135fd8 sp=0xc000135fd0 pc=0x468631

created by testing.(*T).Run

/usr/lib/go/src/testing/testing.go:1042 +0x357

Complete panic two

=== RUN   TestGetStr
fatal error: bulkBarrierPreWrite: unaligned arguments
goroutine 54 [running]:

runtime.throw(0x9ce74f, 0x28)

/usr/lib/go/src/runtime/panic.go:1116 +0x72 fp=0xc000134e28 sp=0xc000134df8 pc=0x436762

runtime.bulkBarrierPreWrite(0xc00009fec0, 0xffffffffffff80a7, 0x28)

/usr/lib/go/src/runtime/mbitmap.go:610 +0x493 fp=0xc000134eb0 sp=0xc000134e28 pc=0x416e53

runtime.typedmemmove(0x95a5a0, 0xc00009fec0, 0xffffffffffff80a7)

/usr/lib/go/src/runtime/mbarrier.go:161 +0xa1 fp=0xc000134ee8 sp=0xc000134eb0 pc=0x415a71

reflect.typedmemmove(0x95a5a0, 0xc00009fec0, 0xffffffffffff80a7)

/usr/lib/go/src/runtime/mbarrier.go:186 +0x3f fp=0xc000134f10 sp=0xc000134ee8 pc=0x415acf

reflect.packEface(0x95a5a0, 0xffffffffffff80a7, 0x199, 0xc0, 0x2a)

/usr/lib/go/src/reflect/value.go:119 +0x9f fp=0xc000134f50 sp=0xc000134f10 pc=0x492a2f

reflect.valueInterface(0x95a5a0, 0xffffffffffff80a7, 0x199, 0x1, 0x76, 0x10)

/usr/lib/go/src/reflect/value.go:1023 +0xe1 fp=0xc000134fa0 sp=0xc000134f50 pc=0x4965e1

reflect.Value.Interface(...)

/usr/lib/go/src/reflect/value.go:993

fmt.(*pp).printValue(0xc0000c6ea0, 0x95a5a0, 0xffffffffffff80a7, 0x199, 0xffffffff00000076, 0x1)

/usr/lib/go/src/fmt/print.go:726 +0x25ad fp=0xc000135178 sp=0xc000134fa0 pc=0x4dc7ad

fmt.(*pp).printValue(0xc0000c6ea0, 0x916580, 0xffffffffffff80a7, 0x16, 0x76, 0x0)

/usr/lib/go/src/fmt/print.go:880 +0x18be fp=0xc000135350 sp=0xc000135178 pc=0x4dbabe

fmt.(*pp).printArg(0xc0000c6ea0, 0x916580, 0xffffffffffff80a7, 0x10100c000000076)

/usr/lib/go/src/fmt/print.go:716 +0x292 fp=0xc0001353e8 sp=0xc000135350 pc=0x4d9ad2

fmt.(*pp).doPrintf(0xc0000c6ea0, 0x9c1808, 0x13, 0xc0001355b0, 0x2, 0x2)

/usr/lib/go/src/fmt/print.go:1030 +0x15a fp=0xc0001354d0 sp=0xc0001353e8 pc=0x4dd2ca

fmt.Fprintf(0xa79bc0, 0xc000010018, 0x9c1808, 0x13, 0xc0001355b0, 0x2, 0x2, 0x0, 0x8af5e0, 0x92b280)

/usr/lib/go/src/fmt/print.go:204 +0x72 fp=0xc000135538 sp=0xc0001354d0 pc=0x4d6212

fmt.Printf(...)

/usr/lib/go/src/fmt/print.go:213

github.com/powerman/rpc-codec/jsonrpc2.wrapError.Error(0xffffffffffff80a7, 0x950f01, 0x7ffa3e1aa380)

/home/powerman/proj/go/rpc-codec/jsonrpc2/errors.go:111 +0xbf fp=0xc000135600 sp=0xc000135538 pc=0x7cc13f

github.com/powerman/rpc-codec/jsonrpc2.(*wrapError).Error(0xc00009fe00, 0x926800, 0xc00009fe00)

:1 +0x3c fp=0xc000135628 sp=0xc000135600 pc=0x7d05fc

github.com/davecgh/go-spew/spew.handleMethods(0xe0ea80, 0xa791a0, 0xc00009fe60, 0x950f20, 0xc00009fe00, 0x19, 0x0)

/home/powerman/go/pkg/mod/github.com/davecgh/go-spew@v1.1.1/spew/common.go:125 +0x316 fp=0xc000135750 sp=0xc000135628 pc=0x88c726

github.com/davecgh/go-spew/spew.(*dumpState).dump(0xc000135960, 0x950f20, 0xc00009fe00, 0x19)

/home/powerman/go/pkg/mod/github.com/davecgh/go-spew@v1.1.1/spew/dump.go:306 +0x1399 fp=0xc0001358f8 sp=0xc000135750 pc=0x8918d9

github.com/davecgh/go-spew/spew.fdump(0xe0ea80, 0xa791a0, 0xc00009fe60, 0xc000135b48, 0x1, 0x1)

/home/powerman/go/pkg/mod/github.com/davecgh/go-spew@v1.1.1/spew/dump.go:465 +0x159 fp=0xc0001359a0 sp=0xc0001358f8 pc=0x891f59

github.com/davecgh/go-spew/spew.(*ConfigState).Sdump(0xe0ea80, 0xc000135b48, 0x1, 0x1, 0xc000135aa8, 0x89af59)

/home/powerman/go/pkg/mod/github.com/davecgh/go-spew@v1.1.1/spew/config.go:281 +0x78 fp=0xc0001359e8 sp=0xc0001359a0 pc=0x88ee78

github.com/powerman/check.newDump(0x950f20, 0xc00009fe00, 0x1, 0xc0000a8f20, 0x0, 0xdbb810)

/home/powerman/go/pkg/mod/github.com/powerman/check@v1.2.1/dump.go:70 +0x8c fp=0xc000135c08 sp=0xc0001359e8 pc=0x8a610c

github.com/powerman/check.(*C).report(0xc0000a2530, 0xdbb800, 0x0, 0x0, 0x0, 0xdbb82f, 0x3, 0xc000135dd8, 0x1, 0x1, ...)

/home/powerman/go/pkg/mod/github.com/powerman/check@v1.2.1/check.go:136 +0x12a fp=0xc000135d58 sp=0xc000135c08 pc=0x89b03a

github.com/powerman/check.(*C).report1(0xc0000a2530, 0x950f20, 0xc00009fe00, 0x0, 0x0, 0x0, 0xc0000b8300, 0x2f)

/home/powerman/go/pkg/mod/github.com/powerman/check@v1.2.1/check.go:204 +0x12c fp=0xc000135df8 sp=0xc000135d58 pc=0x89c09c

github.com/powerman/check.(*C).Nil(0xc0000a2530, 0x950f20, 0xc00009fe00, 0x0, 0x0, 0x0, 0xc00009fe00)

/home/powerman/go/pkg/mod/github.com/powerman/check@v1.2.1/check.go:346 +0x97 fp=0xc000135e48 sp=0xc000135df8 pc=0x89cc87

github.com/dentalcpdpro/mono/ms/id/internal/svcid.TestGetStr(0xc0000e4360)

/home/powerman/go/src/github.com/dentalcpdpro/mono/ms/id/internal/svcid/methods_test.go:27 +0x35b fp=0xc000135f80 sp=0xc000135e48 pc=0x8ab67b

testing.tRunner(0xc0000e4360, 0x9e1380)

/usr/lib/go/src/testing/testing.go:991 +0xdc fp=0xc000135fd0 sp=0xc000135f80 pc=0x503b9c

runtime.goexit()

/usr/lib/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000135fd8 sp=0xc000135fd0 pc=0x468631

created by testing.(*T).Run

/usr/lib/go/src/testing/testing.go:1042 +0x357

What did you expect to see?

No panic. :)

What did you see instead?

Panic. :(

THE WORKAROUND

I was able to work around this issue by changing wrapped error to become a reference: powerman/rpc-codec@c2902e8
But I don't think something was wrong with my code before that commit, so probably it's a Go bug.

[cuonglm]

Can you share your TestGetStr?

Seems duplicated of #29264

[powerman]

@cuonglm Sure, but there is nothing interesting in it - I'm testing jsonrpc2 client using jsonrpc2 server implemented by gomock.

func TestGetStr(tt *testing.T) {
	t := check.T(tt)
	endpoint, mockServer := testNewServer(t)
	c := New(Config{
		Endpoint:    endpoint,
		PlatformKey: "key",
	})
	reqCtx := proto.Ctx{
		PlatformKey: "key",
		ServiceName: def.ProgName,
	}
	req := proto.IDGetStrReq{
		Ctx:    reqCtx,
		Amount: 3,
	}
	resp := proto.IDGetStrResp{"rnd1", "rnd2"}
	err42 := jsonrpc2.NewError(42, "answer")
	mockServer.EXPECT().GetStr(req, gomock.Any()).SetArg(1, resp).Return(nil)
	mockServer.EXPECT().GetStr(req, gomock.Any()).Return(err42)

	res, err := c.GetStr(3)
	t.Nil(err)
	t.DeepEqual(res, resp)
	res, err = c.GetStr(3)
	t.Err(err, err42)
	t.Nil(res)
}

func testNewServer(t *check.C) (string, *proto.MockServer) {
	t.Helper()
	ctrl := gomock.NewController(t)
	mockServer := proto.NewMockServer(ctrl)

	rpcAddr := netx.NewAddr("localhost", netx.UnusedTCPPort("localhost"))
	ctx, cancel := context.WithCancel(ctx)
	errc := make(chan error, 1)
	go func() { errc <- serve.RPCName(ctx, rpcAddr, mockServer, "ID") }()

	t.Cleanup(func() {
		t.Helper()
		cancel()
		t.Nil(<-errc, "serve.RPC")
		ctrl.Finish()
	})

	endpoint := fmt.Sprintf("http://%s/rpc", rpcAddr)
	return endpoint, mockServer
}

[andybons]

@aclements