Skip to content

kvserver/redact: fatal error: invalid pointer found on stack during TestIngest #74386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dt opened this issue Jan 3, 2022 · 3 comments · Fixed by #74655
Closed

kvserver/redact: fatal error: invalid pointer found on stack during TestIngest #74386

dt opened this issue Jan 3, 2022 · 3 comments · Fixed by #74655
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting. T-kv KV Team T-server-and-security DB Server & Security

Comments

@dt
Copy link
Member

dt commented Jan 3, 2022 •
edited by knz
Loading

Hit while running bazel test //pkg/ccl/backupccl:backupccl_test --test_output=streamed --test_arg="-test.v" on an unrelated change working off of 053fe7d (the WORKSPACE file as of that revision specifies go 1.17.3).

FWIW, I am running on a Darwin arm64 host.

I220103 06:31:10.283630 930353 kv/kvserver/replica_command.go:400  [n1,s1,r49/1:/{Table/102/1/…-Max}] 3621  initiating a split of this range at key /Table/103/1/"k3" [r50] (manual)
runtime: bad pointer in frame github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leaseStatus at 0x1400345ade0: 0x1a
fatal error: invalid pointer found on stack

runtime stack:
runtime.throw({0x103f03d6a, 0x1e})
    GOROOT/src/runtime/panic.go:1198 +0x54 fp=0x16fb3a730 sp=0x16fb3a700 pc=0x100aa5424
runtime.adjustpointers(0x1400345ab20, 0x16fb3a820, 0x16fb3ac28, {0x107702368, 0x107df1900})
    GOROOT/src/runtime/stack.go:617 +0x210 fp=0x16fb3a780 sp=0x16fb3a730 pc=0x100ac0880
runtime.adjustframe(0x16fb3ab40, 0x16fb3ac28)
    GOROOT/src/runtime/stack.go:659 +0xd4 fp=0x16fb3a860 sp=0x16fb3a780 pc=0x100ac0964
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0x14005175380, 0x0, 0x0, 0x7fffffff, 0x10593b708, 0x16fb3ac28, 0x0)
    GOROOT/src/runtime/traceback.go:350 +0x778 fp=0x16fb3aba0 sp=0x16fb3a860 pc=0x100acde48
runtime.copystack(0x14005175380, 0x4000)
    GOROOT/src/runtime/stack.go:918 +0x260 fp=0x16fb3ad50 sp=0x16fb3aba0 pc=0x100ac10e0
runtime.newstack()
    GOROOT/src/runtime/stack.go:1097 +0x420 fp=0x16fb3af00 sp=0x16fb3ad50 pc=0x100ac15f0
runtime.morestack()
    GOROOT/src/runtime/asm_arm64.s:303 +0x70 fp=0x16fb3af00 sp=0x16fb3af00 pc=0x100ad9320

goroutine 929148 [copystack]:
github.com/cockroachdb/redact/internal/rfmt.(*pp).handleMethods(0x14006ce43c0, 0x73)
    github.com/cockroachdb/redact/internal/rfmt/print.go:606 +0x95c fp=0x1400345a500 sp=0x1400345a500 pc=0x100c8fbfc
github.com/cockroachdb/redact/internal/rfmt.(*pp).printArg(0x14006ce43c0, {0x105791520, 0x1402161cb70}, 0x73)
    github.com/cockroachdb/redact/internal/rfmt/print.go:810 +0xe18 fp=0x1400345a920 sp=0x1400345a500 pc=0x100c90a18
github.com/cockroachdb/redact/internal/rfmt.(*pp).doPrintf(0x14006ce43c0, {0x103fcede4, 0x41}, {0x1400345ab38, 0x2, 0x2})
    github.com/cockroachdb/redact/internal/rfmt/print.go:1144 +0x180 fp=0x1400345a9e0 sp=0x1400345a920 pc=0x100c92830
github.com/cockroachdb/redact/internal/rfmt.Fprintf({0x10599b160, 0x14010995560}, {0x103fcede4, 0x41}, {0x140018f2b38, 0x2, 0x2})
    github.com/cockroachdb/redact/internal/rfmt/print.go:212 +0x54 fp=0x1400345aa50 sp=0x1400345a9e0 pc=0x100c8cea4
github.com/cockroachdb/redact/builder.(*StringBuilder).Printf(0x14010995560, {0x103fcede4, 0x41}, {0x140018f2b38, 0x2, 0x2})
    github.com/cockroachdb/redact/builder/builder.go:82 +0x74 fp=0x1400345aab0 sp=0x1400345aa50 pc=0x100c94934
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leaseStatus(0x14002f94a80, {0x105a14030, 0x1400faea720}, {{0x0, 0x57ac0a0, 0x1}, 0x1402031e580, {0x1556a080, 0x140, 0x1a, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_range_lease.go:632 +0x49c fp=0x1400345ae80 sp=0x1400345aab0 pc=0x101f37e2c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leaseStatusForRequestRLocked(0x14002f94a80, {0x105a14030, 0x1400faea720}, {0x16c6aef263e67210, 0x0, 0x0}, {0x0, 0x0, 0x0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_range_lease.go:705 +0x134 fp=0x1400345b110 sp=0x1400345ae80 pc=0x101f38314
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leaseStatusAtRLocked(...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_range_lease.go:694
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).ownsValidLeaseRLocked(0x14002f94a80, {0x105a14030, 0x1400faea720}, {0x16c6aef263e67210, 0x0, 0x0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_range_lease.go:721 +0x64 fp=0x1400345b630 sp=0x1400345b110 pc=0x101f38514
github.com/cockroachdb/cockroach/pkg/kv/kvserver.shouldReplicaQuiesce({0x105a14030, 0x1400faea720}, {0x105ab1d60, 0x14002f94a80}, {0x16c6aef263e67210, 0x0, 0x0}, 0x140109954a0)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft_quiesce.go:324 +0x438 fp=0x1400345b920 sp=0x1400345b630 pc=0x101f2d8d8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).maybeQuiesceLocked(0x14002f94a80, {0x105a14030, 0x1400faea720}, {0x16c6aef263e67210, 0x0, 0x0}, 0x140109954a0)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft_quiesce.go:184 +0x64 fp=0x1400345b990 sp=0x1400345b920 pc=0x101f2d014
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick(0x14002f94a80, {0x105a14030, 0x1400faea720}, 0x140109954a0)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:1012 +0x35c fp=0x1400345bda0 sp=0x1400345b990 pc=0x101f273fc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick(0x140088b8000, {0x105a2fd20, 0x1401099a4a0}, 0x24)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:531 +0xe8 fp=0x1400345be60 sp=0x1400345bda0 pc=0x101f79f58
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0x1400821a4e0, {0x105a2fd20, 0x1401099a4a0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:300 +0x258 fp=0x1400345bef0 sp=0x1400345be60 pc=0x101f5abb8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker-fm({0x105a2fd20, 0x1401099a4a0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:250 +0x40 fp=0x1400345bf20 sp=0x1400345bef0 pc=0x101f944d0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2(0x14012fb6090, {0x105a2fd20, 0x1401099a4a0}, 0x1401099a300, 0x0, 0x14007e48970)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:488 +0xe4 fp=0x1400345bfa0 sp=0x1400345bf20 pc=0x10162d9c4
runtime.goexit()
    GOROOT/src/runtime/asm_arm64.s:1133 +0x4 fp=0x1400345bfa0 sp=0x1400345bfa0 pc=0x100adb664
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:478 +0x380
@dt dt added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting. labels Jan 3, 2022
@blathers-crl blathers-crl bot added T-server-and-security DB Server & Security T-kv KV Team labels Jan 3, 2022
@dt
Copy link
Member Author

dt commented Jan 3, 2022

I wonder if this is similar to golang/go#47276?

@knz
Copy link
Contributor

knz commented Jan 10, 2022

This looks more like it: https://go-review.googlesource.com/c/go/+/363358/

@knz
Copy link
Contributor

knz commented Jan 10, 2022

fixed in 1.17.6

@nvanbenschoten nvanbenschoten mentioned this issue Jan 11, 2022
Merged
13 tasks
@craig craig bot closed this as completed in b917f1e Jan 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting. T-kv KV Team T-server-and-security DB Server & Security
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*: upgrade to go 1.17.6 rickystewart/cockroach
2 participants
@dt @knz