demo: seg fault at interactive prompt without specific input

[rytaft]

Here is the output of my terminal, using the latest master:

Rebeccas-MacBook-Pro:cockroach becca$ ./cockroach demo --nodes 9 --global
#
# Welcome to the CockroachDB demo database!
#
# You are connected to a temporary, in-memory CockroachDB cluster of 9 nodes.
#
# This demo session will attempt to enable enterprise features
# by acquiring a temporary license from Cockroach Labs in the background.
# To disable this behavior, set the environment variable
# COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING=true.
#
# Beginning initialization of the movr dataset, please wait...
#
# The cluster has been preloaded with the "movr" dataset
# (MovR is a fictional vehicle sharing company).
#
# Reminder: your changes to data stored in the demo session will not be saved!
#
# Connection parameters:
#   (console) http://127.0.0.1:8080/demologin?password=demo49832&username=demo
#   (sql)     postgres://demo:demo49832@?host=%2Fvar%2Ffolders%2Fwc%2Fhvy8pyy128z3v_f9lhvxw99h0000gn%2FT%2Fdemo943322695&port=26257
#   (sql/tcp) postgres://demo:demo49832@127.0.0.1:26257?sslmode=require
# 
# To display connection parameters for other nodes, use \demo ls.
#
# The user "demo" with password "demo49832" has been created. Use it to access the Web UI!
#
# Server version: CockroachDB CCL v21.1.0-alpha.3-1597-g24e76d3990 (x86_64-apple-darwin20.3.0, built , go1.15.5) (same version as client)
# Cluster ID: 73bb9025-3ed7-4f98-90ea-584cc29fa13a
# Organization: Cockroach Demo
#
# Enter \? for a brief introduction.
#
demo@127.0.0.1:26257/movr> SHOW REGIONS;
     region    |  zones  | database_names | primary_region_of
---------------+---------+----------------+--------------------
  europe-west1 | {b,c,d} | {}             | {}
  us-east1     | {b,d,c} | {}             | {}
  us-west1     | {b,a,c} | {}             | {}
(3 rows)

Time: 1.198s total (execution 1.198s / network 0.000s)

demo@127.0.0.1:26257/movr> SHOW TABLES;
  schema_name |         table_name         | type  | owner | estimated_row_count | locality
--------------+----------------------------+-------+-------+---------------------+-----------
  public      | promo_codes                | table | demo  |                1000 | NULL
  public      | rides                      | table | demo  |                 500 | NULL
  public      | user_promo_codes           | table | demo  |                   0 | NULL
  public      | users                      | table | demo  |                  50 | NULL
  public      | vehicle_location_histories | table | demo  |                1000 | NULL
  public      | vehicles                   | table | demo  |                  15 | NULL
(6 rows)

Time: 420ms total (execution 365ms / network 55ms)

demo@127.0.0.1:26257/movr> ALTER DATABASE movr PRIMARY REGION “us-east1”
;
invalid syntax: statement ignored: at or near "-": syntax error
SQLSTATE: 42601
DETAIL: source SQL:
ALTER DATABASE movr PRIMARY REGION “us-east1”
                                        ^
demo@127.0.0.1:26257/movr> ALTER DATABASE movr PRIMARY REGION "us-east1";
ALTER DATABASE PRIMARY REGION

Time: 6.719s total (execution 3.213s / network 3.507s)

demo@127.0.0.1:26257/movr> SHOW DATABASES;
  database_name | owner | primary_region |  regions   | survival_goal
----------------+-------+----------------+------------+----------------
  defaultdb     | root  | NULL           | {}         | NULL
  movr          | demo  | us-east1       | {us-east1} | zone
  postgres      | root  | NULL           | {}         | NULL
  system        | node  | NULL           | {}         | NULL
(4 rows)

Time: 1.388s total (execution 1.388s / network 0.000s)

demo@127.0.0.1:26257/movr> SHOW REGIONS;
     region    |  zones  | database_names | primary_region_of
---------------+---------+----------------+--------------------
  europe-west1 | {b,c,d} | {}             | {}
  us-east1     | {b,d,c} | {movr}         | {movr}
  us-west1     | {b,a,c} | {}             | {}
(3 rows)

Time: 2.236s total (execution 2.235s / network 0.001s)

demo@127.0.0.1:26257/movr> SHOW TABLES;
  schema_name |         table_name         | type  | owner | estimated_row_count |              locality
--------------+----------------------------+-------+-------+---------------------+--------------------------------------
  public      | promo_codes                | table | demo  |                1000 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | rides                      | table | demo  |                 500 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | user_promo_codes           | table | demo  |                   0 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | users                      | table | demo  |                  50 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | vehicle_location_histories | table | demo  |                1000 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | vehicles                   | table | demo  |                  15 | REGIONAL BY TABLE IN PRIMARY REGION
(6 rows)

Time: 1.015s total (execution 1.015s / network 0.000s)

demo@127.0.0.1:26257/movr> SHOW ZONE CONFIGURATION FOR DATABASE movr;
     target     |                 raw_config_sql
----------------+-------------------------------------------------
  DATABASE movr | ALTER DATABASE movr CONFIGURE ZONE USING
                |     range_min_bytes = 134217728,
                |     range_max_bytes = 536870912,
                |     gc.ttlseconds = 90000,
                |     num_replicas = 3,
                |     num_voters = 3,
                |     constraints = '{+region=us-east1: 1}',
                |     voter_constraints = '[+region=us-east1]',
                |     lease_preferences = '[[+region=us-east1]]'
(1 row)

Time: 345ms total (execution 344ms / network 1ms)

demo@127.0.0.1:26257/movr> ALTER DATABASE movr PRIMARY REGION "europe-west1";
ERROR: region "europe-west1" has not been added to the database
SQLSTATE: 42602
HINT: you must add the region to the database before setting it as primary region, using ALTER DATABASE movr ADD REGION "europe-west1"
demo@127.0.0.1:26257/movr> ALTER DATABASE movr add REGION "europe-west1";
ALTER DATABASE ADD REGION

Time: 9.973s total (execution 1.830s / network 8.143s)

demo@127.0.0.1:26257/movr> ALTER DATABASE movr add REGION "us-west1";
ALTER DATABASE ADD REGION

Time: 9.641s total (execution 1.804s / network 7.836s)

demo@127.0.0.1:26257/movr> ALTER DATABASE movr SURVIVE REGION FAILURE;
ALTER DATABASE SURVIVE

Time: 9.183s total (execution 5.443s / network 3.741s)

demo@127.0.0.1:26257/movr> alter table rides set locality regional by row;
NOTICE: LOCALITY changes will be finalized asynchronously; further schema changes on this table may be restricted until the job completes
ALTER TABLE SET LOCALITY

Time: 30.254s total (execution 3.117s / network 27.136s)

demo@127.0.0.1:26257/movr> show tables;
  schema_name |         table_name         | type  | owner | estimated_row_count |              locality
--------------+----------------------------+-------+-------+---------------------+--------------------------------------
  public      | promo_codes                | table | demo  |                1000 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | rides                      | table | demo  |                 500 | REGIONAL BY ROW
  public      | user_promo_codes           | table | demo  |                   0 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | users                      | table | demo  |                  50 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | vehicle_location_histories | table | demo  |                1000 | REGIONAL BY TABLE IN PRIMARY REGION
  public      | vehicles                   | table | demo  |                  15 | REGIONAL BY TABLE IN PRIMARY REGION
(6 rows)

Time: 1.129s total (execution 1.128s / network 0.000s)

demo@127.0.0.1:26257/movr> unexpected fault address 0xfffffff800000019
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xfffffff800000019 pc=0x5658770]

goroutine 2093790 [running]:
runtime.throw(0x865aaf3, 0x5)
	/usr/local/opt/go/libexec/src/runtime/panic.go:1116 +0x72 fp=0xc0338bdd18 sp=0xc0338bdce8 pc=0x403c7b2
runtime.sigpanic()
	/usr/local/opt/go/libexec/src/runtime/signal_unix.go:749 +0x3e5 fp=0xc0338bdd48 sp=0xc0338bdd18 pc=0x40532e5
github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).PublicColumns(0xc0005adb00, 0x0, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc/table_desc.go:306 +0x30 fp=0xc0338bdd68 sp=0xc0338bdd48 pc=0x5658770
github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkeys.MakeNameMetadataKey(0xc000d04e40, 0xc000d04e40, 0x0, 0x1d00000001, 0xc042661590, 0x4, 0x0, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkeys/keys.go:284 +0x1d6 fp=0xc0338bde20 sp=0xc0338bdd68 pc=0x56c5d16
github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkv.(*oneLevelUncachedDescGetter).GetNamespaceEntries(0xc03ea2a520, 0x9495780, 0xc03205b8c0, 0xc03ed993c0, 0x1, 0x1, 0x3, 0x0, 0xc0325ddae0, 0xc02d8381a0, ...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkv/desc_getter.go:129 +0x91e fp=0xc0338be008 sp=0xc0338bde20 pc=0x56d2e3e
github.com/cockroachdb/cockroach/pkg/sql/catalog.(*validationDescGetterImpl).addNamespaceEntries(0xc03ea2a580, 0x9495780, 0xc03205b8c0, 0xc014fbb8b0, 0x1, 0x1, 0x16a530d8, 0xc03ea2a520, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/validate.go:293 +0x472 fp=0xc0338be108 sp=0xc0338be008 pc=0x55a6ef2
github.com/cockroachdb/cockroach/pkg/sql/catalog.Validate(0x9495780, 0xc03205b8c0, 0x16a530d8, 0xc03ea2a520, 0xc00000001f, 0xc014fbb8b0, 0x1, 0x1, 0xc024abe8b0, 0x24abe8b004076f20)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/validate.go:81 +0x397 fp=0xc0338be1f0 sp=0xc0338be108 pc=0x55a5b77
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).ValidateUncommittedDescriptors(0xc030cb9570, 0x9495780, 0xc03205b8c0, 0xc048bc3e60, 0x24abe8b0, 0xed7e72495)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/collection.go:1531 +0x1ea fp=0xc0338be278 sp=0xc0338be1f0 pc=0x573794a
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransactionInternal(0xc030cb9300, 0x9495780, 0xc03205b8c0, 0x94b4040, 0xc0199f12c0, 0xc032182d80, 0xc031a70328)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:755 +0x69 fp=0xc0338be2c0 sp=0xc0338be278 pc=0x70080a9
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransaction(0xc030cb9300, 0x9495780, 0xc03205b8c0, 0x94b4040, 0xc0199f12c0, 0xc031a703e0, 0x4013290, 0xc0338be380, 0x570018d)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:738 +0x8c fp=0xc0338be318 sp=0xc0338be2c0 pc=0x7007f2c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).handleAutoCommit(0xc030cb9300, 0x9495780, 0xc03205b8c0, 0x94b4040, 0xc0199f12c0, 0xc0338be420, 0x40, 0xc00bd20900, 0x8d50c08)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1443 +0xba fp=0xc0338be3b0 sp=0xc0338be318 pc=0x700cfda
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState.func7(0xc030cb9300, 0xc0338bf270, 0xc0338bf260, 0xc0338bf1f0, 0xc0338be8d8, 0xc0338bf250, 0x94466c0, 0xb3fc8db)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:444 +0xc9 fp=0xc0338be420 sp=0xc0338be3b0 pc=0x71ba809
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc030cb9300, 0x9495780, 0xc03205b8c0, 0x94b4040, 0xc0199f12c0, 0x87e4979, 0x74, 0x4, 0x1, 0xc0317dc0f0, ...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:702 +0x124f fp=0xc0338bf1e8 sp=0xc0338be420 pc=0x70058ef
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc030cb9300, 0x94956c0, 0xc031fe5bc0, 0x94b4040, 0xc0199f12c0, 0x87e4979, 0x74, 0x4, 0x1, 0xc0317dc0f0, ...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:120 +0xb14 fp=0xc0338bf718 sp=0xc0338bf1e8 pc=0x7003f94
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal(0xc030cb9300, 0x94956c0, 0xc031fe5bc0, 0xc0317dc0f0, 0xc031fe5b80, 0x4, 0x4, 0x0, 0x0, 0x0, ...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:207 +0x14e fp=0xc0338bf850 sp=0xc0338bf718 pc=0x70043ee
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2(0x0, 0x0, 0x0, 0x14d22b70, 0xed7e72495, 0x0, 0xc030cb9300, 0xc0338bfc18, 0xc0338bfc08, 0x2, ...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1583 +0x47b fp=0xc0338bfa68 sp=0xc0338bf850 pc=0x71b8efb
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc030cb9300, 0x94956c0, 0xc031fe5bc0, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1585 +0x45c fp=0xc0338bfe90 sp=0xc0338bfa68 pc=0x6ff853c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc030cb9300, 0x9495780, 0xc031cb59e0, 0xc001713180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1437 +0x228 fp=0xc0338bff30 sp=0xc0338bfe90 pc=0x6ff7e68
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1(0xc030cb9300, 0x9495780, 0xc031cb59e0, 0xc000469d40, 0xc03e0b05c0, 0xc031cab740, 0x0, 0xc041bee3c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:202 +0x65 fp=0xc0338bffa0 sp=0xc0338bff30 pc=0x71c6305
runtime.goexit()
	/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0338bffa8 sp=0xc0338bffa0 pc=0x4075761
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:201 +0x2dc

goroutine 1 [syscall]:
github.com/knz/go-libedit/unix._C2func_go_libedit_gets(0x37fe0000, 0x3cb77000, 0x0, 0x0, 0xc027587740, 0xc027587744, 0xc000000001, 0x0, 0x0, 0x0)
	_cgo_gotypes.go:190 +0x56
github.com/knz/go-libedit/unix.EditLine.GetLine.func1(0xc00e0e0e00, 0xc027587740, 0xc027587744, 0xc00e0e0e00, 0xc006cd16a0, 0x78d2c33)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/unix/editline_unix.go:281 +0xe7
github.com/knz/go-libedit/unix.EditLine.GetLine(0x0, 0xb812ea0, 0x0, 0x0, 0xb812ea0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/knz/go-libedit/unix/editline_unix.go:281 +0xaa
github.com/cockroachdb/cockroach/pkg/cli.(*cliState).doReadLine(0xc00b83f770, 0x6, 0x5)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:953 +0x5e5
github.com/cockroachdb/cockroach/pkg/cli.(*cliState).doRunShell(0xc00b83f770, 0x0, 0xc000010010, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:1482 +0xb4
github.com/cockroachdb/cockroach/pkg/cli.runInteractive(...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:1450
github.com/cockroachdb/cockroach/pkg/cli.runClient(0xb77e860, 0xc00b7ef720, 0xc000010010, 0x52, 0xc00b3ef968)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/sql.go:1715 +0xea
github.com/cockroachdb/cockroach/pkg/cli.runDemo(0xb77e860, 0x9440bc0, 0xc000d82360, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/demo.go:367 +0x815
github.com/cockroachdb/cockroach/pkg/cli.init.4.func1(0xb77e860, 0xc000591f80, 0x0, 0x3, 0x1, 0x405436c)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/demo.go:55 +0x33
github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1(0xb77e860, 0xc000591f80, 0x0, 0x3, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:194 +0x79
github.com/spf13/cobra.(*Command).execute(0xb77e860, 0xc000591f20, 0x3, 0x3, 0xb77e860, 0xc000591f20)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:826 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xb77cf60, 0xc0000740f0, 0xc000447e9a, 0x4)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914 +0x30b
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:312
github.com/cockroachdb/cockroach/pkg/cli.doMain(0xb77e860, 0xc000447e9a, 0x4, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:112 +0x11a
github.com/cockroachdb/cockroach/pkg/cli.Main()
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:60 +0x13a
main.main()
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short/main.go:26 +0x25

goroutine 31 [chan receive, 17 minutes]:
github.com/cockroachdb/cmux.muxListener.Accept(...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/pkg/rpc.delayingListener.Accept(0x946fac0, 0xc0026353e0, 0xc0040e7df0, 0x18, 0xc0002a5080, 0x483af9e)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:858 +0x3b
google.golang.org/grpc.(*Server).Serve(0xc0041129c0, 0x94701c0, 0xc003377140, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:621 +0x216
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x9495780, 0xc001234ed0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1972 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000e8b980, 0x9495780, 0xc001234ed0, 0x0, 0xc002695660)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 8 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:75 +0x73
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:41 +0x35

goroutine 9 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:98 +0x12c
created by github.com/cockroachdb/cockroach/pkg/util/log.init.5
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:42 +0x4d

goroutine 34 [syscall, 17 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/opt/go/libexec/src/runtime/sigqueue.go:144 +0x9d
os/signal.loop()
	/usr/local/opt/go/libexec/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
	/usr/local/opt/go/libexec/src/os/signal/signal.go:150 +0x45

goroutine 52 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0009f2320)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57

goroutine 16 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8c180)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8c180, 0xc0003a3c60, 0x1, 0x1, 0xc0005a97b8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787100)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 15 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c88060)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c88060, 0xc0003a3c60, 0x1, 0x1, 0xc0005a8fb8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001786200)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 370 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc003c54540)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc003c58080, 0x9495780, 0xc003c54540, 0x0, 0xc001e086f0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 44 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart.func1(0x9495780, 0xc0013407e0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1310 +0x4b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc001468800, 0x9495780, 0xc0013407e0, 0x0, 0xc002662b80)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 66 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c880c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c880c0, 0xc0003a3c60, 0x1, 0x1, 0xc0005a87b8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787200)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 67 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c88120)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c88120, 0xc0003a3c60, 0x1, 0x1, 0xc000088fb8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787300)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 1749345 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/opt/go/libexec/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc002b86fe0, 0x3f4)
	/usr/local/opt/go/libexec/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc002b86fd0)
	/usr/local/opt/go/libexec/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/pebble/internal/record.(*flusherCond).Wait(...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:193
github.com/cockroachdb/pebble/internal/record.(*LogWriter).flushLoop(0xc002b86f00, 0x9495780, 0xc0143abe00)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:396 +0x10b
runtime/pprof.Do(0x9495780, 0xc0143abe00, 0xc0003a32c0, 0x1, 0x1, 0xc0097227b8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble/internal/record.NewLogWriter.func2(0xc002b86f00)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:314 +0x89
created by github.com/cockroachdb/pebble/internal/record.NewLogWriter
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:313 +0x2ea

goroutine 43 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0xe9b6ed8, 0x72, 0x0)
	/usr/local/opt/go/libexec/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc001468718, 0x72, 0x0, 0x0, 0x865d60a)
	/usr/local/opt/go/libexec/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/opt/go/libexec/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc001468700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/opt/go/libexec/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc001468700, 0x37e11d600, 0x0, 0x0)
	/usr/local/opt/go/libexec/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc001333e40, 0xc004152e10, 0xc004152e18, 0x28)
	/usr/local/opt/go/libexec/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc001333e40, 0x8d4eec8, 0xc00128a200, 0x94fcac0, 0xc037429fa0)
	/usr/local/opt/go/libexec/src/net/tcpsock.go:261 +0x65
github.com/cockroachdb/cmux.(*cMux).Serve(0xc00128a200, 0x0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0xa4
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.2.1()
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1977 +0x2f
sync.(*Once).doSlow(0xc0009a1900, 0xc000ad4f20)
	/usr/local/opt/go/libexec/src/sync/once.go:66 +0xec
sync.(*Once).Do(...)
	/usr/local/opt/go/libexec/src/sync/once.go:57
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.2(0x9495780, 0xc001b90960)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1976 +0x74
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019bc500, 0x9495780, 0xc001b90960, 0x0, 0xc002662b60)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 70 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x94956c0, 0xc00053e480)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:402 +0x51
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x94956c0, 0xc00053e480, 0x0, 0xc0006765e0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 71 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc000dd0100, 0x9495700, 0xc0000740f0, 0xc000cfa0f0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:235 +0x159
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:232 +0x65

goroutine 72 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x9495700, 0xc0000740f0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x17b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495700, 0xc0000740f0, 0x0, 0xc0009eefc0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 65 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc003c54540)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc003c58080, 0x9495780, 0xc003c54540, 0x0, 0xc001e086a0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 64 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc003c54540)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc003c58080, 0x9495780, 0xc003c54540, 0x0, 0xc001e08690)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 75 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000e97aa0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000e97aa0, 0xc0003a3c60, 0x1, 0x1, 0xc0005a2fb8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787700)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 76 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c881e0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c881e0, 0xc0003a3c60, 0x1, 0x1, 0xc0005a37b8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787800)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 77 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000c88240)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000c88240, 0xc0003a3c60, 0x1, 0x1, 0xc0005a4fb8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787900)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 78 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8c450)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8c450, 0xc0003a3c60, 0x1, 0x1, 0xc0005a57b8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001787c00)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 79 [sync.Cond.Wait, 17 minutes]:
runtime.goparkunlock(...)
	/usr/local/opt/go/libexec/src/runtime/proc.go:312
sync.runtime_notifyListWait(0xc00032afe0, 0x0)
	/usr/local/opt/go/libexec/src/runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc00032afd0)
	/usr/local/opt/go/libexec/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/pebble/internal/record.(*flusherCond).Wait(...)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:193
github.com/cockroachdb/pebble/internal/record.(*LogWriter).flushLoop(0xc00032af00, 0x9495780, 0xc000d8c630)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:396 +0x10b
runtime/pprof.Do(0x9495780, 0xc000d8c630, 0xc0003a32c0, 0x1, 0x1, 0xc0005a5fb8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble/internal/record.NewLogWriter.func2(0xc00032af00)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:314 +0x89
created by github.com/cockroachdb/pebble/internal/record.NewLogWriter
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/internal/record/log_writer.go:313 +0x2ea

goroutine 30 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart.func5(0x9495780, 0xc001234ed0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1351 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000e8b980, 0x9495780, 0xc001234ed0, 0x0, 0xc002694e20)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 81 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9990)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 82 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99a0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 83 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99b0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 84 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 85 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99d0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 86 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99e0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 87 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e99f0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 88 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a00)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 89 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a10)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 90 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a20)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 91 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a30)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 92 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a40)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 93 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a50)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 94 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a60)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 95 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a70)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 96 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:105 +0x13d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc0013e9a80)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 3747 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc006468700)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1339 +0x193
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:295 +0x16fc

goroutine 98 [select, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x8d58d50, 0xc000683d10, 0xc0003ee2a0, 0x9495780, 0xc000d183c0, 0x0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:230 +0x12d
created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:223 +0xd0

goroutine 99 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x9495700, 0xc0000740f8)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:113 +0x65
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495700, 0xc0000740f8, 0x0, 0xc000d385a0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 100 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2001 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc000d38600)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 101 [IO wait, 17 minutes]:
internal/poll.runtime_pollWait(0xe9b7530, 0x72, 0x0)
	/usr/local/opt/go/libexec/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000450798, 0x72, 0x0, 0x0, 0x865d60a)
	/usr/local/opt/go/libexec/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/opt/go/libexec/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000450780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/opt/go/libexec/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc000450780, 0xc0014605a0, 0x4013bf8, 0xc000500000)
	/usr/local/opt/go/libexec/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000d385e0, 0xc000a115c0, 0x4013bf8, 0x30)
	/usr/local/opt/go/libexec/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000d385e0, 0x837db00, 0xc0014605a0, 0x7f46660, 0xb73dea0)
	/usr/local/opt/go/libexec/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc00144c000, 0x946dd80, 0xc000d385e0, 0x0, 0x0)
	/usr/local/opt/go/libexec/src/net/http/server.go:2937 +0x266
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2049 +0x3f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc000d38640)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 102 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func1(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1902 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc000d38b00)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 103 [chan receive, 17 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func3(0x9495780, 0xc000d183c0)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1946 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000dd0100, 0x9495780, 0xc000d183c0, 0x0, 0xc000d38b60)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:351 +0xb9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/Users/becca/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xfc

goroutine 104 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc001460600)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc001460600, 0xc0003a3c60, 0x1, 0x1, 0xc0005a7fb8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001412200)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 105 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8c450)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8c450, 0xc0003a3c60, 0x1, 0x1, 0xc0005a77b8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001412300)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 106 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8c780)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8c780, 0xc0003a3c60, 0x1, 0x1, 0xc000a107b8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001412400)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 107 [chan receive, 17 minutes]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1(0x9495780, 0xc000d8d140)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:176 +0x76
runtime/pprof.Do(0x9495780, 0xc000d8d140, 0xc0003a3c60, 0x1, 0x1, 0xc000a177b8)
	/usr/local/opt/go/libexec/src/runtime/pprof/runtime.go:40 +0xcc
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc001412500)
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:175 +0x89
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	/Users/becca/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:167 +0x1a5

goroutine 0 [idle]:
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x130 pc=0x40655b0]

runtime stack:
runtime.throw(0x8708d39, 0x2a)
	/usr/local/opt/go/libexec/src/runtime/panic.go:1116 +0x72
runtime.sigpanic()
	/usr/local/opt/go/libexec/src/runtime/signal_unix.go:726 +0x48c
runtime.traceback1(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc0005abb00, 0x0)
	/usr/local/opt/go/libexec/src/runtime/traceback.go:712 +0x50
runtime.traceback(...)
	/usr/local/opt/go/libexec/src/runtime/traceback.go:690
runtime.tracebackothers(0xc00bd20900)
	/usr/local/opt/go/libexec/src/runtime/traceback.go:944 +0x165
runtime.dopanic_m(0xc00bd20900, 0x403c7b2, 0xc0338bdce8, 0x865aaf3)
	/usr/local/opt/go/libexec/src/runtime/panic.go:1322 +0x2bf
runtime.fatalthrow.func1()
	/usr/local/opt/go/libexec/src/runtime/panic.go:1171 +0x5f
runtime.systemstack(0xec17740)
	/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/usr/local/opt/go/libexec/src/runtime/proc.go:1116

Jira issue: CRDB-6374

[rytaft]

cc @postamar @ajwerner not sure if this actually has anything to do with schema stuff, but it appears that the seg fault happened inside table_desc.go.

[ajwerner]

It almost definitely has something to do with schema stuff.

[postamar]

This bug is really scary, to be honest. I'm unable to repro but I'll find a way somehow. Just out of curiosity @rytaft was this a one-off or were you able to trigger this segfault multiple times?

[rytaft]

I only saw this once -- sorry! Thanks for investigating!

[postamar]

I was able to repro it, but just once! Unfortunately my terminal crashed and I didn't copy/paste the output.

Here's what I did after running cockroach demo --nodes 9 --global

ALTER DATABASE movr PRIMARY REGION "us-east1";
ALTER DATABASE movr add REGION "us-west1";
ALTER DATABASE movr add REGION "europe-west1";
ALTER DATABASE movr SURVIVE REGION FAILURE;
alter table rides set locality regional by row;
SHOW JOBS

edit: not quite the same segfault, but it was something else scary related to runtime corruption!

[blathers-crl]

Hi @postamar, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[ajwerner]

It sort of feels plausibly related to golang/go#43873 -- but far from certainly.

[ajwerner]

I created #62391 to make repro'ing this easier.

[awoods187]

I crashed demo today by running ./cockroach demo movr --nodes 9 followed by the new refresh built in. Here's a gist https://gist.github.com/awoods187/e9289c706ddc529034368fd3a2a25323

[ajwerner]

@awoods187 this confirms this is a memory corruption bug somewhere and has nothing to do with @postamar and schema code. Nevertheless, it's very scary.