Data Race (while running server tests) [FAIL: TestTQMFetchesUserDataFailsAndTriesAgain]

[mxbvkn]

Expected Behavior

All tests pass

Actual Behavior

==================
WARNING: DATA RACE
Write at 0x00010506ff60 by goroutine 586:
  go.temporal.io/server/service/matching.setScoped[...]()
      /Users/mxbvkn/Temporal/Public/temporal-mxbvkn/service/matching/task_queue_manager_test.go:78 +0x1c4
  go.temporal.io/server/service/matching.TestTQMFetchesUserDataFailsAndTriesAgain()
      /Users/mxbvkn/Temporal/Public/temporal-mxbvkn/service/matching/task_queue_manager_test.go:807 +0x230
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1576 +0x188
  testing.(*T).Run.func1()
      /usr/local/go/src/testing/testing.go:1629 +0x40

Previous read at 0x00010506ff60 by goroutine 490:
  go.temporal.io/server/service/matching.(*taskQueueManagerImpl).fetchUserData()
      /Users/mxbvkn/Temporal/Public/temporal-mxbvkn/service/matching/task_queue_manager.go:853 +0x360
  go.temporal.io/server/service/matching.(*taskQueueManagerImpl).fetchUserData-fm()
      <autogenerated>:1 +0x44
  go.temporal.io/server/internal/goro.(*Group).Go.func1()
      /Users/mxbvkn/Temporal/Public/temporal-mxbvkn/internal/goro/group.go:58 +0x94

Goroutine 586 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1629 +0x5e4
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2036 +0x80
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1576 +0x188
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2034 +0x700
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1906 +0x950
  main.main()
      _testmain.go:183 +0x300

Goroutine 490 (running) created at:
  go.temporal.io/server/internal/goro.(*Group).Go()
      /Users/mxbvkn/Temporal/Public/temporal-mxbvkn/internal/goro/group.go:56 +0xf8
  go.temporal.io/server/service/matching.(*taskQueueManagerImpl).Start()
      /Users/mxbvkn/Temporal/Public/temporal-mxbvkn/service/matching/task_queue_manager.go:319 +0xf4
  go.temporal.io/server/service/matching.(*matchingEngineSuite).TestTaskQueueManagerGetTaskBatch_ReadBatchDone()
      /Users/mxbvkn/Temporal/Public/temporal-mxbvkn/service/matching/matching_engine_test.go:1775 +0x1a8
  runtime.call16()
      /usr/local/go/src/runtime/asm_arm64.s:478 +0x74
  reflect.Value.Call()
      /usr/local/go/src/reflect/value.go:370 +0x90
  github.com/stretchr/testify/suite.Run.func1()
      /Users/mxbvkn/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x540
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1576 +0x188
  testing.(*T).Run.func1()
      /usr/local/go/src/testing/testing.go:1629 +0x40
==================
2023-07-06T17:16:42.296-0700	info	none	{"wf-task-queue-name": "/_sys/tq/1", "wf-task-queue-type": "Workflow", "wf-namespace": "", "lifecycle": "Started", "logging-call-at": "task_queue_manager.go:320"}
2023-07-06T17:16:42.397-0700	info	none	{"wf-task-queue-name": "/_sys/tq/1", "wf-task-queue-type": "Workflow", "wf-namespace": "", "lifecycle": "Stopped", "logging-call-at": "task_queue_manager.go:350"}
--- FAIL: TestTQMFetchesUserDataFailsAndTriesAgain (0.10s)

Steps to Reproduce the Problem

1. Clone official master branch (latest as of 07/06/23 ~5:15 PDT)
2. Run steps as outlined in https://github.com/temporalio/temporal/blob/master/CONTRIBUTING.md -- up to make test

Specifications

• Platform: MacOS 13.4.1 (22F82), MBP M1 Max

[dnr]

Also fixed by #4604