Data race in TestMemcacheStop

[bboreham]

Describe the bug
As seen in #4507

==================
WARNING: DATA RACE
Read at 0x00c00004c070 by goroutine 8:
  runtime.chansend()
      /usr/local/go/src/runtime/chan.go:158 +0x0
  github.com/cortexproject/cortex/pkg/chunk/cache.NewMemcached.func1()
      /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached.go:92 +0xee

Previous write at 0x00c00004c070 by goroutine 72:
  runtime.closechan()
      /usr/local/go/src/runtime/chan.go:355 +0x0
  github.com/cortexproject/cortex/pkg/chunk/cache.(*Memcached).fetchKeysBatched()
      /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached.go:217 +0x416
  github.com/cortexproject/cortex/pkg/chunk/cache.(*Memcached).Fetch.func1()
      /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached.go:136 +0xb3
  github.com/weaveworks/common/instrument.CollectedRequest()
      /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/instrument/instrument.go:167 +0x2ec
  github.com/cortexproject/cortex/pkg/chunk/cache.(*Memcached).Fetch()
      /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached.go:135 +0x247
  github.com/cortexproject/cortex/pkg/chunk/cache_test.testMemcachedStopping·dwrap·5()
      /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached_test.go:192 +0x8b

Goroutine 8 (running) created at:
  github.com/cortexproject/cortex/pkg/chunk/cache.NewMemcached()
      /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached.go:81 +0x5cd
  github.com/cortexproject/cortex/pkg/chunk/cache_test.TestMemcacheStop.func2()
      /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached_test.go:171 +0x184
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /usr/local/go/src/testing/testing.go:1306 +0x47

Goroutine 72 (running) created at:
  github.com/cortexproject/cortex/pkg/chunk/cache_test.testMemcachedStopping()
      /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached_test.go:192 +0x4d3
  github.com/cortexproject/cortex/pkg/chunk/cache_test.TestMemcacheStop.func2()
      /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached_test.go:176 +0x191
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /usr/local/go/src/testing/testing.go:1306 +0x47
==================
panic: send on closed channel

goroutine 658 [running]:
github.com/cortexproject/cortex/pkg/chunk/cache.NewMemcached.func1()
        /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached.go:92 +0xef
created by github.com/cortexproject/cortex/pkg/chunk/cache.NewMemcached
        /home/vagrant/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached.go:81 +0x5ce
FAIL    github.com/cortexproject/cortex/pkg/chunk/cache 1.456s
FAIL

To Reproduce

go test -count 100 -race -run Stop ./pkg/chunk/cache/

Expected behavior
No failure

Environment:

• Cortex commit 1d29897
• Linux vagrant 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• go version go1.17.1 linux/amd64