x/tools/gopls/internal/regtest/completion: sandbox cleanup errors with "directory not empty" in TestPostfixSnippetCompletion

[bcmills]

closing test runner: errors closing the test runner:
	error(s) cleaning sandbox: cleaning modcache: <nil>; removing files: unlinkat /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/gopls-regtest-664105203/TestPostfixSnippetCompletion/singleton: directory not empty
FAIL	golang.org/x/tools/gopls/internal/regtest/completion	13.493s

greplogs --dashboard -md -l -e 'error\(s\) cleaning sandbox.*directory not empty'

2022-01-20T01:03:13-7c251d6-e7d5857/darwin-amd64-10_15
2022-01-13T20:54:45-3f6aab1-9de1ac6/darwin-amd64-10_15
2022-01-07T05:34:58-351aaa6-40afced/netbsd-amd64-9_0-n1
2021-12-15T20:26:19-c6ae451-07ed86c/darwin-arm64-11_0-toothrot
2021-11-29T20:12:13-6e52f51-f598e29/netbsd-386-9_0-n2d
2021-11-25T04:02:39-cb80a01-45bae64/darwin-amd64-10_14
2021-11-13T03:33:55-49ce184-c239790/darwin-arm64-11_0-toothrot
2021-11-06T16:43:43-0c60b7c-3544082/linux-amd64-longtest
2021-11-03T20:51:16-84e69e7-88407a8/darwin-arm64-11_0-toothrot
2021-11-03T19:38:32-012c320-88407a8/darwin-amd64-10_14
2021-11-02T15:33:01-4d06874-da7173a/darwin-arm64-11_0-toothrot
2021-10-29T21:19:39-a2be0cd-994049a/darwin-amd64-10_14
2021-10-20T07:21:22-4ea6123-d9421ce/darwin-arm64-11_0-toothrot
2021-10-15T14:09:01-98f6e03-2ac3bdf/darwin-amd64-10_14
2021-10-01T15:56:03-6f5fd9b-04242ac/darwin-amd64-10_14
2021-09-21T18:37:00-0d12d39-552410f/darwin-arm64-11_0-toothrot
2021-08-15T19:56:18-a55d515-717894c/darwin-arm64-11_0-toothrot
2021-08-11T15:16:17-5f5a173-fa6aa87/darwin-amd64-10_14
2021-08-06T19:38:52-d529aec-fa6aa87/darwin-amd64-10_14
2021-08-03T15:00:47-3395cb0-37c117f/darwin-arm64-11_0-toothrot
2021-07-26T20:19:17-4ad98e9-ed8cbbc/darwin-amd64-10_14
2021-07-15T13:35:41-6e9046b-21a04e3/darwin-arm64-11_0-toothrot
2021-07-14T22:42:09-0cf4e27-bc51e93/darwin-arm64-11_0-toothrot
2021-07-10T00:32:22-5b540d3-fb052db/darwin-arm64-11_0-toothrot
2021-07-08T21:02:30-64bd808-296ddf2/darwin-arm64-11_0-toothrot
2021-07-02T16:14:50-7edcfe5-cb4cd9e/darwin-arm64-11_0-toothrot
2021-07-01T08:39:23-20dafe5-cb4cd9e/netbsd-386-9_0
2021-06-14T21:46:05-4b484fb-0fd20ed/dragonfly-amd64
2021-06-11T15:30:39-490eac8-7677616/darwin-amd64-10_14
2021-06-04T20:34:05-1225b6f-831f937/darwin-arm64-11_0-toothrot
2021-05-24T18:10:39-88a9bcc-b836106/darwin-arm64-11_0-toothrot
2021-05-21T23:46:07-1e0c960-f87194c/netbsd-arm64-bsiegert
2021-05-19T16:08:23-49064d2-04cd717/darwin-arm64-11_0-toothrot
2021-05-19T14:12:35-a0f4b7b-44a6805/netbsd-386-9_0
2021-05-19T14:12:35-a0f4b7b-04cd717/darwin-arm64-11_0-toothrot
2021-05-03T20:05:58-19b1717-9f34703/darwin-arm64-11_0-toothrot
2021-04-28T01:46:37-16b25d2-06c9756/freebsd-arm64-dmgk
2021-04-27T18:54:49-6397a11-0ae9c3b/darwin-amd64-10_14
2021-04-27T15:36:10-6397a11-f12cf76/freebsd-386-11_4
2021-04-20T16:44:18-a13dbf1-109d758/dragonfly-amd64
2021-04-07T17:38:53-e78b40c-b3064b6/darwin-arm64-11_0-toothrot
2021-04-06T20:22:15-c602466-f5efa5a/linux-s390x-ibm

(CC @findleyr @muirdm)

[bcmills]

The error comes from x/tools/internal/lsp/fake/sandbox.go:
https://cs.opensource.google/go/x/tools/+/master:internal/lsp/fake/sandbox.go;l=270;drc=384460091cd7916680a7c5252c587572c601197e

However, it appears that the directory for which the error occurs always refers to TestPostfixSnippetCompletion. I suspect a synchronization error in that test: perhaps some process or goroutine started by the test is continuing to write to the directory while the Sandbox is attempting to remove it.

[bcmills]

This is a release-blocker via #11811; it is causing frequent failures in the builders, especially for darwin/amd64 and darwin/arm64 (which are first-class ports).

(If the root cause isn't readily apparent, I suggest that we skip TestPostfixSnippetCompletion to reduce test-failure noise until someone has time to investigate.)

[gopherbot]

Change https://golang.org/cl/379735 mentions this issue: gopls/internal/regtest/completion: temporarily skip TestPostfixSnippetCompletion

[findleyr]

Thanks. I'll look into this with urgency.

I'd really llke to understand this failure mode, as this test is not doing a lot of IO: it doesn't even use any on-disk Go files, just a go.mod. All Go state exists only in in-memory buffers.

Looking at it briefly, I noticed that the test is repeatedly "opening" the same buffer, which I would expect to panic but doesn't (we should change that). However, I wouldn't expect that to cause this failure.

Disabling cleanup leaves the sandbox in the following state. None of this state should change following initial workspace load.

> find .
.
./work
./work/go.mod
./gopath
./gopath/pkg
./gopath/pkg/mod
./gopath/pkg/mod/cache
./gopath/pkg/mod/cache/lock
./proxy