Automatically retry download failures

[bobbrow]

Recently we've been hitting the following problem with our CI on macOS for the CMake Tools extension:

Found /Users/runner/work/vscode-cmake-tools/vscode-cmake-tools/.vscode-test/vscode-darwin-1.68.1. Skipping download.
events.js:[377](https://github.com/microsoft/vscode-cmake-tools/runs/6974144243?check_suite_focus=true#step:4:378)
      throw er; // Unhandled 'error' event
      ^
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketErrorListener (_http_client.js:475:9)
    at TLSSocket.emit (events.js:400:28)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  errno: -54,
  code: 'ECONNRESET',
  syscall: 'read'
}
error Command failed with exit code 1.

It's not consistent, but it is happening enough to be disruptive. I traced it to this line which is not wrapped with try/catch:

vscode-test/lib/runTest.ts

Line 118 in cabeacc

options.vscodeExecutablePath = await downloadAndUnzipVSCode(options);

It seems a related issue was opened a few years ago to make this API catchable: #49 But if I try to wrap my runTests call in a try/catch, it still doesn't work. Reading around the Internet, I also came across this issue which seems related: nodejs/node#27916

Uncaught Error Error: read ECONNRESET
    at __node_internal_captureLargerStackTrace (node:internal/errors:465:5)
    at __node_internal_errnoException (node:internal/errors:595:12)
    at onStreamRead (node:internal/stream_base_commons:217:20)
    at callbackTrampoline (node:internal/async_hooks:130:17)
    --- TickObject ---
    at init (node:internal/inspector_async_hook:25:19)
    at emitInitNative (node:internal/async_hooks:201:43)
    at emitInitScript (node:internal/async_hooks:506:3)
    at nextTick (node:internal/process/task_queues:133:5)
    at onDestroy (node:internal/streams/destroy:96:15)
    at Socket._destroy (node:net:714:5)
    at _destroy (node:internal/streams/destroy:102:25)
    at destroy (node:internal/streams/destroy:64:5)
    at onStreamRead (node:internal/stream_base_commons:217:12)
    at callbackTrampoline (node:internal/async_hooks:130:17)
    --- TLSWRAP ---
    at init (node:internal/inspector_async_hook:25:19)
    at emitInitNative (node:internal/async_hooks:201:43)
    at TLSSocket._wrapHandle (node:_tls_wrap:618:24)
    at TLSSocket (node:_tls_wrap:517:18)
    at connect (node:_tls_wrap:1625:19)
    at createConnection (node:https:143:22)
    at createSocket (node:_http_agent:340:26)
    at addRequest (node:_http_agent:291:10)
    at ClientRequest (node:_http_client:311:16)
    at request (node:https:353:10)
    at get (node:https:387:15)
    at <anonymous> (/Users/bob/git/vscode-cmake-tools/node_modules/vscode-test/out/request.js:18:15)
    at getJSON (/Users/bob/git/vscode-cmake-tools/node_modules/vscode-test/out/request.js:17:12)
    at fetchLatestStableVersion (/Users/bob/git/vscode-cmake-tools/node_modules/vscode-test/out/download.js:21:36)
    at download (/Users/bob/git/vscode-cmake-tools/node_modules/vscode-test/out/download.js:156:25)
    at downloadAndUnzipVSCode (/Users/bob/git/vscode-cmake-tools/node_modules/vscode-test/out/download.js:218:18)
>    at runTests (/Users/bob/git/vscode-cmake-tools/node_modules/vscode-test/out/runTest.js:17:57)
    at main (/Users/bob/git/vscode-cmake-tools/test/smoke/badProject/runTest.ts:17:27)
    at <anonymous> (/Users/bob/git/vscode-cmake-tools/test/smoke/badProject/runTest.ts:29:6)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Module._load (node:internal/modules/cjs/loader:822:12)
    at executeUserEntryPoint (node:internal/modules/run_main:77:12)
    at <anonymous> (node:internal/main/run_main_module:17:47)

To Reproduce

Run the following commands on an M1 mac. (requires yarn v1)

mkdir repro
cd repro
git clone https://github.com/microsoft/vscode-cmake-tools.git
cd vscode-cmake-tools
yarn smokeTests

[connor4312]

Thanks for the issue, I noticed the repo is on quite an old version of this module, and there have since been fixed in a similar vein (e.g. cabeacc a couple weeks ago). Please update to the latest @vscode/test-electron and let me know if you continue to run into errors.

[bobbrow]

Oh, you renamed the module. I guess I missed that. I had checked I had the latest version in VS Code, but I guess I had the wrong module. 😄

I'll give that a shot.

[bobbrow]

It doesn't seem to be fixed with the new package. After the error message below, the terminal just sits and I have to break out of the test with CTL+C.

bob@m1mac vscode-cmake-tools % yarn smokeTests
yarn run v1.22.18
warning cmake-tools@1.10.0: The engine "vscode" appears to be invalid.
$ yarn run pretest && node ./out/test/smoke/badProject/runTest.js && node ./out/test/smoke/goodProject/runTest.js && node ./out/test/smoke/noCtest/runTest.js
warning cmake-tools@1.10.0: The engine "vscode" appears to be invalid.
$ tsc -p test.tsconfig.json
Downloading VS Code 1.68.1 from https://update.code.visualstudio.com/1.68.1/darwin-arm64/stable
Downloading VS Code [==============================] 100%Downloaded VS Code 1.68.1 into /Users/bob/git/vscode-cmake-tools/.vscode-test/vscode-darwin-arm64-1.68.1
2022-06-20 16:32:51.406 Electron[75830:19397598] +[NSXPCSharedListener endpointForReply:withListenerName:]: an error occurred while attempting to obtain endpoint for listener 'ClientCallsAuxiliary': Connection interrupted

[bobbrow]

I submitted a PR with the changes anyway to see if it makes a difference in our CI. Could you let me know if I did anything wrong with the upgrade?
microsoft/vscode-cmake-tools#2610

[connor4312]

Looks good. Note that while you still may get ECONNRESET's (since those are ultimately network errors) they should now be properly catchable and retryable

[bobbrow]

Thanks for looking!
Should the test framework do the retry so that everyone can benefit from it?