Could not establish connection to workspace-unique-name

[IvoPereira]

Bug description

I am on Ubuntu 20.04.3 and I am not being able to use VS Code Desktop feature.

In VS Code Desktop I am receiving a "Could not establish connection to workspace-unique-name".

Steps to reproduce

• Open a Gitpod workspace

• Click "Gitpod" in the bottom left corner

• Click "Gitpod: Open in VSCode"

• Accept all prompts in VSCode side

• Choose Linux when the following prompt appears
  

• As soon as I choose it I receive the error:
  

Workspace affected

No response

Expected behavior

Gitpod and VS Code Desktop should work correctly.

Example repository

No response

Anything else?

The logs produced with remote login terminal open:

[23:52:17.222] Log Level: 2
[23:52:17.225] [email protected]
[23:52:17.225] linux x64
[23:52:17.239] SSH Resolver called for "ssh-remote+scarlet-piranha-5s8yxs48", attempt 1
[23:52:17.240] "remote.SSH.useLocalServer": false
[23:52:17.241] "remote.SSH.showLoginTerminal": true
[23:52:17.241] "remote.SSH.remotePlatform": {}
[23:52:17.241] "remote.SSH.path": undefined
[23:52:17.241] "remote.SSH.configFile": /tmp/gitpod_ssh_config-16037-kfOBdkiHw39I
[23:52:17.242] "remote.SSH.useFlock": true
[23:52:17.242] "remote.SSH.lockfilesInTmp": false
[23:52:17.242] "remote.SSH.localServerDownload": auto
[23:52:17.242] "remote.SSH.remoteServerListenOnSocket": false
[23:52:17.243] "remote.SSH.showLoginTerminal": true
[23:52:17.243] "remote.SSH.defaultExtensions": ["gitpod.gitpod-remote-ssh"]
[23:52:17.243] "remote.SSH.loglevel": 2
[23:52:17.244] SSH Resolver called for host: scarlet-piranha-5s8yxs48
[23:52:17.244] Setting up SSH remote "scarlet-piranha-5s8yxs48"
[23:52:17.250] Using commit id "ccbaa2d27e38e5afa3e5c21c1c7bef4657064247" and quality "stable" for server
[23:52:17.255] Install and start server if needed
[23:52:49.283] PATH: /home/user/bin:/home/user/.local/bin:/home/user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/user/go/bin
[23:52:49.284] Checking ssh with "ssh -V"
[23:52:49.292] > OpenSSH_8.2p1 Ubuntu-4ubuntu0.3, OpenSSL 1.1.1f  31 Mar 2020

[23:52:49.295] Using SSH config file "/tmp/gitpod_ssh_config-16037-kfOBdkiHw39I"
[23:52:49.295] Running script with connection command: ssh -T -D 41893 -o ConnectTimeout=15 -F '/tmp/gitpod_ssh_config-16037-kfOBdkiHw39I' 'scarlet-piranha-5s8yxs48' bash
[23:52:50.730] > Received disconnect from 127.0.0.1 port 46161:2: Too many authentication failures
> Disconnected from 127.0.0.1 port 46161
[23:52:50.730] Got some output, clearing connection timeout
[23:52:51.091] "install" terminal command done
[23:52:51.091] Install terminal quit with output: Disconnected from 127.0.0.1 port 46161
[23:52:51.092] Received install output: Disconnected from 127.0.0.1 port 46161
[23:52:51.092] Failed to parse remote port from server output
[23:52:51.093] Resolver error: Error: 
    at Function.Create (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.66.1/out/extension.js:1:429193)
    at Object.t.handleInstallOutput (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.66.1/out/extension.js:1:427836)
    at Object.t.tryInstall (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.66.1/out/extension.js:1:521703)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.66.1/out/extension.js:1:485356
    at async Object.t.withShowDetailsEvent (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.66.1/out/extension.js:1:488706)
    at async Object.t.resolve (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.66.1/out/extension.js:1:486435)
    at async /home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.66.1/out/extension.js:1:560057

If it matters, I have tried to run it now again and got this:

[03/12/2021, 12:32:09] open workspace window: vscode://gitpod.gitpod-desktop/workspace/REMOVED
[03/12/2021, 12:32:09] acquiring lock: gitpod.io
[03/12/2021, 12:32:10] acquired lock: gitpod.io
[03/12/2021, 12:32:10] fetching the local app from https://gitpod.io/static/bin/gitpod-local-companion-linux
[03/12/2021, 12:32:10] released lock: gitpod.io
[03/12/2021, 12:32:10] failed to access the local app: Error: workspace ssh tunnel not configured
[03/12/2021, 12:32:10] failed to open uri: Error: workspace ssh tunnel not configured

[akosyakov]

It seems that your token is invalid for some reasons. Could you try an advice from here:
#5712 (comment)

[IvoPereira]

For reference, tried restarting, opening a workspace and then choosing to open in VSCode Desktop and got this error:

[03/12/2021, 19:22:26] open workspace window: vscode://gitpod.gitpod-desktop/workspace/WORKSPACE
[03/12/2021, 19:22:26] acquiring lock: gitpod.io
[03/12/2021, 19:22:28] acquired lock: gitpod.io
[03/12/2021, 19:22:28] fetching the local app from https://gitpod.io/static/bin/gitpod-local-companion-linux
[03/12/2021, 19:22:29] installing the local app to /tmp/gitpod-local-companion-13291-3KIcbeacdY1e
[03/12/2021, 19:22:30] installing the local app: {
  "path": "/tmp/gitpod-local-companion-13291-3KIcbeacdY1e",
  "etag": "\"r3hc68ah2g1\""
}
[03/12/2021, 19:22:30] starting the local app with the config: {
  "gitpodHost": "https://gitpod.io",
  "configFile": "file:///tmp/gitpod_ssh_config-13291-AsWF3QijsqZ1",
  "apiPort": 43019
}
[03/12/2021, 19:22:30] failed to start the local app: Error: spawn ETXTBSY
[03/12/2021, 19:22:30] released lock: gitpod.io
[03/12/2021, 19:22:30] failed to open uri: Error: spawn ETXTBSY

Will see what other approaches I might be able to test from the comment you mentioned.

I don't however have a Mac Keychain as I am running in Linux.

Just to be sure, I don’t need to download the Local Companion app as it is now automatically installed and executed, is that right? https://www.gitpod.io/blog/local-app

Is it required to have a default ssh key in ~/.ssh/id_rsa?

[IvoPereira]

Some more notes.

I was checking again the terminal and observed the mentioned ssh command.

Tried to run it in a separate terminal windows and got the Too many authentication failures message indeed. And that probably might be explained because I have several ssh keys that are all probably being tried on so the server ends up rejecting it.

ssh -T -D 41351 -o ConnectTimeout=15 -F '/tmp/gitpod_ssh_config-id' 'username' bash
Received disconnect from 127.0.0.1 port 33955:2: Too many authentication failures
Disconnected from 127.0.0.1 port 33955

ssh -T -D 41351 -o IdentitiesOnly=yes -o ConnectTimeout=15 -F '/tmp/gitpod_ssh_config-id' 'username' bash
Gets me a list of environment variables (I imagine from the workspace?). I suppose if this is run in the correct context (opening workspace in VSCode Desktop) it would work.

I have the following in my ~/.ssh/config but it doesn’t seem to make a difference at all:

Host *
IdentitiesOnly=yes

---

Some more updates.

After the connection failing (for not having IdentitiesOnly), manually editing the ssh config file and adding IdentitiesOnly yes beeneath the IdentityFile does work!

However it is very cumbersome to do this every time I want to open a workspace in VSCode Desktop. Waiting for it to fail, open the config file, add the line and reload the window of the remote connection.

Any ideas @akosyakov?

[akosyakov]

Just to be sure, I don’t need to download the Local Companion app as it is now automatically installed and executed, is that right? https://www.gitpod.io/blog/local-app

Correct, it is bundled with Gitpod extension. ETXTBSY sounds like the extension cannot start the local companion for some reasons. But as far I understand it does not happen to you each time.

Is it required to have a default ssh key in ~/.ssh/id_rsa?

Not, the local companion generates own config and key.

What you explain makes sense https://superuser.com/questions/268776/how-do-i-configure-ssh-so-it-doesnt-try-all-the-identity-files-automatically if you have many identity files. I assume a fix is to add IdentitiesOnly as yes here:

gitpod/components/local-app/pkg/bastion/bastion.go

Line 154 in 5458a49

&ssh_config.KV{Key: "IdentityFile", Value: ws.SSHPrivateFN},

PRs are welcomed!

[loujaybee]

I'm going to re-open as I currently get this error.

ssh: connect to host gitpodio-gitpodpublicap-ezxgr5dx1i4.ssh.ws-eu82.gitpod.io port 22: Operation timed out
local-server-5> ssh child died, shutting down

[08:11:01.377] Resolver error: Error: The operation timed out

[mustard-mh]

@loujaybee could you share the full log if you face it again? 🙏 To make sure it's not related to desktop vscode settings like this issue microsoft/vscode-remote-release#6108 mentioned (as they have similar error message)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dexslender]

I get this error when I try to start the gitpod workspace

[akosyakov]

@dexslender Have you managed to resolve it? If not please follow https://www.gitpod.io/docs/help/troubleshooting#gitpod-logs-in-vs-code-browser-and-desktop

EDIT: I checked your analytics. It seems you still have troubles with the same workspace always. Could you try to check whether it works with another repository? I wonder whether you have changed something that it can affect SSH server running in the workspace.

EDIT: Also FYI we check overall system state and other users are successful connecting. SLO is over 99%

EDIT: from peeking in backend logs I see that SSH server is failing to start. You can see supervisor.log under /var/log/gitpod/supervisor.log. Maybe you can use also gp validate to troubleshoot it, try to pass gp validate --log=debug to see complete debug output from SSH server.

[dexslender]

@dexslender Have you managed to resolve it? If not please follow https://www.gitpod.io/docs/help/troubleshooting#gitpod-logs-in-vs-code-browser-and-desktop

EDIT: I checked your analytics. It seems you still have troubles with the same workspace always. Could you try to check whether it works with another repository? I wonder whether you have changed something that it can affect SSH server running in the workspace.

EDIT: Also FYI we check overall system state and other users are successful connecting. SLO is over 99%

EDIT: from peeking in backend logs I see that SSH server is failing to start. You can see supervisor.log under /var/log/gitpod/supervisor.log. Maybe you can use also gp validate to troubleshoot it, try to pass gp validate --log=debug to see complete debug output from SSH server.

The server display this logs when connecting via VSCode

[2023-10-02 16:38:14] DEBUG sshd flags args=[redacted:nested] file=ssh.go:146 func=handleConn serviceContext=map[service:supervisor version:commit-6d1887f6dbc81f1139f8adf3095efe8658d32bbe]
[2023-10-02 16:38:14]  INFO rate limiting SSH log output limit_kb_per_sec=50
[2023-10-02 16:38:14] DEBUG sshd started file=export.go:59 func=Debug serviceContext=map[service:supervisor version:commit-6d1887f6dbc81f1139f8adf3095efe8658d32bbe]
[2023-10-02 16:38:14] ERROR Connection from UNKNOWN port 65535 on UNKNOWN port 65535
[2023-10-02 16:38:14] ERROR Failed publickey for gitpod from UNKNOWN port 65535 ssh2: ECDSA SHA256:Sy2g/Byp6h2pHYE4RDlx7vvYRjDHQXhdE1Uq8IzYAUc
[2023-10-02 16:38:14] ERROR Connection closed by authenticating user gitpod UNKNOWN port 65535 [preauth]
[2023-10-02 16:38:14] ERROR SSH server stopped error=exit status 255

[akosyakov]

@dexslender Could you share a reproducible repo? Something is off public keys are missing in your workspace for gitpod user. It is somehow related to your setup, since we don't see it happening else there. I would start with the workspace image.

[akosyakov]

@dexslender We identified a regression with non gitpod images and will deploy a fix shortly.