New-SSHSession only sporadically working on 3.0 alpha

[tbhel]

Running the 3.0 alpha on Powershell 7.1.1 on Ubuntu 20.
When using New-SSHSession to connect to Cisco switches, it sometimes work and sometimes doesn't. Connecting to other Linux hosts seem to work better. See output:

PS /home/tbhel> New-SSHSession -ComputerName "10.1.2.20" -Credential $cred
New-SSHSession: Session operation has timed out
PS /home/tbhel> New-SSHSession -ComputerName "10.1.2.20" -Credential $cred

Server SSH Fingerprint
Do you want to trust the fingerprint 5:21:60:b5:fb:5d:7b:9c:c6:8f:4c:24:36:8e:f:bd
[] Y  [] N  [?] Help (default is "N"): y

SessionId  Host                                                               Connected
---------  ----                                                               ---------
    0      10.1.2.20                                                            True

PS /home/tbhel> Get-SSHSession | Remove-SSHSession
True
PS /home/tbhel> New-SSHSession -ComputerName "10.1.2.20" -Credential $cred

SessionId  Host                                                               Connected
---------  ----                                                               ---------
    0      10.1.2.20                                                            True

PS /home/tbhel> Get-SSHSession | Remove-SSHSession
True
PS /home/tbhel> New-SSHSession -ComputerName "10.1.2.20" -Credential $cred
New-SSHSession: Session operation has timed out

Connecting from a regular SSH client always work. Connecting from Posh-SSH 2.2 on a Windows host also always work.

[MVKozlov]

1. Check timeout
2. Try to compare hostkeys used in new alpha and other variants
   You can try to use host key storage from windows if you convert it from registry to json by Convert-SSHRegistryToJSonKnownHostStore

[darkoperator]

Any test equipment I could test against? I do not have Cisco kit to test and see what could be happening.

…

Sent from my iPhone

On Feb 4, 2021, at 6:27 AM, Max Kozlov ***@***.***> wrote:  Check timeout Try to compare hostkeys used in new alpha and other variants You can try to use host key storage from windows if you convert it from registry to json by Convert-SSHRegistryToJSonKnownHostStore — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[darkoperator]

Aslo can you provide information in terms of version of IOS and type of gear? Including key cipher and algorithm of the host and how ssh is setup in it.

…

Sent from my iPhone

On Feb 4, 2021, at 6:09 AM, tbhel ***@***.***> wrote:  Running the 3.0 alpha on Powershell 7.1.1 on Ubuntu 20. When using New-SSHSession to connect to Cisco switches, it sometimes work and sometimes doesn't. Connecting to other Linux hosts seem to work better. See output: PS /home/tbhel> New-SSHSession -ComputerName "10.1.2.20" -Credential $cred New-SSHSession: Session operation has timed out PS /home/tbhel> New-SSHSession -ComputerName "10.1.2.20" -Credential $cred Server SSH Fingerprint Do you want to trust the fingerprint 5:21:60:b5:fb:5d:7b:9c:c6:8f:4c:24:36:8e:f:bd [] Y [] N [?] Help (default is "N"): y SessionId Host Connected --------- ---- --------- 0 10.1.2.20 True PS /home/tbhel> Get-SSHSession | Remove-SSHSession True PS /home/tbhel> New-SSHSession -ComputerName "10.1.2.20" -Credential $cred SessionId Host Connected --------- ---- --------- 0 10.1.2.20 True PS /home/tbhel> Get-SSHSession | Remove-SSHSession True PS /home/tbhel> New-SSHSession -ComputerName "10.1.2.20" -Credential $cred New-SSHSession: Session operation has timed out Connecting from a regular SSH client always work. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[tbhel]

@MVKozlov Increasing timeout does not help. When it does work, it connects in <1 second.
Not sure exactly what you mean by compare hostkeys. The fingerprints seem to be correct.

@darkoperator I don't have any test equipment available unfortunately. I have tried to connect to several different switches, including:

• Catalyst 2960X-48FPS-L running IOS 15.2(2)E5
• Catalyst 4500X-32 running IOS-XE 03.10.00.E

The switches are running ssh version 2 with these algorithms:

• Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr
• MAC Algorithms:hmac-sha1

[darkoperator]

At some point Cisco changed something in their implementation of SSH based that this is an issue many report. The weird part is that it is intermittent

…

Sent from my iPhone

On Feb 4, 2021, at 6:55 AM, tbhel ***@***.***> wrote:  @MVKozlov Increasing timeout does not help. When it does work, it connects in <1 second. Not sure exactly what you mean by compare hostkeys. The fingerprints seem to be correct. @darkoperator I don't have any test equipment available unfortunately. I have tried to connect to several different switches, including: Catalyst 2960X-48FPS-L running IOS 15.2(2)E5 Catalyst 4500X-32 running IOS-XE 03.10.00.E The switches are running ssh version 2 with these algorithms: Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr MAC Algorithms:hmac-sha1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[darkoperator]

Any chance you can do a packet capture of a successful vs failed auth?

…

Sent from my iPhone

On Feb 4, 2021, at 6:59 AM, Carlos Perez ***@***.***> wrote: At some point Cisco changed something in their implementation of SSH based that this is an issue many report. The weird part is that it is intermittent Sent from my iPhone >> On Feb 4, 2021, at 6:55 AM, tbhel ***@***.***> wrote: >> >  > @MVKozlov Increasing timeout does not help. When it does work, it connects in <1 second. > Not sure exactly what you mean by compare hostkeys. The fingerprints seem to be correct. > > @darkoperator I don't have any test equipment available unfortunately. I have tried to connect to several different switches, including: > > Catalyst 2960X-48FPS-L running IOS 15.2(2)E5 > Catalyst 4500X-32 running IOS-XE 03.10.00.E > The switches are running ssh version 2 with these algorithms: > > Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr > MAC Algorithms:hmac-sha1 > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or unsubscribe.

[darkoperator]

Also wonder if debug ip ssh would show something

…

Sent from my iPhone

On Feb 4, 2021, at 6:59 AM, Carlos Perez ***@***.***> wrote: At some point Cisco changed something in their implementation of SSH based that this is an issue many report. The weird part is that it is intermittent Sent from my iPhone >> On Feb 4, 2021, at 6:55 AM, tbhel ***@***.***> wrote: >> >  > @MVKozlov Increasing timeout does not help. When it does work, it connects in <1 second. > Not sure exactly what you mean by compare hostkeys. The fingerprints seem to be correct. > > @darkoperator I don't have any test equipment available unfortunately. I have tried to connect to several different switches, including: > > Catalyst 2960X-48FPS-L running IOS 15.2(2)E5 > Catalyst 4500X-32 running IOS-XE 03.10.00.E > The switches are running ssh version 2 with these algorithms: > > Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr > MAC Algorithms:hmac-sha1 > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or unsubscribe.

[tbhel]

Yeah I saw the other issues with very similar problems. The weird thing is that Posh-SSH 2.2 works completely fine.

I may have to get back to you on a packet capture, but I have the output of debug ip ssh here. Both the working and non-working are from the same switch, just moments between attempts.

Working:

Feb  4 11:06:45.639: SSH1: starting SSH control process
Feb  4 11:06:45.639: SSH1: sent protocol version id SSH-2.0-Cisco-1.25
Feb  4 11:06:45.642: SSH1: protocol version id is - SSH-2.0-Renci.SshNet.SshClient.0.0.1
Feb  4 11:06:45.642: SSH2 1: kexinit sent: encryption algo = aes128-ctr,aes192-ctr,aes256-ctr
Feb  4 11:06:45.642: SSH2 1: kexinit sent: mac algo = hmac-sha1
Feb  4 11:06:45.642: SSH2 1: send:packet of  length 264 (length also includes padlen of 11)
Feb  4 11:06:45.642: SSH2 1: SSH2_MSG_KEXINIT sent
Feb  4 11:06:45.649: SSH2 1: ssh_receive: 536 bytes received
Feb  4 11:06:45.649: SSH2 1: input: total packet length of 1048 bytes
Feb  4 11:06:45.649: SSH2 1: partial packet length(block size)8 bytes,needed 1040 bytes,
               maclen 0
Feb  4 11:06:45.649: SSH2 1: ssh_receive: 512 bytes received
Feb  4 11:06:45.649: SSH2 1: partial packet length(block size)8 bytes,needed 1040 bytes,
               maclen 0
Feb  4 11:06:45.649: SSH2 1: input: padlength 10 bytes
Feb  4 11:06:45.649: SSH2 1: SSH2_MSG_KEXINIT received
Feb  4 11:06:45.649: SSH2 1: kex: client->server enc:aes256-ctr mac:hmac-sha1
Feb  4 11:06:45.649: SSH2 1: kex: server->client enc:aes256-ctr mac:hmac-sha1
Feb  4 11:06:45.649: SSH2 1: Using kex_algo = diffie-hellman-group-exchange-sha1
Feb  4 11:06:45.649: SSH2 1: ssh_receive: 32 bytes received
Feb  4 11:06:45.649: SSH2 1: input: total packet length of 32 bytes
Feb  4 11:06:45.649: SSH2 1: partial packet length(block size)8 bytes,needed 24 bytes,
               maclen 0
Feb  4 11:06:45.649: SSH2 1: input: padlength 14 bytes
Feb  4 11:06:45.649: SSH2 1: SSH2_MSG_KEX_DH_GEX_REQUEST received
Feb  4 11:06:45.649: SSH2 1: Range sent by client is - 1024 < 1024 < 8192
Feb  4 11:06:45.649: SSH2 1:  Modulus size established : 1024 bits
Feb  4 11:06:45.649: SSH2 1: send:packet of  length 152 (length also includes padlen of 8)
Feb  4 11:06:45.688: SSH2 1: expecting SSH2_MSG_KEX_DH_GEX_INIT
Feb  4 11:06:45.702: SSH2 1: ssh_receive: 152 bytes received
Feb  4 11:06:45.702: SSH2 1: input: total packet length of 152 bytes
Feb  4 11:06:45.702: SSH2 1: partial packet length(block size)8 bytes,needed 144 bytes,
               maclen 0
Feb  4 11:06:45.702: SSH2 1: input: padlength 13 bytes
Feb  4 11:06:45.702: SSH2 1: SSH2_MSG_KEXDH_INIT received
Feb  4 11:06:46.058: SSH2 1: signature length 271
Feb  4 11:06:46.058: SSH2 1: send:packet of  length 704 (length also includes padlen of 8)
Feb  4 11:06:46.058: SSH2: kex_derive_keys complete
Feb  4 11:06:46.058: SSH2 1: send:packet of  length 16 (length also includes padlen of 10)
Feb  4 11:06:46.058: SSH2 1: newkeys: mode 1
Feb  4 11:06:46.058: SSH2 1: SSH2_MSG_NEWKEYS sent
Feb  4 11:06:46.058: SSH2 1: waiting for SSH2_MSG_NEWKEYS
Feb  4 11:06:46.079: SSH2 1: ssh_receive: 16 bytes received
Feb  4 11:06:46.079: SSH2 1: input: total packet length of 16 bytes
Feb  4 11:06:46.079: SSH2 1: partial packet length(block size)8 bytes,needed 8 bytes,
               maclen 0
Feb  4 11:06:46.079: SSH2 1: input: padlength 10 bytes
Feb  4 11:06:46.079: SSH2 1: newkeys: mode 0
Feb  4 11:06:46.079: SSH2 1: SSH2_MSG_NEWKEYS received
Feb  4 11:06:46.079: SSH2 1: ssh_receive: 68 bytes received
Feb  4 11:06:46.079: SSH2 1: input: total packet length of 48 bytes
Feb  4 11:06:46.079: SSH2 1: partial packet length(block size)16 bytes,needed 32 bytes,
               maclen 20
Feb  4 11:06:46.079: SSH2 1: MAC compared for #4 :ok
Feb  4 11:06:46.079: SSH2 1: input: padlength 26 bytes
Feb  4 11:06:46.079: SSH2 1: send:packet of  length 32 (length also includes padlen of 10)
Feb  4 11:06:46.079: SSH2 1: computed MAC for sequence no.#4 type 6
Feb  4 11:06:46.083: SSH2 1: ssh_receive: 84 bytes received
Feb  4 11:06:46.083: SSH2 1: input: total packet length of 64 bytes
Feb  4 11:06:46.083: SSH2 1: partial packet length(block size)16 bytes,needed 48 bytes,
               maclen 20
Feb  4 11:06:46.083: SSH2 1: MAC compared for #5 :ok
Feb  4 11:06:46.083: SSH2 1: input: padlength 23 bytes
Feb  4 11:06:46.083: SSH2 1: Using method = none
Feb  4 11:06:46.083: SSH2 1: Authentications that can continue = publickey,keyboard-interactive,password
Feb  4 11:06:46.083: SSH2 1: send:packet of  length 64 (length also includes padlen of 14)
Feb  4 11:06:46.083: SSH2 1: computed MAC for sequence no.#5 type 51
Feb  4 11:06:46.086: SSH2 1: ssh_receive: 100 bytes received
Feb  4 11:06:46.086: SSH2 1: input: total packet length of 80 bytes
Feb  4 11:06:46.086: SSH2 1: partial packet length(block size)16 bytes,needed 64 bytes,
               maclen 20
Feb  4 11:06:46.086: SSH2 1: MAC compared for #6 :ok
Feb  4 11:06:46.086: SSH2 1: input: padlength 19 bytes
Feb  4 11:06:46.086: SSH2 1: Using method = password
Feb  4 11:06:46.093: SSH2 1: send:packet of  length 16 (length also includes padlen of 10)
Feb  4 11:06:46.093: SSH2 1: computed MAC for sequence no.#6 type 52
Feb  4 11:06:46.093: SSH2 1: authentication successful for user

Non-working:

Feb  4 11:09:44.099: SSH1: starting SSH control process
Feb  4 11:09:44.099: SSH1: sent protocol version id SSH-2.0-Cisco-1.25
Feb  4 11:09:54.114: SSH1: protocol version id is - SSH-2.0-Renci.SshNet.SshClient.0.0.1
Feb  4 11:09:54.114: SSH2 1: kexinit sent: encryption algo = aes128-ctr,aes192-ctr,aes256-ctr
Feb  4 11:09:54.114: SSH2 1: kexinit sent: mac algo = hmac-sha1
Feb  4 11:09:54.114: SSH2 1: send:packet of  length 264 (length also includes padlen of 11)
Feb  4 11:09:54.114: SSH2 1: send:packet of  length 72 (length also includes padlen of 7)
Feb  4 11:09:54.215: SSH1: Session disconnected - error 0x00

[tbhel]

Managed to grab packet captures. The capture of the working connection obviously continues beyond what is included, but I did not see any point to including it in the dump.
captures.zip

[darkoperator]

Thanks, will pass this to a ticket to SSH.Net project since it was working before the 2020 update

…

Sent from my iPhone

On Feb 4, 2021, at 8:13 AM, tbhel ***@***.***> wrote:  Managed to grab packet captures. The capture of the working connection obviously continues beyond what is included, but I did not see any point to including it in the dump. captures.zip — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[MVKozlov]

@tbhel, I mean HostkeyName - old SSH.Net version use rsa by default, new -ssh-ed25519
If you convert old keys from registry into new json format or use the registry keystore, the 3.0 version will try to use old rsa keys which it already know as does 2.0

[tbhel]

Just wanted to let you know that the problem still remains in version 3.0.8 unfortunately.

[darkoperator]

Ok, was not expecting a fix for this issue in the releases. As to why is it failing on some of your linux host only thing I can recommend is to enable ssh debugging and see if it sheds some light as to why. Herr are some steps you can follow to do it https://support.cpanel.net/hc/en-us/articles/4402832130967-How-to-enable-debug-logging-for-SSHD-SSH-by-editing-the-etc-ssh-sshd-config