Cisco Router frequent New-SSHSession : Session operation has timed out

[RGlintmeijer]

Hi,

While connecting to a Cisco Router with the following command:

New-SSHSession -ComputerName $Device -ConnectionTimeout 15 -AcceptKey -Credential $credential -Verbose -ErrorAction Stop

Sometimes it succeeds and the connection is working, but most of the time the connection fails with:

New-SSHSession : Session operation has timed out
At D:\Scripts\Get-DeviceConfiguration.ps1:104 char:16
+                 $session = New-SSHSession -ComputerName $Device -ConnectionTimeout 15 ...
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationTimeout: (Renci.SshNet.SshClient:SshClient) [New-SSHSession], SshOperationTimeoutException
    + FullyQualifiedErrorId : SSH.NewSshSession

When i monitor on the Cisco Router with "debug ip ssh", i get the following on success:

Jul 28 13:57:05.701: SSH1: starting SSH control process
Jul 28 13:57:05.701: SSH1: sent protocol version id SSH-2.0-Cisco-1.25
Jul 28 13:57:05.701: SSH1: protocol version id is - SSH-2.0-Renci.SshNet.SshClient.0.0.1
Jul 28 13:57:05.702: SSH2 1: kexinit sent: kex algo = diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
Jul 28 13:57:05.702: SSH2 1: Server certificate trustpoint not found. Skipping hostkey algo = x509v3-ssh-rsa
Jul 28 13:57:05.702: SSH2 1: kexinit sent: hostkey algo = ssh-rsa
Jul 28 13:57:05.702: SSH2 1: kexinit sent: encryption algo = aes128-ctr,aes192-ctr,aes256-ctr
Jul 28 13:57:05.702: SSH2 1: kexinit sent: mac algo = hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
Jul 28 13:57:05.702: SSH2 1: send:packet of  length 312 (length also includes padlen of 4)
Jul 28 13:57:05.702: SSH2 1: SSH2_MSG_KEXINIT sent
etc...

And when the connection fails:

Jul 28 14:45:08.975: SSH3: starting SSH control process
Jul 28 14:45:08.975: SSH3: sent protocol version id SSH-2.0-Cisco-1.25
Jul 28 14:45:23.977: SSH3: protocol version id is - SSH-2.0-Renci.SshNet.SshClient.0.0.1
Jul 28 14:45:23.977: SSH2 3: kexinit sent: kex algo = diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
Jul 28 14:45:23.977: SSH2 3: Server certificate trustpoint not found. Skipping hostkey algo = x509v3-ssh-rsa
Jul 28 14:45:23.977: SSH2 3: kexinit sent: hostkey algo = ssh-rsa
Jul 28 14:45:23.977: SSH2 3: kexinit sent: encryption algo = aes128-ctr,aes192-ctr,aes256-ctr
Jul 28 14:45:23.977: SSH2 3: kexinit sent: mac algo = hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
Jul 28 14:45:23.978: SSH2 3: send:packet of  length 312 (length also includes padlen of 4)
Jul 28 14:45:23.978: SSH2 3: send:packet of  length 72 (length also includes padlen of 7)
Jul 28 14:45:24.078: SSH3: Session disconnected - error 0x00

You can see that when the connection is successfull, everything is being processed within the same second. But when the connection fails, it takes the Timeout (in this case 15 seconds) before it logs the protocol version id of the client and then disconnects.

The Verbose output in Powershell on success is:

VERBOSE: Using SSH Username and Password authentication for connection.
VERBOSE: Fingerprint for ***************: *:**:**:**:*:*:**:**:**:**:**:**:**:**:**:**
VERBOSE: Fingerprint matched trusted fingerprint for host ***************

And on failure:

VERBOSE: Using SSH Username and Password authentication for connection.

I'm using Posh-SSH version 2.3-beta1 on Windows Server 2012 R2.

[rickyrickuk]

I get the same issue, only connects once every 10 or so times.

[darkoperator]

Sadly not much I can do since I don't control the base library

…

Sent from my iPhone

On Dec 9, 2020, at 7:26 AM, rickyrickuk ***@***.***> wrote:  I get the same issue, only connects once every 10 or so times. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[darkoperator]

Hmm the error is happening on the Cisco device, a quick google search there are multiple cases of this with multiple clients. Most of the error seems are SSH RSA key pairs on the device where the hostname changed after generation. The weird part is that is works some times and no others. I don’t think it is a problem with the module code it self and could be a Cisco issue since it is only being reported on those devices.

[Patrick92x]

@RGlintmeijer @rickyrickuk Try my workaround in #354. Seems like something has changed in the SSH.NET build.

[darkoperator]

Seems the author of the library is making some updates and fixes most I can do is release betas with this updates sshnet/SSH.NET@dc9c637 <sshnet/SSH.NET@dc9c637> to see on what point the issue is addressed

[rickyrickuk]

@RGlintmeijer @rickyrickuk Try my workaround in #354. Seems like something has changed in the SSH.NET build.

Hi Patrick92x

Will give this a try today and let you know my findings.

I installed Posh-SSH by downloading and extracting the NuGet package into the Powershell modules folder. If I replace the .dll file in this folder, do I need to refresh the module somehow to test?

Thanks, Richard

[darkoperator]

Import-module posh-ssh -force -verbose

…

On Dec 29, 2020, at 5:12 AM, rickyrickuk ***@***.***> wrote: @RGlintmeijer <https://github.com/RGlintmeijer> @rickyrickuk <https://github.com/rickyrickuk> Try my workaround in #354 <#354>. Seems like something has changed in the SSH.NET build. Hi Patrick92x Will give this a try today and let you know my findings. I installed Posh-SSH by downloading and extracting the NuGet package into the Powershell modules folder. If I replace the .dll file in this folder, do I need to refresh the module somehow to test? Thanks, Richard — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#340 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD7IHTS53JCI7IHB46STDTSXGMRLANCNFSM4PKVCL4Q>.

[Patrick92x]

I installed Posh-SSH by downloading and extracting the NuGet package into the Powershell modules folder. If I replace the .dll file in this folder, do I need to refresh the module somehow to test?

The thing i did is install version 2.2 at the same time by using the following command:
Install-Module posh-ssh -RequiredVersion '2.2' -Force

In the modules folder i've replaced the .dll in the 2.3 folder with the one of version 2.2 (and renaming it with by adding "PS" to the name).

After that i've tested by restarting powershell before running PoSH-SSH commands again. But i guess forcing the module to be imported again like @darkoperator says can work too.

If you want to uninstall version 2.2 afterwards just run:
Uninstall-Module posh-ssh -RequiredVersion '2.2'

[rickyrickuk]

I installed Posh-SSH by downloading and extracting the NuGet package into the Powershell modules folder. If I replace the .dll file in this folder, do I need to refresh the module somehow to test?

The thing i did is install version 2.2 at the same time by using the following command:
Install-Module posh-ssh -RequiredVersion '2.2' -Force

In the modules folder i've replaced the .dll in the 2.3 folder with the one of version 2.2 (and renaming it with by adding "PS" to the name).

After that i've tested by restarting powershell before running PoSH-SSH commands again. But i guess forcing the module to be imported again like @darkoperator says can work too.

If you want to uninstall version 2.2 afterwards just run:
Uninstall-Module posh-ssh -RequiredVersion '2.2'

Just downloaded v2.3 from https://github.com/darkoperator/Posh-SSH/tree/v2.3 and imported using darkoperators 'Import-module posh-ssh -force -verbose', got the following error but strangely now my cisco devices are connected each time (tried 10 times - before the import it was failing most of the time).

.dll file is 413KB in size in the assembly folder, is this the version you have that works @Patrick92x ?
Server is Win2016 std.

result of import-module:
PS C:\WINDOWS\system32> import-module posh-ssh -force -Verbose
VERBOSE: Loading module from path 'C:\Program Files\WindowsPowerShell\Modules\posh-ssh\posh-ssh.psd1'.
VERBOSE: Loading 'Assembly' from path 'C:\Program Files\WindowsPowerShell\Modules\posh-ssh\Assembly\Renci.SshNetPS.dll'.
VERBOSE: Loading 'Assembly' from path 'C:\Program Files\WindowsPowerShell\Modules\posh-ssh\Assembly\Renci.SshNetPS.dll'.
VERBOSE: Loading 'FormatsToProcess' from path 'C:\Program Files\WindowsPowerShell\Modules\posh-ssh\Format\SSHSession.Format.ps1xml'.
VERBOSE: Loading 'FormatsToProcess' from path 'C:\Program Files\WindowsPowerShell\Modules\posh-ssh\Format\SFTPSession.Format.ps1xml'.
VERBOSE: Loading 'FormatsToProcess' from path 'C:\Program Files\WindowsPowerShell\Modules\posh-ssh\Format\Renci.SshNet.SshCommand.Format.ps1xml'.
VERBOSE: Loading 'FormatsToProcess' from path 'C:\Program Files\WindowsPowerShell\Modules\posh-ssh\Format\Renci.SshNet.Sftp.SftpFile.Format.ps1xml'.
import-module : Could not load file or assembly 'file:///C:\Program Files\WindowsPowerShell\Modules\posh-ssh\Assembly\Renci.SshNetPS.dll' or one of its dependencies. The system cannot
find the file specified.
At line:1 char:1

• import-module posh-ssh -force -Verbose

•   + CategoryInfo          : InvalidOperation: (:) [Import-Module], FileNotFoundException
    + FullyQualifiedErrorId : FormatXmlUpdateException,Microsoft.PowerShell.Commands.ImportModuleCommand
  
  

[Patrick92x]

The .dll of version 2.2 that I use is 413KB. The .dll in version 2.3 was 758KB.

I don't get the errors that you describe using the verbose option. I'm on Windows Server 2019.

[rickyrickuk]

The .dll of version 2.2 that I use is 413KB. The .dll in version 2.3 was 758KB.

I don't get the errors that you describe using the verbose option. I'm on Windows Server 2019.

That makes sense then why mine is working and is in line with your findings, just odd that the .dll under v2.3 on the website looks like the v2.2 one, unless I am looking in the wrong place ! @darkoperator ?

[darkoperator]

Did you unprotected the zip file? Any zip download with DLLs and Exe are marked as dangerous and not allowed to execute unless you right click on it, properties and unprotrect

…

Sent from my iPhone

On Dec 29, 2020, at 9:31 AM, rickyrickuk ***@***.***> wrote:  The .dll of version 2.2 that I use is 413KB. The .dll in version 2.3 was 758KB. I don't get the errors that you describe using the verbose option. I'm on Windows Server 2019. That makes sense then why mine is working and is in line with your findings, just odd that the .dll under v2.3 on the website looks like the v2.2 one, unless I am looking in the wrong place ! @darkoperator ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[darkoperator]

This is why I recommend the gallery, manual install has so many variables the same files in zip file are the ones pushed to the gallery. Each versionhas more fixes and code, so newer versions will be different size. If different from the gallery then No clue why there is a difference

…

Sent from my iPhone

On Dec 29, 2020, at 9:31 AM, rickyrickuk ***@***.***> wrote:  The .dll of version 2.2 that I use is 413KB. The .dll in version 2.3 was 758KB. I don't get the errors that you describe using the verbose option. I'm on Windows Server 2019. That makes sense then why mine is working and is in line with your findings, just odd that the .dll under v2.3 on the website looks like the v2.2 one, unless I am looking in the wrong place ! @darkoperator ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[mkanet]

I also have this issue using the latest Posh-SSH 3.0.1.

The only way I can avoid it is to completely kill my Powershell console, then run the command again. Has anyone figured out how to avoid this issue reliably? It's the main reason why I can't use Posh-SSH reliably with our cisco routers.

I am not sure how to even troubleshoot something like this. Surely, there has to be a solution other than using something other than Posh-SSH.