SSH provisioner incompatible with Fedora 33, 34, 35 and CentOS Stream 9; also RHEL 8 with FIPS

[Ramblurr]

Fedora 33 updated the system-wide crypto policy to disallow SHA-1 hashes in signatures. This is a good thing, especially since OpenSSH follows suit as of version 8.8

The upstream golang issue is golang/go#39885

Terraform Version

Terraform v1.1.0
on linux_amd64

Terraform Configuration Files

resource "null_resource" "my_config" {

  provisioner "file" {
    content = file("${path.module}/test.config")
    destination = "/home/fedora/test.config"
  }

  connection {
    type     = "ssh"
    host     = "myhost.com"
    user     = "fedora"
    agent    = "true"
    timeout = "15s"
  }
}

Debug Output

...
2021-12-10T15:00:37.014+0100 [INFO]  Starting apply for null_resource.my_config
2021-12-10T15:00:37.014+0100 [DEBUG] null_resource.my_config: applying the planned Create change
null_resource.my_config: Provisioning with 'file'...
2021-12-10T15:00:37.047+0100 [DEBUG] Connecting to myhost.com:22 for SSH
2021-12-10T15:00:37.050+0100 [DEBUG] Connection established. Handshaking for user fedora
2021-12-10T15:00:37.094+0100 [WARN]  SSH authentication failed (fedora@myhost.com:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-12-10T15:00:37.094+0100 [WARN]  retryable error: SSH authentication failed (fedora@myhost.com:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
...

Expected Behavior

The file should be copied via ssh.

Actual Behavior

The file was not copied due to an ssh connection error

│ Error: file provisioner error
│ 
│   with null_resource.my_config,
│   on apps.tf line 106, in resource "null_resource" "my_config":
│  106:   provisioner "file" {
│ 
│ timeout - last error: SSH authentication failed (fedora@myhost.com:22): ssh: handshake failed:
│ ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
╵
Releasing state lock. This may take a few moments...

Steps to Reproduce

1. Deploy a vanilla Fedora 33, 34, 35 or CentOS 9 Stream box OR a RHEL8 box with update-crypto-policies --set FIPS
2. use the null_resource + file provisioner with ssh settings to copy a file to the host

Additional Context

The workaround is to enable the insecure ssh-rsa key type:

# place this file on the server at /etc/ssh/sshd_config.d/10-insecure-rsa-keysig.conf
PubkeyAcceptedKeyTypes=+ssh-rsa

then systemctl reload sshd

References

There are a host of related issues about this across the go ecosystem:

Related issues:

• hashicorp/packer: ssh handshake failure after changing system-wide crypto policy to FIPS packer#8609
• golang/go: x/crypto/ssh: publicKeyCallback cannot handshake using ssh-rsa keys signed using the ssh-rsa-sha2-256 algorithm golang/go#39885
• golang/go:x/crypto/ssh: cannot sign certificate with different algorithm golang/go#36261
• golang/go:x/crypto/ssh: support RSA SHA-2 host key signatures golang/go#37278

Unless hashicorp wants to maintain a forked version of the golang lib, then there's not much to do but wait. I've opened this ticket to track the issue.

[alisdair]

Thanks @Ramblurr for this detailed issue report! We're tracking this at #27768 also, and once this is resolved upstream we can fix the issue in Terraform too.

[Ramblurr]

Ah, sorry for the duplicate. I did search beforehand, but not well enough it seems :(

I was surprised that it hadn't been reported given F33's age.

Feel free to close this one if you want.

[rogierm]

This is also the case with Amazon Linux 2022 as that is derived from Fedora.

2022/01/14 22:58:26 packer-builder-amazon-ebs plugin: Using host value: 10.X.X.X
2022/01/14 22:58:26 packer-builder-amazon-ebs plugin: [INFO] Attempting SSH connection to 10.X.X.X:22...
2022/01/14 22:58:26 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/01/14 22:58:26 packer-builder-amazon-ebs plugin: [DEBUG] handshaking with SSH
2022/01/14 22:58:27 packer-builder-amazon-ebs plugin: [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2022/01/14 22:58:27 packer-builder-amazon-ebs plugin: [DEBUG] Detected authentication error. Increasing handshake attempts.

[Shilpi-Das1]

Is there any update on this issue

[crw]

Per #30134 (comment), it looks like the upstream issue is not yet resolved.