Packer v1.8.0 can't SSH to AWS Ubuntu 22.04

[marcosdiez]

packer version:
Packer v1.8.0 (running on ubuntu-20.04 AMD64)

command: PACKER_LOG=1 packer build -on-error=ask ubuntu-22.04.json
(ubuntu-22.04.json is in the bottom of this github issue)

trying to use packer with Ubuntu 22.04 AWS AMI, which was released today, does not work. It can't ssh:

==> edr-ubuntu-22.04-: Waiting for SSH to become available...
2022/04/21 15:37:22 packer-builder-amazon-ebs plugin: [INFO] Waiting for SSH, up to timeout: 5m0s
2022/04/21 15:37:22 packer-builder-amazon-ebs plugin: Using host value: 172.19.17.149
2022/04/21 15:37:37 packer-builder-amazon-ebs plugin: [DEBUG] TCP connection to SSH ip/port failed: dial tcp 172.19.17.149:22: i/o timeout
2022/04/21 15:37:42 packer-builder-amazon-ebs plugin: Using host value: 172.19.17.149
2022/04/21 15:37:42 packer-builder-amazon-ebs plugin: [INFO] Attempting SSH connection to 172.19.17.149:22...
2022/04/21 15:37:42 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/04/21 15:37:43 packer-builder-amazon-ebs plugin: [DEBUG] handshaking with SSH
2022/04/21 15:37:44 packer-builder-amazon-ebs plugin: [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2022/04/21 15:37:44 packer-builder-amazon-ebs plugin: [DEBUG] Detected authentication error. Increasing handshake attempts.

from my terminal it works:

mdiez@batman:~$ ssh -v ubuntu@172.19.17.149 -i ~/.ssh/id_rsa_terraform
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/mdiez/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/tmp/xmdiez-ssh-ubuntu@172.19.17.149:22.sock" does not exist
debug1: Connecting to 172.19.17.149 [172.19.17.149] port 22.
debug1: Connection established.
debug1: identity file /home/mdiez/.ssh/id_rsa_terraform type 0
debug1: identity file /home/mdiez/.ssh/id_rsa_terraform-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3
debug1: match: OpenSSH_8.9p1 Ubuntu-3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.19.17.149:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:RuS0DVs/kq2OiVP/4bCe6YDdzd7Zr16Zyh/GlaQbr44
debug1: Host '172.19.17.149' is known and matches the ECDSA host key.
debug1: Found key in /home/mdiez/.ssh/known_hosts:1824
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/mdiez/.ssh/id_rsa RSA SHA256:bPGRz/lG4uVKTPZxZStRlksQqjhtzg205sax/VoQaNM agent
debug1: Will attempt key: /home/mdiez/.ssh/id_rsa_terraform RSA SHA256:1b9TeR6gKb6YXQSyMUhftAhle6u4M1cEWtb6Mg4JwRU explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/mdiez/.ssh/id_rsa RSA SHA256:bPGRz/lG4uVKTPZxZStRlksQqjhtzg205sax/VoQaNM agent
debug1: Authentications that can continue: publickey
debug1: Offering public key: /home/mdiez/.ssh/id_rsa_terraform RSA SHA256:1b9TeR6gKb6YXQSyMUhftAhle6u4M1cEWtb6Mg4JwRU explicit
debug1: Server accepts key: /home/mdiez/.ssh/id_rsa_terraform RSA SHA256:1b9TeR6gKb6YXQSyMUhftAhle6u4M1cEWtb6Mg4JwRU explicit
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to 172.19.17.149 ([172.19.17.149]:22).
debug1: setting up multiplex master socket
debug1: channel 0: new [/tmp/xmdiez-ssh-ubuntu@172.19.17.149:22.sock]
debug1: channel 1: new [client-session]
debug1: Entering interactive session.
debug1: pledge: id
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: /home/ubuntu/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/ubuntu/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-1004-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Apr 21 18:39:56 UTC 2022

  System load:  0.03271484375     Processes:             111
  Usage of /:   18.9% of 7.58GB   Users logged in:       0
  Memory usage: 2%                IPv4 address for ens5: 172.19.17.149
  Swap usage:   0%


0 updates can be applied immediately.


Last login: Thu Apr 21 18:39:58 2022 from 10.26.41.70
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@ip-172-19-17-149:~$

This is the packer settings file ubuntu-22.04.json

{
    "variables": {
      "HOME": "{{env `HOME`}}",
      "TSTAMP": "{{env `TSTAMP`}}"
    },
    "builders": [
      {
        "name": "ubuntu-22.04-{{user `TSTAMP`}}",
        "type": "amazon-ebs",
        "region": "us-east-1",
        "source_ami_filter": {
            "filters": {
                "virtualization-type": "hvm",
                "name": "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*",
                "root-device-type": "ebs"
            },
            "owners": [
                "099720109477"
            ],
            "most_recent": true
        },
        "instance_type": "t3.large",
        "subnet_id": "subnet-XXXXXXXXXX",
        "vpc_id": "vpc-XXXXXXXXXX",
        "ssh_username": "ubuntu",
        "ami_name": "ubuntu-22.04-{{user `TSTAMP`}}",
        "ami_description": "Ubuntu 22.04 GoldenImage {{user `TSTAMP`}}",
        "ssh_keypair_name": "id_rsa_terraform",
        "ssh_private_key_file": "{{user `HOME`}}/.ssh/id_rsa_terraform.pem"
      }
    ],
    "provisioners": [
      {
        "type": "shell",
        "inline": [
          "echo hello world"
        ]
      }
    ]
  }

[mhahl]

Could you try one of the workarounds from here? #11656

[marcosdiez]

This workaround worked: #11656 (comment)

I'll keep the issue open because I still can ssh with my normal keys and packer can't.
Thank you!

[lorengordon]

Probably same root cause as #8609

[FelicianoTech]

For Ubuntu 22.04, OpenSSH was updated to v8.x and rsa host keys are disabled by default. Either a client key using ecc needs to be used, or my temp workaround, was to renable rsa on the host side.

[asottile]

at least for amazon-ebs I was able to get this working by using "temporary_key_pair_type": "ed25519", in my builder configuration

[sc250024]

Not working though for local ISO builds. See my comment here: #11656 (comment)

[sc250024]

I also tried this with the latest Nightly build (https://github.com/hashicorp/packer/releases/tag/nightly) as of today, and it's still the same problem with v1.8.1-dev.

[yukoba]

This problem comes from here. golang/go#49952

[sc250024]

@yukoba Thanks for posting that. I saw that issue in another thread, and it seems pretty core to Packer itself. I've tried to workaround this issue locally with virtualbox-iso builds, but unlike with the amazon-ebs builder, it seems that there's no workaround at the moment.

[sc250024]

I found a temporary workaround which should work for *-iso builds. I tested it on virtualbox-iso. I remembered that the https://github.com/chef/bento project already had a working Ubuntu 20.04 LTS build of the live version of their ISO. I was wondering how they were able to get theirs working.

This is the magic sauce: https://github.com/chef/bento/blob/118ad132f6bd7c09cbf40b4933281d32dfe139fe/packer_templates/ubuntu/http/user-data#L9-L11

Basically, you need to stop SSH during the Subiquity install process so that Packer doesn't freak out, and continues when SSH connectivity is restored.

[MichaelKorn]

Same issue in GCP with Ubuntu 22.04, but temporary_key_pair_type = "ed25519" works as workaround. Thanks for mention.