cmdeploy: "Request certificate" fails in NAT'ed environment

[johays]

(probably related to #414 )

Trying to install CM in a virtualized environment, thus NAT'ed by the container-manager. I run multiple containers on the same hosts, and normally exposing them to the outside world is not a problem.

When running cmdeploy run, all goes well until the certificate-generation (see logs below).

When I query what listens on port 80 it is acme. This is probably due to previous failed cmdeploy run, as this is a fresh image. Killing the listening acme-proccess or completely restarting the machine doesn't help.

additional question: what ports need to be forwarded to the machine? On the landing-page https://github.com/chatmail/relay only protocols (SMTP, IMAP, HTTPS, etc.), not ports (993, 443, etc.) are mentioned. I see acme trying to listen on port 80, but port 80 is not mentioned as a required portforward on the starting page?

---

--> Starting operation: Setup acmetool-redirector service 
    [chat.example.com] Success

--> Starting operation: Request certificate for: chat.example.com, mta-sts.chat.example.com, www.chat.example.com 
    [chat.example.com] 20251009122828 [DEBUG] acmetool.storageops: Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): best certificate satisfying is <nil>, err=Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): no certificate satisfies this target
    [chat.example.com] 20251009122828 [DEBUG] acmetool.storageops: Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): requesting certificate
    [chat.example.com] 20251009122829 [DEBUG] fdb: enforce permissions: keys/67gzuf4t5ieqzneke4ratpppg6w2yyvmvy3wqfckkobvvwb7klca/privkey 0/0 0/0
    [chat.example.com] 20251009122829 [DEBUG] acmetool.storageops: Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): ordering certificate
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "[::]:80" "6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: failed to listen on [::]:80: listen tcp 0.0.0.0:80: bind: address already in use
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port ":80" "6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: failed to listen on :80: listen tcp :80: bind: address already in use
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "[::1]:402" "6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: listening on [::1]:402
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "127.0.0.1:402" "6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: listening on 127.0.0.1:402
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "[::1]:4402" "6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: listening on [::1]:4402
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "127.0.0.1:4402" "6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: listening on 127.0.0.1:4402
    [chat.example.com] 20251009122830 [DEBUG] acme.responder: writing 2 webroot challenge files
    [chat.example.com] 20251009122830 [DEBUG] acme.responder: writing webroot file /var/www/html/.well-known/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA
    [chat.example.com] 20251009122830 [DEBUG] acme.responder: writing webroot file /var/run/acme/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA
    [chat.example.com] 20251009122830 [DEBUG] acme.responder: http-01 self test for "www.chat.example.com"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "[::]:80" "yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: failed to listen on [::]:80: listen tcp 0.0.0.0:80: bind: address already in use
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port ":80" "yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: failed to listen on :80: listen tcp :80: bind: address already in use
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "[::1]:402" "yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "127.0.0.1:402" "yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "[::1]:4402" "yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk"
    [chat.example.com] 20251009122830 [DEBUG] acmetool.reshttp: acquire port "127.0.0.1:4402" "yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk"
    [chat.example.com] 20251009122830 [DEBUG] acme.responder: writing 2 webroot challenge files
    [chat.example.com] 20251009122830 [DEBUG] acme.responder: writing webroot file /var/www/html/.well-known/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk
    [chat.example.com] 20251009122830 [DEBUG] acme.responder: writing webroot file /var/run/acme/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk
    [chat.example.com] 20251009122830 [DEBUG] acme.responder: http-01 self test for "mta-sts.chat.example.com"
    [chat.example.com] 20251009122835 [INFO] acme.responder: http-01 self test failed: www.chat.example.com: Get "http://www.chat.example.com/.well-known/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    [chat.example.com] 20251009122835 [DEBUG] acme.responder: removing webroot file /var/www/html/.well-known/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA
    [chat.example.com] 20251009122835 [DEBUG] acme.responder: removing webroot file /var/run/acme/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA
    [chat.example.com] 20251009122835 [DEBUG] acmetool.solver: challenge start failed: Get "http://www.chat.example.com/.well-known/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    [chat.example.com] 20251009122835 [INFO] acme.responder: http-01 self test failed: mta-sts.chat.example.com: Get "http://mta-sts.chat.example.com/.well-known/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    [chat.example.com] 20251009122835 [DEBUG] acme.responder: removing webroot file /var/run/acme/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk
    [chat.example.com] 20251009122835 [DEBUG] acme.responder: removing webroot file /var/www/html/.well-known/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk
    [chat.example.com] 20251009122835 [DEBUG] acmetool.solver: challenge start failed: Get "http://mta-sts.chat.example.com/.well-known/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    [chat.example.com] 20251009122835 [DEBUG] acmetool.solver: challenge start failed: could not install DNS challenge, no hooks succeeded
    [chat.example.com] 20251009122835 [DEBUG] acmetool.solver: challenge instantiation failed: challenge type not supported
    [chat.example.com] 20251009122835 [DEBUG] acmetool.solver: challenge instantiation failed: challenge type not supported
    [chat.example.com] 20251009122835 [DEBUG] acmetool.solver: challenge start failed: could not install DNS challenge, no hooks succeeded
    [chat.example.com] 20251009122835 [ERROR] acmetool.storageops: Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): failed to request certificate: the following errors occurred:
    [chat.example.com] exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz/2714463071/595295664711" [due to inner error: the following errors occurred:
    [chat.example.com] Get "http://www.chat.example.com/.well-known/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA": context deadline exceeded (Client.Timeout exceeded while awaiting headers); 
    [chat.example.com] could not install DNS challenge, no hooks succeeded; 
    [chat.example.com] challenge type not supported]; 
    [chat.example.com] exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz/2714463071/595295664641" [due to inner error: the following errors occurred:
    [chat.example.com] Get "http://mta-sts.chat.example.com/.well-known/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk": context deadline exceeded (Client.Timeout exceeded while awaiting headers); 
    [chat.example.com] challenge type not supported; 
    [chat.example.com] could not install DNS challenge, no hooks succeeded]
    [chat.example.com] 20251009122835 [DEBUG] acmetool.storageops: done processing targets, reconciliation complete, 1 errors occurred
    [chat.example.com] 20251009122835 [ERROR] acmetool.storageops: error while processing targets: the following errors occurred:
    [chat.example.com] error satisfying Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): the following errors occurred:
    [chat.example.com] exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz/2714463071/595295664711" [due to inner error: the following errors occurred:
    [chat.example.com] Get "http://www.chat.example.com/.well-known/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA": context deadline exceeded (Client.Timeout exceeded while awaiting headers); 
    [chat.example.com] could not install DNS challenge, no hooks succeeded; 
    [chat.example.com] challenge type not supported]; 
    [chat.example.com] exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz/2714463071/595295664641" [due to inner error: the following errors occurred:
    [chat.example.com] Get "http://mta-sts.chat.example.com/.well-known/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk": context deadline exceeded (Client.Timeout exceeded while awaiting headers); 
    [chat.example.com] challenge type not supported; 
    [chat.example.com] could not install DNS challenge, no hooks succeeded]
    [chat.example.com] 20251009122835 [ERROR] acmetool.storageops: failed to reconcile: the following errors occurred:
    [chat.example.com] error satisfying Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): the following errors occurred:
    [chat.example.com] exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz/2714463071/595295664711" [due to inner error: the following errors occurred:
    [chat.example.com] Get "http://www.chat.example.com/.well-known/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA": context deadline exceeded (Client.Timeout exceeded while awaiting headers); 
    [chat.example.com] could not install DNS challenge, no hooks succeeded; 
    [chat.example.com] challenge type not supported]; 
    [chat.example.com] exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz/2714463071/595295664641" [due to inner error: the following errors occurred:
    [chat.example.com] Get "http://mta-sts.chat.example.com/.well-known/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk": context deadline exceeded (Client.Timeout exceeded while awaiting headers); 
    [chat.example.com] challenge type not supported; 
    [chat.example.com] could not install DNS challenge, no hooks succeeded]
    [chat.example.com] 20251009122835 [DEBUG] acmetool.storageops: disjoint hostname mapping: "chat.example.com" -> Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0)
    [chat.example.com] 20251009122835 [DEBUG] acmetool.storageops: disjoint hostname mapping: "mta-sts.chat.example.com" -> Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0)
    [chat.example.com] 20251009122835 [DEBUG] acmetool.storageops: disjoint hostname mapping: "www.chat.example.com" -> Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0)
    [chat.example.com] 20251009122835 [DEBUG] acmetool.storageops: could not find certificate satisfying Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): no certificate satisfies this target
    [chat.example.com] 20251009122835 [DEBUG] acmetool.storageops: could not find certificate satisfying Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): no certificate satisfies this target
    [chat.example.com] 20251009122835 [DEBUG] acmetool.storageops: could not find certificate satisfying Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): no certificate satisfies this target
    [chat.example.com] 20251009122835 [CRITICAL] acmetool: fatal: reconcile: the following errors occurred:
    [chat.example.com] error satisfying Target(chat.example.com,mta-sts.chat.example.com,www.chat.example.com;https://acme-v02.api.letsencrypt.org/directory;0): the following errors occurred:
    [chat.example.com] exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz/2714463071/595295664711" [due to inner error: the following errors occurred:
    [chat.example.com] Get "http://www.chat.example.com/.well-known/acme-challenge/6rZ2WDDS3LUk8vo58IO49GZ_b12YyfpXVCI0TvIOWsA": context deadline exceeded (Client.Timeout exceeded while awaiting headers); 
    [chat.example.com] could not install DNS challenge, no hooks succeeded; 
    [chat.example.com] challenge type not supported]; 
    [chat.example.com] exhausted all possible challenges in authorization "https://acme-v02.api.letsencrypt.org/acme/authz/2714463071/595295664641" [due to inner error: the following errors occurred:
    [chat.example.com] Get "http://mta-sts.chat.example.com/.well-known/acme-challenge/yoVMXk1QS7rzJii8e_9pRcDM-T3XwLn_CFGX4VHj_fk": context deadline exceeded (Client.Timeout exceeded while awaiting headers); 
    [chat.example.com] challenge type not supported; 
    [chat.example.com] could not install DNS challenge, no hooks succeeded]
    [chat.example.com] Error: executed 0 commands

--> Disconnecting from hosts...
--> pyinfra error: No hosts remaining!
Deploy failed

[johays]

When running cmdeploy dns to test dns-settings I get these warning and recommendation.
However, I have set DNS as described on the landing-page https://github.com/chatmail/relay#getting-started as

chat.example.com. 3600 IN A 198.51.100.5
(IPv6 removed, no static IPv6 address available at ISP)
www.chat.example.com. 3600 IN CNAME chat.example.com.
 mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.

OUTPUT cmdeploy dns:

# ./scripts/cmdeploy dns
[ssh] login to chat.example.com
Collecting initial DNS settings..............
Check expected zone file entries...........................................
Please set required DNS entries at your DNS provider:

chat.example.com.                   MX 10 chat.example.com.
_mta-sts.chat.example.com.          TXT "v=STSv1; id=202510091251"
opendkim._domainkey.chat.example.com. TXT "v=DKIM1;k=rsa;p=;s=email;t=s"

If the DKIM entry above does not work with your DNS provider, you can try this one:

opendkim._domainkey.chat.example.com. TXT "v=DKIM1;k=rsa;p=;s=email;t=s"

WARNING: these recommended DNS entries are not set:

chat.example.com.                   TXT "v=spf1 a ~all"
_dmarc.chat.example.com.            TXT "v=DMARC1;p=reject;adkim=s;aspf=s"
chat.example.com.                   CAA 0 issue "letsencrypt.org;accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/2714463071"
_adsp._domainkey.chat.example.com.  TXT "dkim=discardable"
_submission._tcp.chat.example.com.  SRV 0 1 587 chat.example.com.
_submissions._tcp.chat.example.com. SRV 0 1 465 chat.example.com.
_imap._tcp.chat.example.com.        SRV 0 1 143 chat.example.com.
_imaps._tcp.chat.example.com.       SRV 0 1 993 chat.example.com.

[link2xt]

On the landing-page https://github.com/chatmail/relay only protocols (SMTP, IMAP, HTTPS, etc.), not ports (993, 443, etc.) are mentioned.

Currently readme mentions "SMTP/SUBMISSIONS/IMAPS/HTTPS ports". submissions is not really a protocol, it is an SMTP port 993.

You need TCP ports smtp (25), https (443), http (80), submission (587), imap (143), submissions (465), imaps (993) and UDP port 3478 (for STUN, used by iroh relay).