Skip to content

Internal error on SSL certificates #180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
IIPoliII opened this issue Jul 29, 2019 · 64 comments
Closed

Internal error on SSL certificates #180

IIPoliII opened this issue Jul 29, 2019 · 64 comments
Labels

Comments

@IIPoliII
Copy link

I tried to add new SSL certificates and i go this issue when try to generate it :
image
image
it was working perfectly fine 2 days ago i pulled the last container and for some reason i got this message (unhealthy) with docker ps
image

@IIPoliII IIPoliII added the bug label Jul 29, 2019
@IIPoliII
Copy link
Author

I founded this is docker inspect looks like something is blocking curl wich should not be the case

[
    {
        "Id": "5396c4e1207e3486ccb714d7cf9f2027f8be83c35fcbc5c0393d7cb3120a4609",
        "Created": "2019-07-26T12:05:10.838857676Z",
        "Path": "/init",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 15761,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-07-29T07:06:18.91961627Z",
            "FinishedAt": "2019-07-29T07:05:44.533099275Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 23,
                "Log": [
                    {
                        "Start": "2019-07-29T07:11:08.475218801Z",
                        "End": "2019-07-29T07:11:08.698302026Z",
                        "ExitCode": 1,
                        "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 9876: Connection refused\n"
                    },
                    {
                        "Start": "2019-07-29T07:11:23.708785289Z",
                        "End": "2019-07-29T07:11:23.931405635Z",
                        "ExitCode": 1,
                        "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 9876: Connection refused\n"
                    },
                    {
                        "Start": "2019-07-29T07:11:38.941957592Z",
                        "End": "2019-07-29T07:11:39.161505674Z",
                        "ExitCode": 1,
                        "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 9876: Connection refused\n"
                    },
                    {
                        "Start": "2019-07-29T07:11:54.171975308Z",
                        "End": "2019-07-29T07:11:54.424361521Z",
                        "ExitCode": 1,
                        "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 9876: Connection refused\n"
                    },

@IIPoliII IIPoliII changed the title Internal error on SSL certificates Internal error on SSL certificates (CURL error surely docker-proxy) Jul 29, 2019
@IIPoliII
Copy link
Author

IIPoliII commented Jul 29, 2019

When trying to ping my own ip on port 9876 (docker proxy i get this)

image
( founded there EOSIO/eosio-project-demux-example#25)

@IIPoliII
Copy link
Author

I finnaly went back to another snapshot i think the issue was that i auto run the container but if i am right it already does it ?
https://askubuntu.com/questions/620930/how-do-i-autostart-docker-container-at-system-reboot
I let this issue open in case someone has the same issue and Jc is willing to give a look at it c:

@IIPoliII
Copy link
Author

Okay i said nothing it did it again and i founded what is the issue. So basiclly create a new proxy host "Request a new SSL certificate". Then confirm, once you have done it remodify the host and add some more domaine, subdomain and go back in ssl certificates and "Request a new SSL certificate" save it and there it is the internal error.

@IIPoliII
Copy link
Author

After that it's impossible to use let's encrypt ever again in an ways
image

@IIPoliII IIPoliII changed the title Internal error on SSL certificates (CURL error surely docker-proxy) Internal error on SSL certificates Jul 29, 2019
@s4b3rt0oth
Copy link

I ran into this issue just today. I proceeded under the assumption that it was a bug.

While reviewing the docker logs I noticed it was looking for "npm-4" and "npm-5", but it couldn't find them. Eventually I came around to deleting the error SSL and remaking the proxy host, requesting the SSL again.

When you view docker logs what do they contain?

@Wadera
Copy link

Wadera commented Aug 22, 2019

I've got the same issue.

image

image

image

There is definitely a bug with Let's Encrypt certs.

Currently my Enginx Proxy Manager works only on port 81 (so I've got access to panel). Whatever I try to change - I've got internal error (but after web page refresh changes looks like saved). Unfortunately all my web traffic is dead now regarding to this issue :(

@Wadera
Copy link

Wadera commented Aug 22, 2019

I've managed to make it running by simple copying any other cert on place where script looking for and everything start's working:

image

@IIPoliII
Copy link
Author

I've managed to make it running by simple copying any other cert on place where script looking for and everything start's working:

image

Can you explain me what you did exactly and it's working now ? (Also i am a noob in docker so if you could provide the command to access the docker shell with it would be really nice from you c: !)

@Wadera
Copy link

Wadera commented Aug 23, 2019

I've used Portainer (Docker GUI), but you can do it running bellowed command on your node server:

To list all docker containers:

$ docker ps
CONTAINER ID  IMAGE    COMMAND  CREATED      STATUS      PORTS  NAMES
3xamp13id  my_image          X hours ago  Up X hours         my_container

To enter into container:

$ docker exec -it 3xamp13id bash 

Then inside just go into /etc/letsencrypt/live directories and copy any other certificate on place of missing one.

You can also do it on volume level as letsencrypt directories should be keep outside dicker container.

On default config:

    volumes:
      # Make sure this config.json file exists as per instructions above:
      - ./config.json:/app/config/production.json
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt

@IIPoliII
Copy link
Author

IIPoliII commented Aug 23, 2019

I see intressting thanks ! i hope @jc21 see this so maybe he can foudn a fix for it

@EpicLPer
Copy link

Yup, just got the same issue.

Has anyone found a fix yet that doesn't involve messing with the container itself? I'm not at all familiar with Docker, I'm happy that I got it running somehow haha :)

Let's ping @jc21 again to see if it helps ;)

@bpjobin
Copy link

bpjobin commented Oct 23, 2019

Same here. Running on docker as well.

@Shurov
Copy link

Shurov commented Nov 11, 2019

same, but I have problems with NAT Loopback (because of my router). Could this be an issue?

@Wadera
Copy link

Wadera commented Nov 11, 2019

I've didn't check my logs deeply when it occur, but it can be a case (I'm using it on home configuration and behind of home router as well).

@theDwiz01
Copy link

Same issues as above. Don't know Docker - friend setup for me. Worked forever but had to re-image OS drive (All Virtual stuff through HyperV and none of this was even on OS drive). After that nothing works. I try to delete all hosts and SSL certs and just get "Internal Error" for everything. Hoping somebody posts some things to try / command to fix

@EpicLPer
Copy link

I think I found the issue. When the URL you gave it isn't reachable it'll act as if everything worked instead of telling you what went wrong. I accidentally gave it an URL that hadn't been configured properly yet.

@lachlan334
Copy link

I started having this same issue yesterday after updating to 2.1.0. Rolling back to 2.0.14 seems to have fixed it.

@jc21
Copy link
Member

jc21 commented Feb 21, 2020

@lachlan334 it would be nice to know why it failed, the docker logs would have the info. FWIW I'm using 2.1.0 in production and not having these problems.

@SAOPP
Copy link

SAOPP commented Feb 21, 2020

Guys, I also have unhealthy status of npm-container, but it's works... what will be the recommendations?

@lachlan334
Copy link

@lachlan334 it would be nice to know why it failed, the docker logs would have the info. FWIW I'm using 2.1.0 in production and not having these problems.

The output I'm getting from the health check is % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (7) Failed to connect to localhost port 9876: Connection refused

Logs are as follows:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.,
[s6-init] ensuring user provided files have correct perms...exited 0.,
[fix-attrs.d] applying ownership & permissions fixes...,
[fix-attrs.d] done.,
[cont-init.d] executing container initialization scripts...,
[cont-init.d] done.,
[services.d] starting services,
[services.d] done.,
[2/21/2020] [2:53:12 AM] [Migrate  ] › ℹ  info      Current database version: 20190227065017,
[2/21/2020] [2:53:12 AM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...,
[2/21/2020] [2:53:12 AM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json,
[2/21/2020] [2:53:15 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4,
[2/21/2020] [2:53:15 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6,
[2/21/2020] [2:53:15 AM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized,
[2/21/2020] [2:53:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[2/21/2020] [2:53:15 AM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized,
[2/21/2020] [2:53:15 AM] [Global   ] › ℹ  info      Backend PID 206 listening on port 3000 ...,
[2/21/2020] [2:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [2:53:16 AM] [SSL      ] › ℹ  info      Renew Complete,
`QueryBuilder#allowEager` method is deprecated. You should use `allowGraph` instead. `allowEager` method will be removed in 3.0,
`QueryBuilder#eager` method is deprecated. You should use the `withGraphFetched` method instead. `eager` method will be removed in 3.0,
QueryBuilder#omit is deprecated. This method will be removed in version 3.0,
Connection Error: Error: read ECONNRESET,
[2/21/2020] [3:53:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[2/21/2020] [3:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [3:53:16 AM] [SSL      ] › ℹ  info      Renew Complete,
Connection Error: Error: read ECONNRESET,
Connection Error: Error: Packets out of order. Got: 1 Expected: 16,
Connection Error: Error: read ECONNRESET,
[2/21/2020] [4:53:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[2/21/2020] [4:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [4:53:16 AM] [SSL      ] › ℹ  info      Renew Complete,
Connection Error: Error: read ECONNRESET,
[2/21/2020] [5:53:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[2/21/2020] [5:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [5:53:16 AM] [SSL      ] › ℹ  info      Renew Complete,
Connection Error: Error: read ECONNRESET,
[2/21/2020] [6:53:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[2/21/2020] [6:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [6:53:16 AM] [SSL      ] › ℹ  info      Renew Complete,
Connection Error: Error: read ECONNRESET,
[2/21/2020] [7:53:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[2/21/2020] [7:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [7:53:16 AM] [SSL      ] › ℹ  info      Renew Complete,
Connection Error: Error: read ECONNRESET,
[2/21/2020] [8:53:15 AM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...,
[2/21/2020] [8:53:15 AM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json,
[2/21/2020] [8:53:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[2/21/2020] [8:53:15 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4,
[2/21/2020] [8:53:15 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6,
[2/21/2020] [8:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [8:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [8:53:16 AM] [SSL      ] › ℹ  info      Renew Complete,
Connection Error: Error: read ECONNRESET,
[2/21/2020] [9:53:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[2/21/2020] [9:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [9:53:16 AM] [SSL      ] › ℹ  info      Renew Complete,
Connection Error: Error: read ECONNRESET,
[2/21/2020] [10:53:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[2/21/2020] [10:53:16 AM] [Nginx    ] › ℹ  info      Reloading Nginx,
[2/21/2020] [10:53:16 AM] [SSL      ] › ℹ  info      Renew Complete,
Connection Error: Error: read ECONNRESET,

@SAOPP
Copy link

SAOPP commented Feb 21, 2020

@lachlan334 hey buddy I have the same.. but npm is working fine.

Last output | % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (7) Failed to connect to localhost port 9876: Connection refused
-- | --

@GregHilston
Copy link

If it means anything, I've been getting this error with version 2.0.14 running the docker image jlesage/nginx-proxy-manager

@jc21
Copy link
Member

jc21 commented Feb 23, 2020

@GregHilston Yep that means a lot. I don't maintain jlesage's images. We're up to 2.1.1 now and port 9876 isn't used internally anymore.

@lachlan334 Looks like you were using that 2.1.1 image, but it doesn't make sense why the Healthcheck is still using a setting from a previous docker image. The correct healthcheck will eventually run a curl command: curl --silent http://127.0.0.1:81/api/ from within the NPM container.

Perhaps try pulling down the stack entirely and putting it back up, assuming your have directory mounted volumes: docker-compose down && docker-compose up -d

@0ShinyNinetales0
Copy link

0ShinyNinetales0 commented Feb 24, 2020

I'm running into a similar problem. Requesting a new SSL certificate generates an Internal Error.
Inspecting the docker container shows it's unhealthy and it's spitting some errors:
`
[

{

    "Id": "9d2d526acb3512e60ca4db020b9ef86b93e859735267d224223ca40f5054b3d4",
    "Created": "2020-02-21T02:30:55.239750232Z",
    "Path": "/init",
    "Args": [],
    "State": {
        "Status": "running",
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 3473,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2020-02-21T02:30:58.116917696Z",
        "FinishedAt": "0001-01-01T00:00:00Z",
        "Health": {
            "Status": "unhealthy",
            "FailingStreak": 20095,
            "Log": [
                {
                    "Start": "2020-02-24T16:56:39.096936098+01:00",
                    "End": "2020-02-24T16:56:39.294612731+01:00",
                    "ExitCode": 1,
                    "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to localhost port 9876: Connection refused\n"
                },
                {
                    "Start": "2020-02-24T16:56:54.409165629+01:00",
                    "End": "2020-02-24T16:56:54.726601709+01:00",
                    "ExitCode": 1,
                    "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to localhost port 9876: Connection refused\n"
                },
                {
                    "Start": "2020-02-24T16:57:09.810871898+01:00",
                    "End": "2020-02-24T16:57:10.019143483+01:00",
                    "ExitCode": 1,
                    "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to localhost port 9876: Connection refused\n"
                },
                {
                    "Start": "2020-02-24T16:57:25.125134859+01:00",
                    "End": "2020-02-24T16:57:25.335763382+01:00",
                    "ExitCode": 1,
                    "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to localhost port 9876: Connection refused\n"
                },
                {
                    "Start": "2020-02-24T16:57:40.435902211+01:00",
                    "End": "2020-02-24T16:57:40.643842955+01:00",
                    "ExitCode": 1,
                    "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to localhost port 9876: Connection refused\n"
                }
            ]
        }

`
Here I saw the connection refused on 9876, not knowing what's causing it.
I'm using jc21's image, running version 2.1.1.

I don't mean to hijack, but perhaps this is the same problem as OP's running into.

EDIT: Code formatting

@WhistleMaster
Copy link

WhistleMaster commented Feb 26, 2020

Here I saw the connection refused on 9876, not knowing what's causing it.
I'm using jc21's image, running version 2.1.1.

Same here ! Using jc21's image, version 2.1.1, on docker bridge.

EDIT: deleting the container and creating a new one from scratch did fix it though.

@jc21
Copy link
Member

jc21 commented Feb 26, 2020

Not sure if it's a docker bug or expected behaviour but it feels like upgrading a docker image within an already-created container, doesn't update the healthcheck. There were quite a few changes between 2.0.* and 2.1.* of NPM so I recommend bringing down the stack entirely and recreating it.

Using existing data and letsencrypt data mounts will be fine though.

@SAOPP
Copy link

SAOPP commented Mar 10, 2020

So guys, what I need to do? Use the develop of the container? 'Coz I have unhealthy still. Thanks in advance.

@nldoty
Copy link

nldoty commented Mar 10, 2020

@SAOPP unhealthy typically means the container took more than 5 seconds to return an OK status. If you're able to create new SSL certificates, you should be good to go. The unhealthy status isn't a big deal.

@SAOPP
Copy link

SAOPP commented Mar 10, 2020

Yes I understood it, just interesting, and yes I can create a new certs, but I see guys talked about not possible auto renewal certs or I misunderstood?

@Foxglove-X
Copy link

Please forgive my being uninformed on Docker, but when will this get pushed out to places like Dockerhub? I'm using an Unraid environment.
Is there a way I can upload and use my own certificate in the meantime, such as a Cloudflare certificate?

@nldoty
Copy link

nldoty commented Mar 15, 2020

@Foxglove-X I believe it was already pushed about 4 days ago.
Check the tags under the Docker Hub page.
https://hub.docker.com/r/jc21/nginx-proxy-manager/tags

@Foxglove-X
Copy link

Ah I see! Mine was pulled from jlesage/nginx-proxy-manager/ which is 5 months out of date.
That must be some other Docker image that's not as well maintained. I should probably use this one instead. Thank you.

@jc21
Copy link
Member

jc21 commented Mar 15, 2020

jlesage will have to update his own docker images each time I release mine. I don't have an unraid setup myself so I don't know what changes are required to make it work

@rto54
Copy link

rto54 commented Apr 7, 2020

Hello,
I just installed the latest version of npm.

docker-compose.yml

version: "3"
services:
  app:
    image: jc21/nginx-proxy-manager:latest
    restart: always
    ports:
      - 80:80
      - 81:81
      - 443:443
    volumes:
      - ./config.json:/app/config/production.json
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
    environment:
    # if you want pretty colors in your docker logs:
    - FORCE_COLOR=1
  db:
    image: mariadb:latest
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: "npm"
      MYSQL_DATABASE: "npm"
      MYSQL_USER: "npm"
      MYSQL_PASSWORD: "npm"
    volumes:
      - ./data/mysql:/var/lib/mysql

config.json

{
  "database": {
    "engine": "mysql",
    "host": "db",
    "name": "npm",
    "user": "npm",
    "password": "npm",
    "port": 3306
  }
}

I tried to add new SSL certificates and i go this issue when try to generate it :

image

image

image

image

image

I have

image

Thank for all

@nldoty
Copy link

nldoty commented Apr 7, 2020

@rto54 without the logs from the docker containers themselves, there's not a whole lot anyone can debug. Can you provide them?

@rto54
Copy link

rto54 commented Apr 7, 2020

@nldoty, You can help me for this.
I am a beginner with the docker :-(
Thank

@rto54
Copy link

rto54 commented Apr 7, 2020

4/7/2020] [6:55:06 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [6:55:07 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-2" --agree-tos --email "[email protected]" --preferred-challenges "dns,http" --webroot --domains "unifi.ophiuchus.fr" ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
Plugins selected: Authenticator webroot, Installer None,
Obtaining a new certificate,
Performing the following challenges:,
http-01 challenge for unifi.ophiuchus.fr,
Using the webroot path /data/letsencrypt-acme-challenge for all unmatched domains.,
Waiting for verification...,
Challenge failed for domain unifi.ophiuchus.fr,
http-01 challenge for unifi.ophiuchus.fr,
Cleaning up challenges,
Some challenges have failed.,
,
[4/7/2020] [7:03:09 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:03:24 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:03:32 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:03:42 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #2: unifi.ophiuchus.fr,
[4/7/2020] [7:03:43 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot renew --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-2" --preferred-challenges "dns,http" --disable-hook-validation ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
No certificate found with name npm-2 (expected /etc/letsencrypt/renewal/npm-2.conf).,
,
[4/7/2020] [7:03:46 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #1: jeedom.ophiuchus.fr,
[4/7/2020] [7:03:48 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot renew --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-1" --preferred-challenges "dns,http" --disable-hook-validation ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
No certificate found with name npm-1 (expected /etc/letsencrypt/renewal/npm-1.conf).,
,
[4/7/2020] [7:05:41 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #1: jeedom.ophiuchus.fr,
[4/7/2020] [7:05:43 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot renew --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-1" --preferred-challenges "dns,http" --disable-hook-validation ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
No certificate found with name npm-1 (expected /etc/letsencrypt/renewal/npm-1.conf).,
,
Connection Error: Error: read ECONNRESET,
[4/7/2020] [7:22:47 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #1: jeedom.ophiuchus.fr,
[4/7/2020] [7:22:48 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot renew --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-1" --preferred-challenges "dns,http" --disable-hook-validation ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
No certificate found with name npm-1 (expected /etc/letsencrypt/renewal/npm-1.conf).,
,
[4/7/2020] [7:23:19 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:23:23 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:23:53 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:24:08 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #1: jeedom.ophiuchus.fr,
[4/7/2020] [7:24:09 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot renew --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-1" --preferred-challenges "dns,http" --disable-hook-validation ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
No certificate found with name npm-1 (expected /etc/letsencrypt/renewal/npm-1.conf).,
,
[4/7/2020] [7:24:14 PM] [SSL      ] › ℹ  info      Revoking Let'sEncrypt certificates for Cert #1: jeedom.ophiuchus.fr,
[4/7/2020] [7:24:22 PM] [SSL      ] › ℹ  info      Revoking Let'sEncrypt certificates for Cert #2: unifi.ophiuchus.fr,
[4/7/2020] [7:25:17 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:25:51 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:25:51 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #3: unifi.ophiuchus.fr,
[4/7/2020] [7:25:56 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:25:56 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-3" --agree-tos --email "[email protected]" --preferred-challenges "dns,http" --webroot --domains "unifi.ophiuchus.fr" ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
Plugins selected: Authenticator webroot, Installer None,
Obtaining a new certificate,
Performing the following challenges:,
http-01 challenge for unifi.ophiuchus.fr,
Using the webroot path /data/letsencrypt-acme-challenge for all unmatched domains.,
Waiting for verification...,
Challenge failed for domain unifi.ophiuchus.fr,
http-01 challenge for unifi.ophiuchus.fr,
Cleaning up challenges,
Some challenges have failed.,
,
[4/7/2020] [7:26:25 PM] [Access   ] › ℹ  info      Building Access file #1 for: ophiuchus,
[4/7/2020] [7:26:25 PM] [Access   ] › ℹ  info      Adding: rodolphe,
[4/7/2020] [7:26:26 PM] [Access   ] › ✔  success   Built Access file #1 for: ophiuchus,
[4/7/2020] [7:33:08 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:33:08 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #4: jeedom.ophiuchus.fr,
[4/7/2020] [7:33:13 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:33:13 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-4" --agree-tos --email "[email protected]" --preferred-challenges "dns,http" --webroot --domains "jeedom.ophiuchus.fr" ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
Plugins selected: Authenticator webroot, Installer None,
Obtaining a new certificate,
Performing the following challenges:,
http-01 challenge for jeedom.ophiuchus.fr,
Using the webroot path /data/letsencrypt-acme-challenge for all unmatched domains.,
Waiting for verification...,
Challenge failed for domain jeedom.ophiuchus.fr,
http-01 challenge for jeedom.ophiuchus.fr,
Cleaning up challenges,
Some challenges have failed.,
,
[4/7/2020] [7:33:15 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[4/7/2020] [7:33:17 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[4/7/2020] [7:33:17 PM] [SSL      ] › ℹ  info      Renew Complete,
[4/7/2020] [7:33:17 PM] [SSL      ] › ✖  error     Certificate is not valid (Command failed: openssl x509 -in /etc/letsencrypt/live/npm-3/fullchain.pem -subject -noout,
Can't open /etc/letsencrypt/live/npm-3/fullchain.pem for reading, No such file or directory,
140650706484552:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/etc/letsencrypt/live/npm-3/fullchain.pem','r'),
140650706484552:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:,
unable to load certificate,
),
[4/7/2020] [7:33:17 PM] [SSL      ] › ✖  error     Certificate is not valid (Command failed: openssl x509 -in /etc/letsencrypt/live/npm-4/fullchain.pem -subject -noout,
Can't open /etc/letsencrypt/live/npm-4/fullchain.pem for reading, No such file or directory,
140360297569608:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/etc/letsencrypt/live/npm-4/fullchain.pem','r'),
140360297569608:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:,
unable to load certificate,
),
[4/7/2020] [7:33:41 PM] [Nginx    ] › ℹ  info      Reloading Nginx,

@rto54
Copy link

rto54 commented Apr 7, 2020

Logs letsencrypt

[root@docker-70a1dec97cfe:/var/log/letsencrypt]# vi letsencrypt.log
2020-04-07 20:01:27,880:DEBUG:certbot._internal.main:certbot version: 1.0.0
2020-04-07 20:01:27,881:DEBUG:certbot._internal.main:Arguments: ['--non-interactive', '--config', '/etc/letsencrypt.ini', '--cert-name', 'npm-3', '--preferred-challenges', 'dns,http', '--disable-hook
2020-04-07 20:01:27,881:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-04-07 20:01:27,919:DEBUG:certbot._internal.log:Root logging level set at 20
2020-04-07 20:01:27,919:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-04-07 20:01:27,920:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==1.0.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3.8/site-packages/certbot/main.py", line 14, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 1350, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 1259, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 391, in handle_renewal_request
    conf_files = [storage.renewal_file_for_certname(config, config.certname)]
  File "/usr/lib/python3.8/site-packages/certbot/_internal/storage.py", line 52, in renewal_file_for_certname
    raise errors.CertStorageError("No certificate found with name {0} (expected "
certbot.errors.CertStorageError: No certificate found with name npm-3 (expected /etc/letsencrypt/renewal/npm-3.conf).

@Nornode
Copy link

Nornode commented Apr 17, 2020

I seem to have a very similar issue...
I had the issue prior to pulling the latest image too
docker logs nginx_proxy

[4/16/2020] [3:17:52 PM] [Migrate  ] › ℹ  info      Current database version: 20200410143839
[4/16/2020] [3:17:52 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
[4/16/2020] [3:17:52 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[4/16/2020] [3:17:52 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
[4/16/2020] [3:17:53 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
[4/16/2020] [3:17:53 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
[4/16/2020] [3:17:53 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[4/16/2020] [3:17:53 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
[4/16/2020] [3:17:53 PM] [Global   ] › ℹ  info      Backend PID 203 listening on port 3000 ...
`QueryBuilder#allowEager` method is deprecated. You should use `allowGraph` instead. `allowEager` method will be removed in 3.0
`QueryBuilder#eager` method is deprecated. You should use the `withGraphFetched` method instead. `eager` method will be removed in 3.0
QueryBuilder#omit is deprecated. This method will be removed in version 3.0
[4/16/2020] [3:18:46 PM] [SSL      ] › ✖  error     Error: Command failed: /usr/bin/certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
Challenge failed for domain cloud.domain.net
Challenge failed for domain www.domain.net
Attempting to renew cert (npm-3) from /etc/letsencrypt/renewal/npm-3.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

    at ChildProcess.exithandler (child_process.js:295:12)
    at ChildProcess.emit (events.js:210:5)
    at maybeClose (internal/child_process.js:1028:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)

container-start.sh

docker create \
  --name $name \
  -p 80:80 \
  -p 443:443 \
  -p 81:81 \
  -v /mnt/z.raid1/software.docker/A-dockerStartFiles/nginx-proxy/config.json:/app/config/production.json \
  -v /mnt/z.raid1/software.docker/nginx_proxy/data:/data \
  -v /mnt/z.raid1/software.docker/nginx_proxy/letsencrypt:/etc/letsencrypt \
  -e TZ=Europe/Helsinki \
  -e FORCE_COLOR=1 \
  --restart unless-stopped \
  --log-opt max-size=1m \
    jc21/nginx-proxy-manager:latest

docker image inspect jc21/nginx-proxy-manager:latest

[
    {
        "Id": "sha256:ed18ab76b851f62ab3672b5a35b19f43671a4380c4bb8d95b4209a08c6de36e5",
        "RepoTags": [
            "jc21/nginx-proxy-manager:latest"
        ],
        "RepoDigests": [
            "jc21/nginx-proxy-manager@sha256:940681f8439604c9a5a2b88d2036adbcdd23675908bf915806a3289fa8c6b226"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2020-04-15T06:02:31.803807951Z",

logs in container:
(/var/log/letsencrypt/letsencrypt.log)

2020-04-17 08:21:32,578:INFO:certbot._internal.renewal:Cert not yet due for renewal
2020-04-17 08:21:32,580:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2020-04-17 08:21:32,583:DEBUG:certbot._internal.cli:Var pref_challs=dns,http (set by user).
2020-04-17 08:21:32,583:DEBUG:certbot._internal.cli:Var authenticator=webroot (set by user).
2020-04-17 08:21:32,584:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2020-04-17 08:21:32,584:DEBUG:certbot._internal.cli:Var webroot_map={'webroot_path'} (set by user).
2020-04-17 08:21:32,584:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2020-04-17 08:21:32,609:INFO:certbot._internal.renewal:Cert not yet due for renewal
2020-04-17 08:21:32,612:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2020-04-17 08:21:32,612:ERROR:certbot._internal.renewal:All renewal attempts failed. The following certs could not be renewed:
2020-04-17 08:21:32,613:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
2020-04-17 08:21:32,613:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==1.0.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3.8/site-packages/certbot/main.py", line 14, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 1350, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 1259, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 473, in handle_renewal_request
    raise errors.Error("{0} renew failure(s), {1} parse failure(s)".format(
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)

Let me know if there are any other information missing from above.

@mike-ford-igt
Copy link

Not sure if anyone else has run into this. I'm running the proxy container on VM inside corporate network and when I try and create the proxy in Nginx Proxy Manager, I too get the 'Internal Error' and in the log I can see that lets encrypt can't validate my domain. Under SSL Certificates there is an entry but its immediately expired. In the DNS Manager we use I have my domain mapped to the host IP of the machine where I'm running the proxy container in the Forward Lookup Zone. Am I correct in thinking letsencrypt just can't reach my domain to validate that 'I own it'?

2020-05-04 17:10:39,713:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/4362994539 HTTP/1.1" 200 625
2020-05-04 17:10:39,714:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 04 May 2020 17:11:21 GMT
Content-Type: application/json
Content-Length: 625
Connection: keep-alive
Boulder-Requester: 85028888
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: 0002Z8epRT7DdbUiCHQ6gsOQuvNlarPnL7KRDGYsEg7qoCM
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"identifier": {
"type": "dns",
"value": ""
},
"status": "invalid",
"expires": "2020-05-11T17:11:19Z",
"challenges": [
{
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:ietf:params:acme:error:dns",
"detail": "DNS problem: NXDOMAIN looking up A for - check that a DNS record exists for this domain",
"status": 400
},
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/4362994539/zHEy1w",
"token": "PCdm1kOKDDqyzF4LX6-gAt0omNYuV0iwRNBtQ0NLR5Q"
}
]
}

@s4b3rt0oth
Copy link

@jmford775 Yes, you are correct. Letsencrypt requires you to verify ownership of a domain before issuing a certificate. It does this by first looking for a valid domain response, then reaching out to the domain to verify the server is sitting there. For this to happen you will need to point the domain to the IP you plan on using and open port 80 on that IP and have it point to your proxy container.

From the sound of your post you plan on using this internally. Note that setting up internal DNS to point to your machine will not be sufficient. There must be a valid public IP address with matching DNS pointed to your container. Likewise, the certificate will be valid ONLY for that external address and will not work internally (unless you utilize the same naming scheme).

For example:
External Public IP: 1.1.1.x
Internal Public IP: 10.0.10.x
Public DNS: public.com
Internal DNS: internal.com

Your container sits on 10.0.10.y and it requests a certificate for container.internal.com. Letsencrypt does not know where your container is, because it is using an internal DNS schema. If you request container.public.com then letsencrypt will be able to see that you pointed container.public.com to your external IP (1.1.1.x) and will reach out to that IP (Port 80 [forwarded on firewall]) to verify that the container is actually sitting there.

@AnonJervis
Copy link

@s4b3rt0oth I have the same problem as @jmford775 but mine is to connect from the outside. My port 80 and 443 are open and pointing to NPM server but it still shows internal error. The exact same thing happened to me where once trying to add SSL cert, it shows "Internal Error" and will show the cert expired same date and time when I tried adding.

@bpjobin
Copy link

bpjobin commented May 5, 2020 via email

@mike-ford-igt
Copy link

@s4b3rt0oth Yeah I realized it was going to be a no-go being inside our corporate network and thanks for the reply. I decided since we are already behind super strict firewalls that I would just run a no-auth registry and joxit ui

@lhend941
Copy link

lhend941 commented Sep 8, 2020

Same thing here. Using the Nginx Proxy Manager Add On for Home Assistant

@vinhmh
Copy link

vinhmh commented Mar 14, 2021

are you change expose port of npm app difference 80 ?

@postmaster11
Copy link

Got the same issue... I have 4 proxy-hosts, only one is unable to get a new certificate. Finally ended up in NGINX logs where I found that NGINX is not able to fire up the webserver for verification (cannot bind IP-address to port 80).
After comparing this NPM container with another one, I found that I changed the Docker-network from 172.... to 10.10.0.0. It's possible that I changed this after deploying the first proxy-host. I guess the old IP or subnet is still somewhere in NGINX configfiles - does somebody have an idea where? I'm not really used with NGINX...

@MarkIvanowich
Copy link

Strangely, I had certs reporting as expired, but the browser was reporting up-to-date certificates. One of my certs was disconnected for a while, so of course it's cert didn't renew. Connecting and trying to renew showed me this internal error.

I restarted my docker containers, then changed the 'default site' option in settings to 'custom page'. It was previously set to '404 page'. I was thinking that the 404 was returned before the challenge file, interfering with renewals.

When I looked at the certificate page, all my certs were up-to-date. So I'm not sure if it was the container reboot, or if it was changing the 'default site' option that solved my problem. Give it a try?

@pgk83
Copy link

pgk83 commented Sep 19, 2021

Same issue here, i change the Default Site from Congratulations Page to Redirect and i give internal error on cert renew. i think npm redirect the request before the certbot find the /.well-known/-acme-chellenge/

i change it back for congrat page but i have a same erreor...

Strangely, I had certs reporting as expired, but the browser was reporting up-to-date certificates. One of my certs was disconnected for a while, so of course it's cert didn't renew. Connecting and trying to renew showed me this internal error.

I restarted my docker containers, then changed the 'default site' option in settings to 'custom page'. It was previously set to '404 page'. I was thinking that the 404 was returned before the challenge file, interfering with renewals.

When I looked at the certificate page, all my certs were up-to-date. So I'm not sure if it was the container reboot, or if it was changing the 'default site' option that solved my problem. Give it a try?

@Foxglove-X
Copy link

I had this problem at one point. I use CloudFlare DNS, and I was able to resolve by disabling CloudFlare's proxy.
There was not much helpful error information when the certificate generation failed, but I could confirm I had the issue anytime proxy was enabled, issue resolved once it was disabled.

@archef2000
Copy link

archef2000 commented Oct 16, 2021

it has to do with the dns it cant resolve the url just set your own dns server in /etc/resolv.conf or in docker-compose
dns:
-172.17.0.1
-172.20.0.1 (for me)

@chaptergy
Copy link
Collaborator

As this issue is very old, there have been huge changes in how certificates work since then. And as "Internal error" is just a generic error and does not provide any information about the actual error, all newer reports probably have nothing to do with the original issue reported in this ticket. To prevent people from posting "I see an internal error too", I will close and lock this issue. If you encounter any problems please look for other issues describing the same issue as you have, or if you can't find any, open a new issue. Include as much information as possible in this new issue like logs. Also see #1271 for more information.

@NginxProxyManager NginxProxyManager locked and limited conversation to collaborators Oct 25, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests