Unable to start npm container...

[BHuck74]

Dear all,

I've spent hours trying to get the following stack working but still encounter the following poor log file:

Checklist

• Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
  • Yes
• Are you sure you're not using someone else's docker image?
  • No
• Have you searched for similar issues (both open and closed)?
  • Yes

Describe the bug
Stack init seems to get stuck...

Nginx Proxy Manager Version
Not able to get the stack accessible.

To Reproduce
Steps to reproduce the behavior:

Docker-compose file:

version: '3'
services:      
  nginx-proxy-manager:
    container_name: nginx-proxy-manager
    restart: always
    image: jc21/nginx-proxy-manager:latest
    ports:
      - "81:81"
      - "80:80"
      - "4000:443"
    volumes:
      - "/mnt/docker/npm/data:/data"
      - "/mnt/docker/npm/letsencrypt:/etc/letsencrypt"
    environment:
      - DB_MYSQL_HOST=nginx-proxy-manager-db
      - DB_MYSQL_PORT=3306
      - DB_MYSQL_USER=XXX
      - DB_MYSQL_PASSWORD=XXX
      - DB_MYSQL_NAME=npm

    networks:
      - proxy_network
      - npm_network
  
  nginx-proxy-manager-db:
    container_name: nginx-proxy-manager-db
    image: jc21/mariadb-aria:10.4.15-innodb
    restart: always
    environment:
      - MYSQL_DATABASE=npm
      - MYSQL_USER=npm
      - MYSQL_PASSWORD=mysqlpass
      - MYSQL_ROOT_PASSWORD=mysqlrootpass
    volumes:
      - "/mnt/docker/npm/mysql:/var/lib/mysql"
    networks:
      - npm_network
    
      
networks:
  npm_network:
    driver: bridge
  proxy_network:
    external: true 

** Portainer log file **

❯ Configuring npm user ...
❯ Configuring npm group ...
❯ Checking paths ...
❯ Setting ownership ...

Operating System
Debian 10

[dziubalabs]

@BHuck74
Is your volume folder /mnt/docker/npm/mysql an NFS/samba share?

[SleepingPanda]

Stuck at the exact same spot with the same logs on a fresh install of 2.10.3:

❯ Configuring npm user ...
❯ Configuring npm group ...
❯ Checking paths ...
❯ Setting ownership ...

And 2.10.0, 2.10.1 and 2.10.2 refuse to run as well. All returning this log:

s6-rc: info: service s6rc-oneshot-runner: starting

s6-rc: info: service s6rc-oneshot-runner successfully started

s6-rc: info: service fix-attrs: starting

s6-rc: info: service fix-attrs successfully started

s6-rc: info: service legacy-cont-init: starting

s6-rc: info: service legacy-cont-init successfully started

s6-rc: info: service prepare: starting

❯ Configuring npmuser ...

id: 'npmuser': no such user

❯ Checking paths ...

❯ Setting ownership ...

s6-rc: fatal: timed out

s6-sudoc: fatal: unable to get exit status from server: Operation timed out

/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.

Here's the docker command I used to try running the service (changing out latest for 2.10.0, 2.10.1 and 2.10.2 as needed for trying older images):

$ docker run -d -h nginxproxymanager-2 --network macvlan.docker --ip 192.168.0.28 \
--mac-address 02:42:c0:a8:00:8c --name=nginxproxymanager-2 \
-v /mnt/docker/applications/nginx-proxy-manager-2/data:/data \
-v /mnt/docker/applications/nginx-proxy-manager-2/letsencrypt:/etc/letsencrypt \
-p 80:80 -p 443:443 -p 81:81 --restart unless-stopped jc21/nginx-proxy-manager:latest

I've also tried setting the following variables with no success: -e PUID=1000, -e PGID=1000, -e TZ=America/Port_of_Spain, -e DISABLE_IPV6=true .

Do note that I've used rm -rf /mnt/docker/applications/nginx-proxy-manager-2/* between each unsuccessful attempt to get the container working.

To answer what @dziubalabs asked above, no this directory is not an NFS or SAMBA share.

[BHuck74]

@BHuck74
Is your volume folder /mnt/docker/npm/mysql an NFS/samba share?

Actually not, local ext4 FS...

[BHuck74]

Dear all,

Thank you for your comments.

I finally succeeded with bringing up the stack by doing... nothing. The process just take a while, don't ask me why 🤷‍♂️ It's not just a matter of minutes, but more.

Cheers

[dziubalabs]

Great! I was asking about mounted FS, since I had similar issues with other docker. I first added 777 permissions to the main folder and it went OK. Then I needed to figure out what permissions and ownership the FS needed.

[SleepingPanda]

Huh... That's very interesting. I'll try leaving it alone then. @BHuck74 is that only for the first start or does it need time for every start?

[viktak]

I'm in the same shoes...
For me it's not a fresh install. It's been up and running for more than a month now, and today after a server reboot this happened...
If you say I need to let it run overnight, it's fine :) Will report back in the morning...

[viktak]

yes, in the morning all is good. Even if I restart it, it starts up quickly. So I guess it was only trying to do some maintenance or something.

[fahrulseptiana]

** Portainer log file **

❯ Configuring npm user ...
❯ Configuring npm group ...
❯ Checking paths ...
❯ Setting ownership ...

Well, today I have the same issue. It takes a long time to start up or restart, and the logs are exactly the same.

[lenkunz]

Before this happen to me, I try to backup NPM instance.
It is absurdly long time to copy. I inspect it and found that certbot generate more than 100,000 file for each letsencrypt/csr and letsencrypt/key.

The "Setting ownership ..." is a part of script to chmod all files related. I guess that's where the problems came from.

[Lwfrancisco]

Just found that certbot created well over a thousand keys for my small homeserver setup in the same folders described above. Confused and experiencing the same issue as OP with container taking far too long to start (actually, I've only waited 20 minutes so far and it's failed to start entirely so far). Honestly, I'm just trying to solve my initial issue of not being able to renew SSL Certs and seem to be running into more and more issues lol.

[davipro34]

Hi,
same here with a new installation!

[lippoliv]

Same here. There's just 163 files in the mount, but "changing ownership" takes roughly 90 seconds. It's a ZFS pool, so data should be in-memory, I think this should / could be improved?

But maybe it's not about permissions, but some logging output missing?

[timob]

I suspect it might be:

nginx-proxy-manager/docker/rootfs/etc/s6-overlay/s6-rc.d/prepare/30-ownership.sh

Line 27 in e08a4d4

chown -R "$PUID:$PGID" /opt/certbot

$ docker run -it --entrypoint /bin/bash jc21/nginx-proxy-manager:latest
 _   _       _            ____                      __  __
| \ | | __ _(_)_ __ __  _|  _ \ _ __ _____  ___   _|  \/  | __ _ _ __   __ _  __ _  ___ _ __
|  \| |/ _` | | '_ \\ \/ / |_) | '__/ _ \ \/ / | | | |\/| |/ _` | '_ \ / _` |/ _` |/ _ \ '__|
| |\  | (_| | | | | |>  <|  __/| | | (_) >  <| |_| | |  | | (_| | | | | (_| | (_| |  __/ |
|_| \_|\__, |_|_| |_/_/\_\_|   |_|  \___/_/\_\\__, |_|  |_|\__,_|_| |_|\__,_|\__, |\___|_|
       |___/                                  |___/                          |___/
Version 2.10.4 (d1d1819) 2023-11-22 01:06:49 UTC, OpenResty 1.21.4.2, debian 10 (buster), Certbot certbot 2.5.0
Base: debian:buster-slim, linux/amd64
Certbot: jc21/nginx-full:latest, linux/amd64
Node: jc21/nginx-full:certbot, linux/amd64

[root@docker-584873b15840:/app]# find /opt/certbot | wc -l
3567
[root@docker-584873b15840:/app]#

It's changing 3.5k files on startup.

Note sure if we need a new issue: "Container is slow to start"

[timob]

I suspect it might be: chown -R "$PUID:$PGID" /opt/certbot

I did a time on that command:

real    4m1.987s
user    0m0.017s
sys     0m0.806s

I'm running on:

CPU:
  Info: quad core model: Intel Core2 Quad Q9650 bits: 64 type: MCP
    arch: Core Yorkfield rev: A cache: L1: 256 KiB L2: 12 MiB
  Speed (MHz): avg: 2000 high: 2001 min/max: 1998/2997 cores: 1: 2000
    2: 2000 3: 2000 4: 2001 bogomips: 23995
  Flags: ht lm nx pae sse sse2 sse3 sse4_1 ssse3 vmx
Drives:
  Local Storage: total: raw: 5.57 TiB usable: 2.85 TiB used: 1.13 TiB (39.6%)
  ID-1: /dev/sda vendor: OCZ model: VERTEX4 size: 119.24 GiB
    speed: 3.0 Gb/s serial: OCZ-792JIM96K1LYQ7R8
  ID-2: /dev/sdb vendor: Toshiba model: HDWD130 size: 2.73 TiB
    speed: 3.0 Gb/s serial: 68HUNSHAS
  ID-3: /dev/sdc vendor: Toshiba model: HDWD130 size: 2.73 TiB
    speed: 3.0 Gb/s serial: 68HUEVYAS

With the docker filesystems running on the spinning HDDs.

[arthurnn]

thanks @timob for #3361 . I just encounter this same issue, would you mind releasing a new version with the improvement?

thanks all

[timob]

My change was included, but then inadvertently broken again.

nginx-proxy-manager/docker/rootfs/etc/s6-overlay/s6-rc.d/prepare/30-ownership.sh

Line 28 in 209c1b3

chown -R "$PUID:$PGID" /opt/certbot/lib/python*/site-packages

The -R is not needed. The existing sub directories/files don't need their permissions changed. Only the site-packages directory needs to be write-able by the user.

The existing ones are part of the root install that is readable by all.

[woodmichl]

If anyone is encountering this issue my PR #3617 fixes it for me. Instead of having to wait >2 hours for startup it only took 1 Minute and 30 seconds in my case.
As timob already mentioned the main culprit was the chown -R is taking eternities for this many files.
Therefore I replaced it with a find command that executes chown on every file that is not owned by $PUID.