7.4.21-fpm-alpine3.14 missing linux/ppc64le and linux/s390x

[jokay]

Using 7.4.21-fpm-alpine3.14 linux/ppc64le and linux/s390x are missing.

7.4.20-fpm-alpine3.13

7.4.21-fpm-alpine3.14

Can we expect them to get build or won't this happen anymore for these architectures? 🧐

[tianon]

Interestingly, we're getting build failures with all the hallmarks of opencontainers/runc#2750 but we're on Docker 20.10.7 with runc 1.0.0-rc95 (so that shouldn't be the problem). When I try the offending command with strace, I see that indeed, it is getting EPERM on faccessat2 (which shouldn't be happening), but if I run the same exact command directly it works fine, so I have to imagine this is something specific to make on these platforms?? 😬

...
writev(1, [{iov_base="", iov_len=0}, {iov_base="/bin/sh /usr/src/php/libtool --s"..., iov_len=563}], 2/bin/sh /usr/src/php/libtool --silent --preserve-dup-deps --mode=compile cc -Iext/opcache/ -I/usr/src/php/ext/opcache/ -I/usr/src/php/include -I/usr/src/php/main -I/usr/src/php -I/usr/src/php/ext/date/lib -I/usr/include/libxml2 -I/usr/src/php/TSRM -I/usr/src/php/Zend  -D__MUSL__  -Wno-clobbered -Wall -Wextra -Wno-strict-aliasing -Wno-implicit-fallthrough -Wno-unused-parameter -Wno-sign-compare -g -O2 -fvisibility=hidden -DZEND_SIGNALS    -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -c /usr/src/php/ext/opcache/zend_shared_alloc.c -o ext/opcache/zend_shared_alloc.lo 
) = 563
madvise(0x3ff85d25000, 4096, MADV_FREE) = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
faccessat2(AT_FDCWD, "/bin/sh", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: /bin/sh: Operation not per"..., iov_len=39}], 2make: /bin/sh: Operation not permitted
) = 39
...

[tianon]

I've reproduced with a much simpler reproducer:

$ docker run -it --rm -w /tmp alpine:3.14
/tmp # apk add --no-cache make strace
...
/tmp # echo -e 'all:\n\techo yay' > Makefile
/tmp # make
echo yay
make: echo: Operation not permitted
make: *** [Makefile:2: all] Error 127
/tmp # strace make
...
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
faccessat2(AT_FDCWD, "/usr/local/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/local/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: echo: Operation not permit"..., iov_len=36}], 2make: echo: Operation not permitted
) = 36
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: *** [Makefile:2: all] Erro"..., iov_len=38}], 2make: *** [Makefile:2: all] Error 127
) = 38
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
munmap(0x3ffabb4d000, 4096)             = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
chdir("/tmp")                           = 0
close(1)                                = 0
exit_group(2)                           = ?
+++ exited with 2 +++

Docker 20.10.7, runc 1.0.1, containerd 1.5.4, libseccomp 2.4.1 😞

(Should be plenty new in all respects for us to not see this.)

[thaJeztah]

@nirmannarang @anujajakhade FYI; perhaps one of you has ideas what's different here for s390x? FWIW, the faccessat2 has been on the "allow" list in docker's seccomp profile since docker 20.10.0 (see moby/moby#41353), so seccomp shouldn't be blocking it.

[nirmannarang]

@nirmannarang @anujajakhade FYI; perhaps one of you has ideas what's different here for s390x? FWIW, the faccessat2 has been on the "allow" list in docker's seccomp profile since docker 20.10.0 (see moby/moby#41353), so seccomp shouldn't be blocking it.

Will take a look.

[clnperez]

@tianon thanks for all those details. I don't know why this would happen only for Power and z. I remember the horrendous problem of another syscall in golang for which the value for p and z was different than that for x86/amd -- but I double-checked and it's 439 across the board.

The requirements in the faccessat2 section here are a little confusing: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0. Could we try with libseccomp 2.4.4?

[tianon]

Apparently my reproducer works fine (as in, does not reproduce any issue and properly prints yay) on ppc64le, however I can definitely reproduce with that minimal make on s390x even with libseccomp version 2.5.1 from Debian Bullseye.

[tianon]

Confirmed, apparently ppc64le has been fixed since this issue was filed! All those tags are built/pushed successfully. 😄

[anujajakhade]

I think this issue is resolved on s390x as well. I am not able to reproduce it. @tianon @jokay can you please confirm.

dropper # docker run -it --rm -w /tmp alpine:3.14
/tmp # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.14.1
PRETTY_NAME="Alpine Linux v3.14"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/tmp # uname -a
Linux 71fca25f7540 5.3.18-57-default #1 SMP Wed Apr 28 10:54:41 UTC 2021 (ba3c2e9) s390x Linux
/tmp 

[jokay]

@anujajakhade sorry I can't test this, I only use the Docker image to build multi-arch Docker images :)

[thaJeztah]

I wasn't able to fit an s390x server in my closet, so not sure if it adds anything to the table, but I can confirm it works on Docker Desktop for Mac (using QEMU emulation), or at least, it prints yay;

docker run -it --rm --platform=linux/s390x -w /tmp alpine:3.14
Unable to find image 'alpine:3.14' locally
3.14: Pulling from library/alpine
625f57562315: Pull complete
Digest: sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
Status: Downloaded newer image for alpine:3.14
/tmp # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.14.1
PRETTY_NAME="Alpine Linux v3.14"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/tmp # uname -a
Linux f4b3c3366135 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 s390x Linux
/tmp # apk add --no-cache make strace
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/s390x/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/s390x/APKINDEX.tar.gz
(1/2) Installing make (4.3-r0)
(2/2) Installing strace (5.12-r0)
Executing busybox-1.33.1-r3.trigger
OK: 7 MiB in 16 packages
/tmp # echo -e 'all:\n\techo yay' > Makefile
/tmp # make
echo yay
yay
/tmp # strace make
/usr/bin/strace: test_ptrace_get_syscall_info: PTRACE_TRACEME: Function not implemented
/usr/bin/strace: ptrace(PTRACE_TRACEME, ...): Function not implemented
/usr/bin/strace: PTRACE_SETOPTIONS: Function not implemented
/usr/bin/strace: detach: waitpid(37): No child process
/usr/bin/strace: Process 37 detached

[clnperez]

Good news all around. I would really love to know what fixed it where, if anyone has more info!