os/user: LookupGroup does not identify all cases of UnknownGroupError

[anonymouse64]

What version of Go are you using (go version)?

$ go version
go version go1.14.5 linux/amd64

Does this issue reproduce with the latest release?

I only tested Go 1.14.5, I did not test Go 1.14.6 or tip.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOENV="/home/user/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/user/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/snap/go/6090"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/snap/go/6090/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build704426793=/tmp/go-build -gno-record-gcc-switches"

What did you do?

https://play.golang.org/p/qEi8JbjG5-4 when built on Linux and run inside a Ubuntu 20.10 chroot inside a Ubuntu 16.04 LXD container inside an Ubuntu 16.04 VM (this setup comes from a cloud building environment) fails to identify the returned error as a user.UnknownGroupError.

I suspect there is something about the implementation that is specific to the chroot, as running the same program outside of the chroot (inside the LXD container) successfully returns. See:

# ./golang-group-lookup 
successful error handling
# chroot /tmp/tmp.evWDSKSvUt/ /golang-group-lookup
error, not matchable (type *errors.errorString), user: lookup groupname thisdoesnotexist: no such process
# uname -a
Linux first-marlin-ubuntu-bartender 4.4.0-185-generic #215-Ubuntu SMP Mon Jun 8 21:53:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

It's possible something about getgrpnam_r in the version of glibc in Ubuntu 20.10 has changed to return a different error code, but as per the man page of getgrpnam_r under "Notes", there are numerous possible error codes that could be used to identify user.UnknownGroupError:

The formulation given above under "RETURN VALUE" is from POSIX.1-2001. It does not call "not found" an error, hence does not specify what value errno might have in this situation. But that makes it impossible to recognize errors. One might argue that according to POSIX errno should be left unchanged if an entry is not found. Experiments on various UNIX-like systems shows that lots of different values occur in this situation: 0, ENOENT, EBADF, ESRCH, EWOULDBLOCK, EPERM and probably others.

As such, I would expect Go to do a "best effort" job at identifying errors returned as user.UnknownGroupError, including at least the documented ENOENT, ESRCH, EBADF, EPERM cases mentioned in the man page (in this case it is ESRCH that is being returned).

Finally, note that the same problem hypothetically affects user.Lookup and the associated user.UnknownUserError, but getpwnam_r is not acting up the same way getgrpnam_r is here.

What did you expect to see?

I expected the program when run inside the chroot to work the same as outside of the chroot.

What did you see instead?

Inside the chroot it fails to identify the returned error as user.UnknownGroupError.

[anonymouse64]

Further debugging of the issue revealed the issue to actually be related to the fact that we are in the chroot, and hence we do not have things like systemd userdb to respond, however the chroot still had this in it's /etc/nsswitch.conf:

# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         files systemd
group:          files systemd
shadow:         files
gshadow:        files

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

notice group: files systemd.

So it seems to be that really we have an issue with NSS in this chroot that gets bubbled up to getgrpnam_r as ESRCH, which Go then doesn't recognize as user.UnknownGroupError.

[toothrot]

/cc @bradfitz @kevinburke

[kevinburke]

Interesting... I'll try to take a look later this week. I don't think this is important enough to get through the code freeze so we'd be targeting 1.15.1 or 1.16 (another way of saying that the fix will not show up in a public release for a while).

[xnox]

Checking through nss_systemd code, I do not see it returning anything unusual on the nss module level. It returns things like NSS_STATUS_UNAVAIL NSS_STATUS_NOTFOUND NSS_STATUS_TRYAGAIN. But I didn't check how that translates between libc & nss interface, and golang & libc.

[kevinburke]

I'm having trouble reproducing. I am running the same program you are:

package main

import (
	"fmt"
	"os/user"
)

func main() {
	_, err := user.LookupGroup("thisdoesnotexist")
	fmt.Printf("%#v\n", err)
	switch err.(type) {
	case user.UnknownGroupError:
		fmt.Println("successful error handling")
	default:
		fmt.Printf("error, not matchable (type %T), %v\n", err, err)
	}
}

in tmp/test.go, then compiling the Go binary in a Linux container:

$ docker run --tty -v $(PWD)/tmp:/go --interactive meterup/ubuntu-golang:latest /usr/local/go/bin/go build -ldflags '-extldflags "-static"' -o /go/test/test-program ./

then sharing the compiled binary into a Docker container:

$ docker run --tty -v $(PWD)/tmp:/go --interactive meterup/ubuntu-golang:latest /bin/bash --login

running ldd to find remaining links in the binary:

$ ldd /go/test/test-program
	linux-vdso.so.1 (0x00007ffe801af000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007feed8a62000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007feed8671000)
	/lib64/ld-linux-x86-64.so.2 (0x00007feed8c81000)

copying these files into the chroot:

# mkdir /go/test/lib/x86_64-linux-gnu 
# cp -r --dereference /lib/x86_64-linux-gnu/* /go/test/lib/x86_64-linux-gnu

(and similar for the other files)

then, finally, running the program:

chroot /go/test /test-program
&errors.errorString{s:"user: lookup groupname thisdoesnotexist: no such file or directory"}
error, not matchable (type *errors.errorString), user: lookup groupname thisdoesnotexist: no such file or directory

Running with strace (you have to install it into that container):

[pid   456] openat(AT_FDCWD, "/etc/group", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid   456] futex(0x55f6b8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid   457] <... futex resumed> )       = 0
[pid   457] clock_gettime(CLOCK_MONOTONIC, {tv_sec=2321, tv_nsec=911496659}) = 0
[pid   457] clock_gettime(CLOCK_MONOTONIC, {tv_sec=2321, tv_nsec=911587159}) = 0
[pid   456] write(1, "&errors.errorString{s:\"user: loo"..., 92 <unfinished ...>
[pid   457] futex(0xc00003e948, FUTEX_WAKE_PRIVATE, 1&errors.errorString{s:"user: lookup groupname thisdoesnotexist: no such file or directory"}

So it looks like my program is trying to read /etc/group and failing (a failure I would expect in the chroot). My understanding is /etc/group parse is fallback logic if cgo is not enabled, though I am compiling on Linux with a linux target, so cgo should be enabled... I don't know strace well enough to determine whether getgrnam_r is the one that is trying and failing to read /etc/group, ie. cgo is enabled but it's still failing to read the file, though I am concerned that I am getting different output than you got.

[kevinburke]

Odd... I copied the contents of os/user to a third party package so I could add print statements to it, and when I do that, it compiles and runs successfully.

$ docker run --tty -v $(PWD)/tmp:/go --interactive meterup/ubuntu-golang:latest /usr/local/go/bin/go build -ldflags '-extldflags "-static"' -o /go/test/test-program ./
# _/go
/tmp/go-link-139166518/000002.o: In function `mygetgrouplist':
/go/user2/getgrouplist_unix.go:16: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-139166518/000001.o: In function `mygetgrgid_r':
/go/user2/cgo_lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-139166518/000001.o: In function `mygetgrnam_r':
/go/user2/cgo_lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-139166518/000001.o: In function `mygetpwnam_r':
/go/user2/cgo_lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-139166518/000001.o: In function `mygetpwuid_r':
/go/user2/cgo_lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

# chroot /go/test /test-program
calling lookup Group using mygetgrnam_r
lookup err: <nil>
"thisdoesnotexist"
successful error handling

[anonymouse64]

I can share a reproducer environment when I get a chance, sorry been busy with other things... should be soon though

[mvo5]

Fwiw, I can easily reproduce this by installing the sssd package on Ubuntu which will add the sss to the nsswitch.conf. The fix for this seems fairly straightforward
0001-user-handle-ENOENT-from-getpw-uid-nam-_r.patch.gz

(tested the issue with sssd on 20.04 and 21.10 and the fix as well).

Alternatively the fmt.Errorf() return could use %w instead of the current %v to make it possible for the upper layer to reliably identify this.

[benhoyt]

I didn't realize the issue I opened and fixed recently (#67912) was -- I believe -- actually a duplicate of this one. And that my (previous) colleague @mvo5 had actually found the same repro (sss/nsswitch.conf) a couple of years earlier. :-) In any case, I think this issue can now be closed with my fix at aed1f92

[xnox]

Looks good to me, I think this issue can be closed.

[ianlancetaylor]

Thanks.