[CI]: gomodjail target catch-all issues

[apostasie]

The newly introduced gomodjail target likely has a few rough edges that we need to blunt.

This here is to discuss them in one place without overloading other PRs.

List of currently tracked issues:

• gomodjail signal handling is likely problematic AkihiroSuda/gomodjail#51
• FIXED: unpacking needs flock AkihiroSuda/gomodjail#50

[apostasie]

@AkihiroSuda Is it possible that XDG_CACHE_HOME gets garbage collected by some automated system? systemd-tmpfiles?

[AkihiroSuda]

Is it possible that XDG_CACHE_HOME gets garbage collected by some automated system? systemd-tmpfiles?

I don't think so.
Maybe we need sync() after close(fd) and rename(tmp, "go.mod") ? 🤔
https://github.com/AkihiroSuda/gomodjail/blob/v0.1.1/pkg/ziputil/ziputil.go#L123
https://github.com/AkihiroSuda/gomodjail/blob/v0.1.1/pkg/ziputil/ziputil.go#L133

[apostasie]

Intuition: flock.
Unless I missed it, this does not look like this is safe to use concurrently: https://github.com/AkihiroSuda/gomodjail/blob/v0.1.1/pkg/ziputil/ziputil.go#L102-L135

[apostasie]

Is it possible that XDG_CACHE_HOME gets garbage collected by some automated system? systemd-tmpfiles?

I don't think so. Maybe we need sync() after close(fd) and rename(tmp, "go.mod") ? 🤔 https://github.com/AkihiroSuda/gomodjail/blob/v0.1.1/pkg/ziputil/ziputil.go#L123 https://github.com/AkihiroSuda/gomodjail/blob/v0.1.1/pkg/ziputil/ziputil.go#L133

I have seen sync being used in eg: https://github.com/containerd/continuity/blob/main/ioutils.go
It makes sense (only) if you care about the data and want to provide guarantees that it will survive catastrophic failure (eg: brutal power loss).

In the case here (where we don't care), I do no think this serves a purpose and that would just (slightly) slow things down.

[apostasie]

Linking in AkihiroSuda/gomodjail#51

[apostasie]

Update: FIXED

@AkihiroSuda here is a new one (redacting the gocap noise):

=== RUN   TestTailFollowRotateLogs
    container_logs_test.go:341: assertion failed: true (true bool) != false (len(tailLogs) > linesPerFile bool): 

        time=2025-05-02T19:47:49.276Z level=WARN msg=***Blocked*** pid=59939 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=inotify_init1 entry=/go/pkg/mod/github.com/fsnotify/fsnotify@v1.9.0/backend_inotify.go:141:github.com/fsnotify/fsnotify.newBackend module=github.com/fsnotify/fsnotify
        time=2025-05-02T19:47:49.276Z level=WARN msg=***Blocked*** pid=59939 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/github.com/fsnotify/fsnotify@v1.9.0/backend_inotify.go:150:github.com/fsnotify/fsnotify.newBackend module=github.com/fsnotify/fsnotify
        time=2025-05-02T19:47:49.276Z level=WARN msg=***Blocked*** pid=59939 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=inotify_add_watch entry=/go/pkg/mod/github.com/fsnotify/fsnotify@v1.9.0/backend_inotify.go:265:github.com/fsnotify/fsnotify.(*inotify).AddWith.(*inotify).AddWith.func1.(*inotify).register.func3 module=github.com/fsnotify/fsnotify
        time="2025-05-02T19:47:49Z" level=fatal msg="read : bad file descriptor"
        time=2025-05-02T19:47:49.278Z level=ERROR msg="exiting with an error" error="exit code 1" exitCode=1

Presumably we should unconfine fsnotify?

Note: what I find confusing here is that this seems to only happen occasionally.
Why?
Maybe @fahedouch would know?

[apostasie]

UPDATE: FIXED

We probably need to unconfine cdi as well:

=== FAIL: cmd/nerdctl/container TestRunDeviceCDIWithNerdctlConfig (re-run 2) (0.13s)
    container_run_linux_test.go:701: assertion failed: 
        Command:  /usr/local/bin/nerdctl.gomodjail --namespace=nerdctl-test run --rm --device vendor1.com/device=foo ghcr.io/stargz-containers/alpine:3.13-org env
        ExitCode: 2
        Error:    exit status 2
        Stdout:   
        Stderr:   time=2025-05-03T17:10:11.595Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=openat entry=/go/pkg/mod/github.com/syndtr/gocapability@v0.0.0-20200815063812-42c35b437635/capability/capability_linux.go:53:github.com/syndtr/gocapability/capability.initLastCap module=github.com/syndtr/gocapability
        time=2025-05-03T17:10:11.595Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/github.com/syndtr/gocapability@v0.0.0-20200815063812-42c35b437635/capability/capability_linux.go:53:github.com/syndtr/gocapability/capability.initLastCap module=github.com/syndtr/gocapability
        time=2025-05-03T17:10:11.595Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/github.com/syndtr/gocapability@v0.0.0-20200815063812-42c35b437635/capability/capability_linux.go:53:github.com/syndtr/gocapability/capability.initLastCap module=github.com/syndtr/gocapability
        time=2025-05-03T17:10:11.596Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/github.com/syndtr/gocapability@v0.0.0-20200815063812-42c35b437635/capability/capability_linux.go:53:github.com/syndtr/gocapability/capability.initLastCap module=github.com/syndtr/gocapability
        time=2025-05-03T17:10:11.615Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=openat entry=/go/pkg/mod/github.com/syndtr/gocapability@v0.0.0-20200815063812-42c35b437635/capability/capability_linux.go:53:github.com/syndtr/gocapability/capability.initLastCap module=github.com/syndtr/gocapability
        time=2025-05-03T17:10:11.615Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/github.com/syndtr/gocapability@v0.0.0-20200815063812-42c35b437635/capability/capability_linux.go:53:github.com/syndtr/gocapability/capability.initLastCap module=github.com/syndtr/gocapability
        time=2025-05-03T17:10:11.615Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/github.com/syndtr/gocapability@v0.0.0-20200815063812-42c35b437635/capability/capability_linux.go:53:github.com/syndtr/gocapability/capability.initLastCap module=github.com/syndtr/gocapability
        time=2025-05-03T17:10:11.615Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/github.com/syndtr/gocapability@v0.0.0-20200815063812-42c35b437635/capability/capability_linux.go:53:github.com/syndtr/gocapability/capability.initLastCap module=github.com/syndtr/gocapability
        time=2025-05-03T17:10:11.700Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=openat entry=/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/spec-dirs.go:106:tags.cncf.io/container-device-interface/pkg/cdi.scanSpecDirs module=tags.cncf.io/container-device-interface
        time=2025-05-03T17:10:11.701Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/spec-dirs.go:106:tags.cncf.io/container-device-interface/pkg/cdi.scanSpecDirs module=tags.cncf.io/container-device-interface
        time=2025-05-03T17:10:11.701Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/spec-dirs.go:106:tags.cncf.io/container-device-interface/pkg/cdi.scanSpecDirs module=tags.cncf.io/container-device-interface
        time=2025-05-03T17:10:11.701Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=fcntl entry=/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/spec-dirs.go:106:tags.cncf.io/container-device-interface/pkg/cdi.scanSpecDirs module=tags.cncf.io/container-device-interface
        time=2025-05-03T17:10:11.702Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=getdents64 entry=/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/spec-dirs.go:106:tags.cncf.io/container-device-interface/pkg/cdi.scanSpecDirs module=tags.cncf.io/container-device-interface
        time=2025-05-03T17:10:11.703Z level=WARN msg=***Blocked*** pid=227083 exe=/home/rootless/.cache/gomodjail/1f381f194c295770/nerdctl syscall=flock entry=/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/spec-dirs.go:106:tags.cncf.io/container-device-interface/pkg/cdi.scanSpecDirs module=tags.cncf.io/container-device-interface
        panic: runtime error: slice bounds out of range [:227083] with capacity 8192
        
        goroutine 1 [running]:
        os.(*File).readdir(0xc00012c440, 0xffffffffffffffff, 0x0)
        	/usr/local/go/src/os/dir_unix.go:92 +0xc66
        os.(*File).Readdirnames(0xc000047340?, 0x38?)
        	/usr/local/go/src/os/dir.go:73 +0x1a
        path/filepath.readDirNames({0xc000047340?, 0xc00001ec68?})
        	/usr/local/go/src/path/filepath/path.go:442 +0x3f
        path/filepath.walk({0xc000047340, 0x38}, {0x15d1b10, 0xc00001ec30}, 0xc000116448)
        	/usr/local/go/src/path/filepath/path.go:348 +0x68
        path/filepath.Walk({0xc000047340, 0x38}, 0xc000116448)
        	/usr/local/go/src/path/filepath/path.go:427 +0x66
        tags.cncf.io/container-device-interface/pkg/cdi.scanSpecDirs({0xc00079a620?, 0x1, 0x13c3254?}, 0xc0001166e8)
        	/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/spec-dirs.go:77 +0xcc
        tags.cncf.io/container-device-interface/pkg/cdi.(*Cache).refresh(0xc00079caa0)
        	/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/cache.go:174 +0x171
        tags.cncf.io/container-device-interface/pkg/cdi.(*Cache).configure(0xc00079caa0, {0xc000116948, 0x2, 0x2800000000000040?})
        	/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/cache.go:120 +0x145
        tags.cncf.io/container-device-interface/pkg/cdi.newCache({0xc000116948, 0x2, 0x2})
        	/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/cache.go:87 +0x1bf
        tags.cncf.io/container-device-interface/pkg/cdi.getOrCreateDefaultCache.func1()
        	/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/default-cache.go:33 +0x28
        sync.(*Once).doSlow(0x472499?, 0x20?)
        	/usr/local/go/src/sync/once.go:78 +0xab
        sync.(*Once).Do(...)
        	/usr/local/go/src/sync/once.go:69
        tags.cncf.io/container-device-interface/pkg/cdi.getOrCreateDefaultCache({0xc000116948?, 0xf9cb9e?, 0xc0007a66f0?})
        	/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/default-cache.go:32 +0x6a
        tags.cncf.io/container-device-interface/pkg/cdi.Configure({0xc000116948, 0x2, 0x2})
        	/go/pkg/mod/tags.cncf.io/container-device-interface@v1.0.1/pkg/cdi/default-cache.go:48 +0x25
        github.com/containerd/nerdctl/v2/pkg/cmd/container.Create.withCDIDevices.func3({0x15cd148, 0xc000792390}, {0x15bf920, 0xc0001a6ea0}, 0xc0007823c0, 0xc000373170)
        	/go/src/github.com/containerd/nerdctl/pkg/cmd/container/run_cdi.go:39 +0xbe
        github.com/containerd/containerd/v2/pkg/oci.ApplyOpts({0x15cd148, 0xc000792390}, {0x15bf920, 0xc0001a6ea0}, 0xc0007823c0, 0xc000373170, {0xc0003de0e0, 0x19, 0x0?})
        	/go/pkg/mod/github.com/containerd/containerd/v2@v2.1.0-rc.0/pkg/oci/spec.go:110 +0x89
        github.com/containerd/nerdctl/v2/pkg/cmd/container.Create.WithSpec.func30({0x15cd148?, 0xc000792390?}, 0xc0001a6ea0?, 0xc0007823c0)
        	/go/pkg/mod/github.com/containerd/containerd/v2@v2.1.0-rc.0/client/container_opts.go:318 +0x48
        github.com/containerd/containerd/v2/client.(*Client).NewContainer(0xc0001a6ea0, {0x15cd180?, 0xc000434140?}, {0xc000047d80, 0x40}, {0xc000277c80, 0xa, 0x0?})
        	/go/pkg/mod/github.com/containerd/containerd/v2@v2.1.0-rc.0/client/client.go:309 +0x283
        github.com/containerd/nerdctl/v2/pkg/cmd/container.Create({_, _}, _, {_, _, _}, {_, _}, {{0x15bea80, 0xc000192038}, ...})
        	/go/src/github.com/containerd/nerdctl/pkg/cmd/container/create.go:361 +0x329f
        github.com/containerd/nerdctl/v2/cmd/nerdctl/container.runAction(0xc00046c908, {0xc0004[3400](https://github.com/containerd/nerdctl/actions/runs/14812695121/job/41589442003?pr=4191#step:12:3401)0, 0x2, 0x5})
        	/go/src/github.com/containerd/nerdctl/cmd/nerdctl/container/container_run.go:380 +0x40c
        github.com/spf13/cobra.(*Command).execute(0xc00046c908, {0xc0001ba030, 0x5, 0x5})
        	/go/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1015 +0xa94
        github.com/spf13/cobra.(*Command).ExecuteC(0xc00046c308)
        	/go/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1148 +0x40c
        github.com/spf13/cobra.(*Command).Execute(...)
        	/go/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1071
        main.xmain()
        	/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:145 +0x8f
        main.main()
        	/go/src/github.com/containerd/nerdctl/cmd/nerdctl/main.go:128 +0x13
        time=2025-05-03T17:10:11.706Z level=ERROR msg="exiting with an error" error="exit code 2" exitCode=2
        
        
        Failures:
        ExitCode was 2 expected 0
        Expected stdout to contain "FOO=injected"
        Expected no error