[devchris123]

I intentionally kept the commits as three commits for easier review. I can later squash with a commit message mentioning the issue as mentioned in the guidelines.

Problem

#29555 reported an
issue with memory-backed emptyDir volumes used by podman kube play.
#29565 addressed
sharing the tmpfs between regular containers that are running concurrently.
#29670 shows the
remaining case: data written by a one-shot init container is unavailable to a
regular container that starts afterward.

Root cause

A memory-backed emptyDir is represented by an anonymous named volume. Podman
mounts a tmpfs at the volume's host-side mountpoint and bind-mounts that path
into the init container.

When a one-shot init container is removed, its volume mounts are released. If
this reduces the volume mount reference count to zero, Podman unmounts the
host-side tmpfs and its contents are lost.

Approach

The change adds an internal mount of every memory-backed emptyDir volume to
the pod's infra container.

Infra-container mounts are normally inherited by containers subsequently added
to the pod. That behavior supports pod-wide mounts such as:

podman pod create --name mypod --volume myvolume:/data
podman run --pod mypod alpine ls /data

I used an additional NoInherit option to both specgen.NamedVolume and libpod.ContainerNamedVolume work around this.

Fixes: #29670
Related: #29555

Checklist

Ensure you have completed the following checklist for your pull request to be reviewed:

• I have read and understood our contributing guidelines and will not have more than two open PRs as a new contributor.
• PR description, commit message, and GitHub comments are human-written, per LLM Policy
• Certify you wrote the patch or otherwise have the right to pass it on as an open-source patch by signing all
  commits. (git commit -s). (If needed, use git commit -s --amend). The author email must match
  the sign-off email address. See CONTRIBUTING.md
  for more information.
• Referenced issues using Fixes: #00000 in commit message (if applicable)
• Tests have been added/updated (or no tests are needed)
• Documentation has been updated (or no documentation changes are needed)
• All commits pass make validatepr (format/lint checks)
• Release note entered in the section below (or None if no user-facing changes)

Does this PR introduce a user-facing change?

Fixed a bug in podman kube play that caused data written by an init container to a memory-backed emptyDir volume to be lost when the init container exited. 
Regular containers in the pod can now access that data as expected. 
See [podman-kube-play(1)](https://docs.podman.io/en/latest/markdown/podman-kube-play.1.html).

[packit-as-a-service]

[NON-BLOCKING] Packit jobs failed. @podman-container-tools/packit-jobs please check. Everyone else, feel free to ignore.

[devchris123]

Hello @magic-peach,
since you opened the initial PR on this issue, would you mind giving your opinion on this?

[TomSweeneyRedHat]

LGTM

From a code standpoint. Could you clean up the Title please a bit, at least remove your branch name?

thanks @devchris123

[devchris123]

@TomSweeneyRedHat

I fixed the PR title and squash down to one commit.

Thanks for your review.

[devchris123]

@TomSweeneyRedHat
One test failed after my force push. Could you retry that one?

[Honny1]

Code, LGTM.

I am not convinced the approach mentioned in #24930 (comment) is better then implemented aproach. WDYT? @mheon @Luap99

[devchris123]

@Honny1
An alternative approach might be to make the volume lifetime for this particular type of volume explicit. That would require persisting the volume as part of the pod's persistence and tie the lifecycle to the pods lifecycle. E.g. mount the tmpfs on the host upon pod creation and unmount it upon pod removal. But a couple of other failure and edge cases must be considered to ensure the tmpfs is properly cleaned up on partial failures etc.

There are some other problems I found with the current implemenation, which might need fixing in additional commits:

• a tmpfs based PVC also loses data from init-to-app containers
  • I tested this with podman volume create --opts type=tmpfs <vol-name>
  • start pod with init and app container
  • write data to the volume in an init container
  • read from the volume from an app container -> data gone
• a pod stop with an image-backed infra container will stop the container and then also unmount the tmpfs effectively losing the data, whereas the default image infra container does not have this behaviour. This is likely incidental
• first-use initialization of a FS could lead to unexpected results, because the infra container touches the volume first (this is not clear to a user) -> this needs further investigation, I haven't checked what mechanism exist

This could be done in this PR or a follow up PR, not inherently blockers I would say.
This PR does not introduce these issues (maybe only change the first-use initialization to some extent, but this remains to be checked)

[Honny1]

@l0rd

[mheon]

This isn't the right approach. We should use an anonymous named volume using mount options to make it a tmpfs (from the command line, this would be podman volume create --opt type=tmpfs --opt device=tmpfs myvol). This will mimic the K8S behavior properly and doesn't require us to hack in an extra directory for Kubernetes temporary volumes, with all the lifecycle problems that brings

[devchris123]

Hello @mheon,
we are already using an anonymous named volume, this was introduced in #29565.
The issue is that the tmpfs mount is not tied to the lifetime of the named volume, which it also shouldn't, otherwise the inmemory filesystem would potentially and unexpectedly outlive pods that reference that volume.

The mounted tmpfs itself is currently tied to the container count currently using the volume. Inbetween the stop of an init container and the start of an app container the tmpfs is unmounted and data is lost.

The infra container uses the existing mount/umount machinery based on usage count and keeps the mount count above 0 during the pod lifetime. E.g. it acts as a proxy for the pod lifetime.

[devchris123]

Hello @Honny1 @l0rd,

do you have an opinion on how we should move forward with this?

If you believe this approach is correct, the PR is basically ready. I would deliver the follow up fixes for the PVC and image-backed infra container in a separate PR.

[Honny1]

+1 to @mheon aproach.

[devchris123]

@mheon
It is not clear to me how this solves the problem. We are already using an anonymous named volume.
The problem is that as long as no container references that volume, no mount is happening and since there is no backing directory as in the case of host directories, the data does not survive inbetween init containers and regular containers, because the container count goes to zero.

The current implementation is keeping the mount alive for the lifetime of the infra container as a proxy for the pod lifetime.

Alternatively we would need to tie the lifetime of the tmpfs to the pod itself, but this requires a larger rearchitecturing. Is that what you are suggesting here?

Bottom line the tmpfs mount must survive the lifetime of the pod, but at the same time not exist any longer than that.
Other volumes do not have this problem because the data exists on the host filesystem regardless if mounted to a container or not.

[Honny1]

PTAL @mheon @podman-container-tools/podman-maintainers

[mheon]

If the concern is about init containers, we should be able to paper over the cracks by mounting it manually via API, starting the pod, then unmounting it once the pod is started. Mount count will remain at 1, the volume will not be unmounted until start is complete, and we don't need to scatter untracked mounts in our filesystem