Skip to content

fix(docker): restore portable uv cache hardlinks - #5163

Closed
ko3n1g wants to merge 1 commit into
mainfrom
ko3n1g/fix/uv-cache-hardlinks-20261111
Closed

ko3n1g wants to merge 1 commit into
mainfrom
ko3n1g/fix/uv-cache-hardlinks-20261111

Conversation

@ko3n1g

@ko3n1g ko3n1g commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Background

The uv 0.13.6 upgrade in #4863 makes the shipped offline cache unextractable on the GCP H100 worker filesystem. Protected-main build 42890640535 published rl@sha256:51891dd230c2159cb6d51ef05414ca6910352319789c9ddd60f73f354a928907; containerd repeatedly fails to unpack layer sha256:5938d661b78b0d1f15680f99f35c927cb78f091ee3926bd15d001aa744c4d319 with too many links. The link source is an empty Python file and the destination is a FlashInfer cubin lock file under /opt/nemo_rl_cache/uv/archive-v0/.

The project enables all uv preview features. With uv 0.13.6, content-addressed caching deduplicates repeated empty files into one inode. A dependency-free reproduction containing 40,000 empty lock files produces an 80,003-link group across the cache and installed environment. The same wheel with uv 0.11.28 produces groups of at most two links.

What changed

Restore docker/Dockerfile's previous uv pin, 0.11.28. Its version already participates in both the BuildKit cache-mount identity and registry seed tag, so the rollback also selects the pre-upgrade cache namespace.

Details

The existing lockfile-aware cache pruning, shared offline cache, hardlink installations, runtime ownership, and cache export remain enabled. Returning to the earlier uv version trades the newer cache retention/deduplication behavior for portable image extraction.

This addresses the observed image extraction regression. A separate CI OCI build exited with code 128 without its final lifecycle marker; that cause remains unproved and is not claimed fixed here.

Tested

  • uv run --no-project --python .venv/bin/python -m pytest --noconftest -o addopts= tests/unit/tools/test_docker_hybridep_build.py tests/unit/tools/test_prune_uv_cache.py -q: 29 passed.
  • PRE_COMMIT_HOME=/tmp/pr3456-rl-uv-pre-commit-cache VIRTUALENV_APP_DATA=/tmp/pr3456-rl-uv-virtualenv-data uv run --no-project --python .venv/bin/python -m pre_commit run --files docker/Dockerfile: passed; applicable recipe minimization check passed and file-type-specific hooks reported no applicable files.
  • git diff --check: passed.
  • Cross-version installation on tmpfs, with UV_PREVIEW=1 and --link-mode hardlink: uv 0.13.6 → 80,003 links in the widest empty-file group; uv 0.11.28 → 2 links. Both installations completed.
  • OCI-style tar extraction of the measured groups onto local ext4: 0.13.6 failed with Too many links (exit 2); 0.11.28 passed (exit 0). This independently reproduces the containerd failure mechanism.
  • Original failing workload: inventory case in CI-Events-Workflows run 42901565410, original artifact 13814932991; original failure and normal cleanup result are preserved.

A rebuilt image still requires a successful extraction and runtime check. No full-image recovery is claimed by the local tests above.

Signed-off-by: oliver könig <okoenig@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 11, 2026

Copy link
Copy Markdown

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ko3n1g

ko3n1g commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 9fcf45b

@ko3n1g

ko3n1g commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

/review-approve reset=true model=claude

@ko3n1g ko3n1g added the CI:L1 Run doctests, unit tests, and functional tests label Nov 11, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rolling back to uv 0.11.28 looks right. I didn't find any part of the build or the runtime that needs a newer uv. The cache-mount ids and CI's seed tag both move to the 0.11.28 namespace together, as the description says.

  1. (C1) This is a non-blocking suggestion and not something this PR broke. The pin at docker/Dockerfile L162 doesn't say why it's held back, and no build step checks hardlink counts. That means the next uv bump could publish another image that containerd can't unpack.

I couldn't confirm offline that uv 0.11.28 reads a uv.lock last written by 0.13.6. Both files are revision = 3 (uv.lock L1-L2). If 0.11.28 can't read it, the build fails loudly at the first uv sync --frozen or at uv lock --check before anything is published. As the description notes, the rebuilt image still needs an extraction and runtime check.

Why I think the rollback is complete
Review coverage note

This review was split into three parallel areas. Only the area covering the late image layers and runtime code sent its report back. I checked the other two areas myself: the cache identity and CI seed tag, and the dependency layer with its cache helper and prune scripts. The list above includes both.

Review generated by claude. This was a full review. Request a full review with /review reset=true.

Comment thread docker/Dockerfile
# 65532) can execute them: /root is not traversable by other users, and the project
# venv's python is a symlink into uv's managed-Python directory.
ARG UV_VERSION=0.13.6
ARG UV_VERSION=0.11.28

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C1 · 2 action items. TL;DR — Nothing stops the next uv bump from bringing this failure back. preview = true still enables uv's content-addressed cache on newer uv, this pin doesn't say why it's held back, and no build step checks hardlink counts.

This gap predates the PR; the rollback itself is right. The broken image passed the CI build and was published. It only failed later, when containerd unpacked it on a worker. _remove_unlinked_files documents that preview = true turns on the content-addressed-cache feature. A routine bump of this line could therefore publish another image that can't be pulled, and CI wouldn't catch it.

AI-1

Explain the hold next to the pin, so whoever bumps it checks link counts first:

Suggested change
ARG UV_VERSION=0.11.28
# Held at 0.11.x on purpose. pyproject.toml sets `preview = true`, which on uv 0.13.6 turns on
# the content-addressed cache: identical files (every empty file included) in the cache and in
# all hardlinked venvs share one inode. That inode got more links than the node filesystem
# allows (ext4: 65000), and containerd could not unpack the image ("too many links").
# See #5163 before bumping.
ARG UV_VERSION=0.11.28

AI-2

Make the build fail when any file under /opt has more than 65000 hardlinks. Add this at the end of the root-owned RUN at L818-L834, just before its EOF. It can't be a suggestion block because that line isn't in the diff.

# containerd unpacks each layer with link(2), and ext4 allows at most 65000 links per inode.
# An image over that limit builds and pushes fine, then fails to pull with "too many links",
# so fail the build here instead.
too_many=$(find /opt -xdev -type f -links +65000 -print -quit)
if [[ -n "${too_many}" ]]; then
    echo "[ERROR] ${too_many} has more than 65000 hardlinks; ext4 hosts cannot unpack this image" >&2
    exit 1
fi

That RUN executes as root after every uv-writing RUN (L285, L444, L713, L766), so it sees every link group the release image ships. Leave that RUN's existing chown/chmod lines as they are.

Acceptance checks. I haven't run any of these.

  • The image build at this head should pass the check. With 0.11.28, the largest link group should be about 1 plus the number of venvs.
  • A build with --build-arg UV_VERSION=0.13.6 should stop at this check with the error above instead of publishing. I expect this from the extraction failure in the PR description.
  • A STRIP_UV_CACHE=1 build should still pass.
Why the in-build link count matches what containerd sees

containerd unpacks each layer into its own directory, so a file's links can only come from that same layer. Inside the build, a file from an earlier layer shows that layer's link count through overlayfs. The venvs are hardlinked to the cache in the same RUN that fills it (L338-L339, L409-L415), so find -links +65000 counts the same group the unpack recreates. This comes from reading the source; I haven't run it.

@ko3n1g

ko3n1g commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Additional evidence from CI-Events cluster-e2e: this PR also fixes the OCI build exit 128 noted under Background.

Image extraction on gcp-h100 (the (gpu, rl) / (inventory, rl) lanes)

  • In Cloud Logging, namespace nemo-ci-agents, there are 153 ErrImagePull … too many links events in 3 days.
  • They cover 7 RL main digests built after build(docker): prune the uv cache, keep Dynamo out of it, bump uv to 0.13.6 #4863 (96e54ae4, a5e107df, 4421f275, b0d2d09f, 51891dd2, 375770bc, 01791a9d), starting 2026-11-10 10:36Z.
  • Example from deploy run 42919707663: GPU smoke trainjob-debug-f95682bc36af scheduled at once, then hit failed to extract layer sha256:6cb803c6f26e … link …/opt/nemo_rl_cache/uv/archive-v0/…/pandas/_libs/window/__init__.py → …/flashinfer_cubin/…cubin.lock: too many links from 00:52:53Z. It looped ImagePullBackOff until the 5400 s deadline, with the pod showing as Pending the whole time.
  • The last good image, fde4271c (RL ec7f3fd5), predates build(docker): prune the uv cache, keep Dynamo out of it, bump uv to 0.13.6 #4863.

OCI validation build (the (build, rl) lane, gVisor BuildKit)

  • Every RL build since build(docker): prune the uv cache, keep Dynamo out of it, bump uv to 0.13.6 #4863 dies in [hermetic 10/15] … uv sync right after Prepared 273 packages. The sandbox exits 128, with no BuildKit ERROR: line and no stage end marker.
  • Builds: oci-2c1bcbef…, oci-42535135…, oci-998cc612…, oci-4efad7fb…, oci-b6239d81…, oci-c85db6db…, oci-65309f77….
  • The last uv 0.11.28 build (oci-90d494c3…) printed Installed 273 packages in 3.69s for the same lock revision and pushed.
  • This happens at the hardlink-mode install step, where uv 0.13.6's content-addressed cache creates the oversized link groups described above. The 0.11.28 pin in this PR restores the configuration that has built cleanly every time.

@ko3n1g
ko3n1g requested a review from kajalj22 November 11, 2026 08:43
@ko3n1g ko3n1g closed this Nov 11, 2026
@ko3n1g
ko3n1g removed the request for review from kajalj22 November 11, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants