Repository navigation
Conversation
Signed-off-by: oliver könig <okoenig@nvidia.com>
|
Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test 9fcf45b |
|
/review-approve reset=true model=claude |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
Rolling back to uv 0.11.28 looks right. I didn't find any part of the build or the runtime that needs a newer uv. The cache-mount ids and CI's seed tag both move to the 0.11.28 namespace together, as the description says.
- (C1) This is a non-blocking suggestion and not something this PR broke. The pin at
docker/DockerfileL162 doesn't say why it's held back, and no build step checks hardlink counts. That means the next uv bump could publish another image that containerd can't unpack.
I couldn't confirm offline that uv 0.11.28 reads a uv.lock last written by 0.13.6. Both files are revision = 3 (uv.lock L1-L2). If 0.11.28 can't read it, the build fails loudly at the first uv sync --frozen or at uv lock --check before anything is published. As the description notes, the rebuilt image still needs an extraction and runtime check.
Why I think the rollback is complete
- Cache identity: every
id=uv-cache-${TARGETARCH}-${BASE_IMAGE}-${UV_VERSION}-${HYBRID_EP_MULTINODE}mount uses the same id: L285, L444, L713, L766 and L868. CI's seed tag reads the onlyARG UV_VERSION=default (L162) throughresolve_argin_build_container.ymlL236-L264. By reading the source, I expecttest_uv_cache_mount_id_matches_build_container_workflowto keep passing; I didn't run it. - uv commands: the dependency layer, the TRT-LLM layer, the release stage,
nemo_rl/utils/venvs.py,tools/build-custom-vllm.shandtools/build-custom-flashinfer.shonly use long-standing flags. The config already has a 0.11-compatible spelling:pyproject.tomlL588-L590 writesnemo_gymbecause "uv < 0.12 compares conflict entries against unnormalized extra names". - Cache pruning:
docker/prune_uv_cache.pyonly acts on the.okcheckout markers and thefiles-v*objects when they exist. With 0.11.28's cache layout those steps do nothing. - Other pins: no test or doc pins 0.13.6.
Dockerfile.ngc_pytorchL48 has its own0.11.18pin, which this PR doesn't touch.
Review coverage note
This review was split into three parallel areas. Only the area covering the late image layers and runtime code sent its report back. I checked the other two areas myself: the cache identity and CI seed tag, and the dependency layer with its cache helper and prune scripts. The list above includes both.
Review generated by claude. This was a full review. Request a full review with /review reset=true.
| # 65532) can execute them: /root is not traversable by other users, and the project | ||
| # venv's python is a symlink into uv's managed-Python directory. | ||
| ARG UV_VERSION=0.13.6 | ||
| ARG UV_VERSION=0.11.28 |
There was a problem hiding this comment.
C1 · 2 action items. TL;DR — Nothing stops the next uv bump from bringing this failure back. preview = true still enables uv's content-addressed cache on newer uv, this pin doesn't say why it's held back, and no build step checks hardlink counts.
This gap predates the PR; the rollback itself is right. The broken image passed the CI build and was published. It only failed later, when containerd unpacked it on a worker. _remove_unlinked_files documents that preview = true turns on the content-addressed-cache feature. A routine bump of this line could therefore publish another image that can't be pulled, and CI wouldn't catch it.
AI-1
Explain the hold next to the pin, so whoever bumps it checks link counts first:
| ARG UV_VERSION=0.11.28 | |
| # Held at 0.11.x on purpose. pyproject.toml sets `preview = true`, which on uv 0.13.6 turns on | |
| # the content-addressed cache: identical files (every empty file included) in the cache and in | |
| # all hardlinked venvs share one inode. That inode got more links than the node filesystem | |
| # allows (ext4: 65000), and containerd could not unpack the image ("too many links"). | |
| # See #5163 before bumping. | |
| ARG UV_VERSION=0.11.28 |
AI-2
Make the build fail when any file under /opt has more than 65000 hardlinks. Add this at the end of the root-owned RUN at L818-L834, just before its EOF. It can't be a suggestion block because that line isn't in the diff.
# containerd unpacks each layer with link(2), and ext4 allows at most 65000 links per inode.
# An image over that limit builds and pushes fine, then fails to pull with "too many links",
# so fail the build here instead.
too_many=$(find /opt -xdev -type f -links +65000 -print -quit)
if [[ -n "${too_many}" ]]; then
echo "[ERROR] ${too_many} has more than 65000 hardlinks; ext4 hosts cannot unpack this image" >&2
exit 1
fiThat RUN executes as root after every uv-writing RUN (L285, L444, L713, L766), so it sees every link group the release image ships. Leave that RUN's existing chown/chmod lines as they are.
Acceptance checks. I haven't run any of these.
- The image build at this head should pass the check. With 0.11.28, the largest link group should be about 1 plus the number of venvs.
- A build with
--build-arg UV_VERSION=0.13.6should stop at this check with the error above instead of publishing. I expect this from the extraction failure in the PR description. - A
STRIP_UV_CACHE=1build should still pass.
Why the in-build link count matches what containerd sees
containerd unpacks each layer into its own directory, so a file's links can only come from that same layer. Inside the build, a file from an earlier layer shows that layer's link count through overlayfs. The venvs are hardlinked to the cache in the same RUN that fills it (L338-L339, L409-L415), so find -links +65000 counts the same group the unpack recreates. This comes from reading the source; I haven't run it.
|
Additional evidence from CI-Events cluster-e2e: this PR also fixes the OCI build exit 128 noted under Background. Image extraction on gcp-h100 (the
OCI validation build (the
|
Background
The uv 0.13.6 upgrade in #4863 makes the shipped offline cache unextractable on the GCP H100 worker filesystem. Protected-main build 42890640535 published
rl@sha256:51891dd230c2159cb6d51ef05414ca6910352319789c9ddd60f73f354a928907; containerd repeatedly fails to unpack layersha256:5938d661b78b0d1f15680f99f35c927cb78f091ee3926bd15d001aa744c4d319withtoo many links. The link source is an empty Python file and the destination is a FlashInfer cubin lock file under/opt/nemo_rl_cache/uv/archive-v0/.The project enables all uv preview features. With uv 0.13.6, content-addressed caching deduplicates repeated empty files into one inode. A dependency-free reproduction containing 40,000 empty lock files produces an 80,003-link group across the cache and installed environment. The same wheel with uv 0.11.28 produces groups of at most two links.
What changed
Restore
docker/Dockerfile's previous uv pin,0.11.28. Its version already participates in both the BuildKit cache-mount identity and registry seed tag, so the rollback also selects the pre-upgrade cache namespace.Details
The existing lockfile-aware cache pruning, shared offline cache, hardlink installations, runtime ownership, and cache export remain enabled. Returning to the earlier uv version trades the newer cache retention/deduplication behavior for portable image extraction.
This addresses the observed image extraction regression. A separate CI OCI build exited with code 128 without its final lifecycle marker; that cause remains unproved and is not claimed fixed here.
Tested
uv run --no-project --python .venv/bin/python -m pytest --noconftest -o addopts= tests/unit/tools/test_docker_hybridep_build.py tests/unit/tools/test_prune_uv_cache.py -q: 29 passed.PRE_COMMIT_HOME=/tmp/pr3456-rl-uv-pre-commit-cache VIRTUALENV_APP_DATA=/tmp/pr3456-rl-uv-virtualenv-data uv run --no-project --python .venv/bin/python -m pre_commit run --files docker/Dockerfile: passed; applicable recipe minimization check passed and file-type-specific hooks reported no applicable files.git diff --check: passed.UV_PREVIEW=1and--link-mode hardlink: uv 0.13.6 → 80,003 links in the widest empty-file group; uv 0.11.28 → 2 links. Both installations completed.Too many links(exit 2); 0.11.28 passed (exit 0). This independently reproduces the containerd failure mechanism.13814932991; original failure and normal cleanup result are preserved.A rebuilt image still requires a successful extraction and runtime check. No full-image recovery is claimed by the local tests above.