Repository navigation
build(docker): prune the uv cache, keep Dynamo out of it, bump uv to 0.13.6 - #4863
Conversation
The release image's shipped uv cache (UV_CACHE_DIR) is append-only: uv.lock's own cache-seed key is BASE_IMAGE+UV_VERSION+HYBRID_EP_MULTINODE, not the lock content, so every routine dependency-version bump (torch, vLLM, SGLang, ...) leaves its predecessor's wheel behind forever as long as that key stays the same -- which is most of the time. Add docker/prune_uv_cache.py, run once after all uv-writing RUNs finish and before the ownership handoff. It parses uv.lock for every (name, version) pin across all extras (installed in this build or not, so it cannot break the offline `uv run --extra <backend>` promise UV_CACHE_DIR exists for) and removes only cache entries nothing in the current lock still references. Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test a8d69c9 |
update-uv-cache and update-uv-cache-gb200 had no status function in their if:, so GitHub implied success(), which also requires every transitive need to succeed. Since cicd-wait-in-queue (and later prepare-megatron-lock) became needs of the build jobs, and both are skipped on main pushes, the refresh has been skipped on every main run. The seed went stale, its cache key no longer matches, and every build that reruns the dependency layer starts from an empty uv cache. Use the always() ... !cancelled() pattern the build jobs already use, with explicit result checks on the jobs these depend on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test 55dd90f |
…hives - Garbage-collect archive-v0 by scanning every remaining link under wheels-v*/ and sdists-v*/ (as uv cache prune does), instead of only kept pypi entries. The old scan deleted archives still referenced by index/, url/ and built-from-source entries, leaving dangling links that break offline installs of e.g. torch. - Prune alternative-index trees (index/<digest>/, used for torch and flashinfer) by (name, version), not just pypi/. - Prune git source builds and checkouts whose commit uv.lock no longer pins. - Judge each cache file by its own parsed version; the old prefix glob removed 1.0.1's files when pruning 1.0. - Keep the truncated form of long versions, matching uv's wheel cache keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test f2d2057 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
Reviewed all five changed files and traced the affected CI, Docker, cache, and test paths. Refreshing the seed addresses a useful problem, but the pruning integration has build-compatibility and image-size defects, plus two edge cases that delete still-pinned cache entries. There is also one non-blocking type-check coverage suggestion.
Validation was static only: tests, Docker builds, linters, and live GitHub Actions were not executed. Findings were checked against the inspected callers and regression tests; no independent teammate review was available.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Deleting cache files in a later RUN only hides them behind whiteouts; the layers that wrote them still ship every byte. The release-stage prune therefore left the image size unchanged (61 vs 60 layers, the extra one ~0 bytes). - Prune at the end of the dependency RUN, which seeds and fills the cache, so superseded versions never get committed to a layer. - Install Dynamo with a throwaway UV_CACHE_DIR: it syncs its own Python 3.12 lock into a --link-mode copy venv that nothing re-syncs at runtime, so its wheels were stored twice. - Drop the release-stage prune RUN, and add a test that pins the prune to the dependency RUN and the Dynamo install to a throwaway cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test a6378c1 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
Follow-up review against f2d2057, with current inspection of all five changed files.
The same-layer dependency prune addresses the earlier finding about shipping superseded seed bytes. The Dynamo installation now uses a temporary cache, consistent with its copy-mode environment.
Four prior findings remain: source-ref build compatibility, two pinned-cache deletion cases, and one non-blocking type-check coverage omission. They are detailed inline. Subsequent changes and their directly affected Docker, cache, and test paths were checked; no additional escalation-eligible regression was established.
Validation was static only: tests, Docker builds, pre-commit, and GitHub Actions were not executed. No independent teammate review was available. Suggested edits and acceptance checks require author verification.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
Pruning at the end of the dependency RUN also deleted the build-isolation deps the syncs resolve outside uv.lock (setuptools, setuptools-scm, poetry-core, a build-time torch 2.14.1 + CUDA wheels, ...), which the TRT-LLM step then downloaded again. Run the prune right after the seed is copied, so it only judges what older locks left behind. Also, from review: - copy the script from the recipe's context, so building an older NRL_GIT_REF that lacks it still works - compare sdist version directories verbatim instead of stripping a metadata suffix from the version - keep archives of a package named `src` - type-check the script with pyrefly Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test 0d8214e |
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
LGTM from static review of all six changed files and their directly affected CI, Docker, cache, and test paths. No actionable findings remain.
The four findings from the latest review are addressed: the helper now comes from the recipe context, sdist version directories are compared verbatim, packages named src remain traversable, and the helper is included in type-check coverage. The pruning placement and relevant regression assertions were also inspected.
Validation was static only: tests, Docker builds, pre-commit, and live GitHub Actions were not executed. No independent teammate review was available.
Review generated by codex. Full review with RESET=true.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
main (#4340) replaced the update-uv-cache jobs with an in-build cache mount and seed export, so this branch drops its workflow change and takes main's. - Prune right after `uv_cache_restore force`, before any sync; the RUN's `uv_cache_persist prune` trap mirrors the result into the cache mount. - Dynamo keeps its throwaway UV_CACHE_DIR, so its RUN no longer needs the cache mount or the restore/persist helpers. - Placement test and docs updated for the new flow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test cbbf92d |
main (#4340) replaced the update-uv-cache jobs, so retarget the test at the new mechanism: both builds use the uv-cache-test tag prefix, and the "Update uv cache image" step also runs on this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
The uv cache mount id and the seed tag are now keyed by HYBRID_EP_MULTINODE, and deep_ep's match-runtime torch pin is part of uv's built-wheel cache key, so a cached DeepEP build can no longer be reused for the wrong multinode setting or torch ABI. The clean only forced a rebuild on every build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
…ile-prune Signed-off-by: Kajal Jain <kajalj@nvidia.com> # Conflicts: # pyrefly.toml
|
/ok to test 32e57ac |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
LGTM for this follow-up. I compared the changes since the previously reviewed head, inspected all six changed files, and consolidated three delegated reviews. No qualifying new regressions were established.
The restored-cache prune remains before dependency syncs and within the layer importing the cache. The persistence helper mirrors those deletions. The previously verified recipe-context helper acquisition, sdist-version and package-named-src regression cases, and type-check inclusion remain intact.
Validation was static only: tests, Docker builds, pre-commit, and live CI were not executed. External uv cache-key internals were not independently verified because their source isn't mounted.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
uv 0.13 keeps git and no-build-isolation wheels (deep-ep, deep-gemm, mamba-ssm, ...) through `uv cache prune --ci`, so the exported seed no longer forces those source builds on every rebuild. With `preview = true`, 0.13 also enables the content-addressed cache: archive files are hardlinks into files-v0/. Adapt to it: - prune_uv_cache.py drops files-v0 objects nothing links to any more (link count 1, uv's own rule) and the new <sha>.ok checkout markers; otherwise removing an archive frees nothing. - rsync the cache with -H so hardlinks survive the restore/persist round trip instead of doubling the cache. Tests move to the 0.13 bucket layout, cover the files-v0 cleanup, and allowlist the fake commit SHAs for the secrets detector. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test e8e3338 |
|
/review reset=true |
@kajalj22, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
I reviewed all six changed files and their directly affected Docker, cache, and test paths. No blocking correctness issue was established. C1 is one non-blocking guideline suggestion.
The recipe-context helper acquisition, same-layer pruning before dependency syncs, and sdist-version and package-named-src regressions address the earlier concrete failures.
Validation was static only: tests, Docker builds, pre-commit, and live CI were not executed. External uv cache-layout and cache-key behavior could not be independently verified from mounted upstream source. Three review topics were delegated; the pin-selection child's final report wasn't delivered, so I inspected its pin-selection and CLI scope directly.
Review generated by codex. Full review with RESET=true.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
The Dockerfile already explains why the prune runs where it does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test e4dce7e |
…ile-prune Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test b8ad614 |
1 similar comment
|
/ok to test b8ad614 |
|
/review-approve |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
LGTM for this follow-up. I compared the current head with the previously reviewed head, inspected all six changed files and their directly affected paths, and found no unresolved prior finding or qualifying new regression.
Prior C1 is fixed: the placement test now uses the one-line invariant, while its ordering, recipe-context, persistence, and Dynamo assertions remain intact.
The follow-up inspection covered the uv upgrade, hardlink-preserving cache transfers, checkout-marker cleanup, and content-addressed file cleanup. The updated regression cases check stale-entry removal and retained-entry preservation.
Validation was static only: tests, Docker builds, pre-commit, and live CI were not executed. Upstream uv cache-layout and upgrade behavior could not be independently verified from mounted source. Three review topics were delegated; two reports arrived. The garbage-collection child completed without delivering its report, so I inspected that scope directly.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
docker/Dockerfile conflicted with NVIDIA-NeMo#4863, which ran docker/dynamo/install.sh with a throwaway uv cache. On this branch install.sh only downloads etcd and nats-server, so the RUN keeps calling it directly. test_prune_uv_cache.py's Dockerfile test keeps its no-cache-mount check and drops the two lines that asserted the throwaway cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Summary
The release image ships the uv download cache (
/opt/nemo_rl_cache/uv) for offlineuv run --extra <backend>. Since #4340, each build restores that cache from a BuildKit cache mount, seeded from theuv-cache-<arch>-<key>registry image. Two things still grow it:uv synconly adds to the cache. The seed export'suv cache prune --cidrops prebuilt wheels but keeps every version ever built from source (DeepEP, TE, flash-attn, ...), and a long-lived builder's cache mount keeps everything.uv cache prunewithout--cican't help either: it never removes a valid entry for an older version. The newdocker/prune_uv_cache.pyremoves index wheels and source builds (pypi/andindex/<digest>/, e.g. torch and flashinfer) whose(name, version)uv.lockno longer pins, git builds/checkouts for commits it no longer pins, and then thearchive-v0entries no remaining link points to.--link-mode copyvenv that nothing re-syncs at runtime, but its wheels also land in the shared cache. It now installs with a throwawayUV_CACHE_DIR, so its RUN no longer needs the cache mount.uv cache clean deep-epis removed. It was added before the cache was keyed byHYBRID_EP_MULTINODE. Now the cache mount id and seed tag include it, anddeep_ep'smatch-runtimetorch pin is part of uv's built-wheel cache key, so a stale DeepEP build can't be reused. The clean only forced a rebuild every time.uv cache prune --ci, so they stop being rebuilt every build. Withpreview = trueit also turns on uv's content-addressed cache (archive files are hardlinks intofiles-v0/), so the prune also dropsfiles-v0objects nothing links to any more, and the cache rsyncs use-Hto keep those hardlinks.Where the prune runs
The prune runs in the dependency RUN, right after
uv_cache_restore forceand before anyuv sync:uv.lock(setuptools, a build-time torch, ...). An earlier revision that pruned after the syncs deleted those, and the TRT-LLM step downloaded them again.uv_cache_persist prunetrap mirrors the pruned cache back into the cache mount.A test pins this placement. The script is copied from the recipe's own build context, so building an older
NRL_GIT_REFthat predates it still works.Why the prune is safe for offline extras
The keep set is every
(name, version)and git commituv.lockpins across every extra, installed in this build or not, which is exactly what offlineuv run --extra <backend>can need.url/,path/,editable/andgit-v*/dbare left untouched, and so are the archives they point to. A--min-keepfloor refuses to prune ifuv.lockparses to a suspiciously small set.Test plan
tests/unit/tools/test_prune_uv_cache.pyandtest_docker_hybridep_build.pypassdubefore/after the prunerl:main(40.0 vs 47.4 GB; the gap is Dynamo's wheels, which main ships twice)uv cache prunenothing to remove, offline re-sync works, a git-built wheel survives--ci🤖 Generated with Claude Code