Skip to content

build(docker): prune the uv cache, keep Dynamo out of it, bump uv to 0.13.6 - #4863

Merged
kajalj22 merged 11 commits into
mainfrom
kajalj/uv-cache-lockfile-prune
Nov 10, 2026
Merged

kajalj22 merged 11 commits into
mainfrom
kajalj/uv-cache-lockfile-prune

Conversation

@kajalj22

@kajalj22 kajalj22 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The release image ships the uv download cache (/opt/nemo_rl_cache/uv) for offline uv run --extra <backend>. Since #4340, each build restores that cache from a BuildKit cache mount, seeded from the uv-cache-<arch>-<key> registry image. Two things still grow it:

  • Superseded versions are never evicted. uv sync only adds to the cache. The seed export's uv cache prune --ci drops prebuilt wheels but keeps every version ever built from source (DeepEP, TE, flash-attn, ...), and a long-lived builder's cache mount keeps everything. uv cache prune without --ci can't help either: it never removes a valid entry for an older version. The new docker/prune_uv_cache.py removes index wheels and source builds (pypi/ and index/<digest>/, e.g. torch and flashinfer) whose (name, version) uv.lock no longer pins, git builds/checkouts for commits it no longer pins, and then the archive-v0 entries no remaining link points to.
  • Dynamo's wheels are stored twice. Dynamo syncs its own Python 3.12 lock into a --link-mode copy venv that nothing re-syncs at runtime, but its wheels also land in the shared cache. It now installs with a throwaway UV_CACHE_DIR, so its RUN no longer needs the cache mount.
  • uv cache clean deep-ep is removed. It was added before the cache was keyed by HYBRID_EP_MULTINODE. Now the cache mount id and seed tag include it, and deep_ep's match-runtime torch pin is part of uv's built-wheel cache key, so a stale DeepEP build can't be reused. The clean only forced a rebuild every time.
  • uv 0.11.28 → 0.13.6. 0.13 keeps git and no-build-isolation wheels (DeepEP, DeepGEMM, mamba-ssm, ...) through the seed export's uv cache prune --ci, so they stop being rebuilt every build. With preview = true it also turns on uv's content-addressed cache (archive files are hardlinks into files-v0/), so the prune also drops files-v0 objects nothing links to any more, and the cache rsyncs use -H to keep those hardlinks.

Where the prune runs

The prune runs in the dependency RUN, right after uv_cache_restore force and before any uv sync:

  • It runs in that same layer because deleting cache files in a later layer only hides them; the image still ships every byte.
  • It runs before the syncs so it only judges restored content. The syncs also download build-isolation deps that are resolved outside uv.lock (setuptools, a build-time torch, ...). An earlier revision that pruned after the syncs deleted those, and the TRT-LLM step downloaded them again.
  • The RUN's uv_cache_persist prune trap mirrors the pruned cache back into the cache mount.

A test pins this placement. The script is copied from the recipe's own build context, so building an older NRL_GIT_REF that predates it still works.

Why the prune is safe for offline extras

The keep set is every (name, version) and git commit uv.lock pins across every extra, installed in this build or not, which is exactly what offline uv run --extra <backend> can need. url/, path/, editable/ and git-v*/db are left untouched, and so are the archives they point to. A --min-keep floor refuses to prune if uv.lock parses to a suspiciously small set.

Test plan

  • tests/unit/tools/test_prune_uv_cache.py and test_docker_hybridep_build.py pass
  • H100 release build succeeds; the dependency layer logs du before/after the prune
  • Release-stage offline check still passes
  • Compressed image size and per-layer sizes compared against rl:main (40.0 vs 47.4 GB; the gap is Dynamo's wheels, which main ships twice)
  • Prune removes stale entries from a real seed (test PR test(ci): uv cache seed refresh/reuse/prune test for #4863 (DO NOT MERGE) #4891: 4 stale entries, 5.2G to 5.1G, 0.3s)
  • uv 0.13.6 end to end: prune leaves uv cache prune nothing to remove, offline re-sync works, a git-built wheel survives --ci
  • H100 build on uv 0.13.6

🤖 Generated with Claude Code

The release image's shipped uv cache (UV_CACHE_DIR) is append-only:
uv.lock's own cache-seed key is BASE_IMAGE+UV_VERSION+HYBRID_EP_MULTINODE,
not the lock content, so every routine dependency-version bump (torch,
vLLM, SGLang, ...) leaves its predecessor's wheel behind forever as long
as that key stays the same -- which is most of the time.

Add docker/prune_uv_cache.py, run once after all uv-writing RUNs finish
and before the ownership handoff. It parses uv.lock for every (name,
version) pin across all extras (installed in this build or not, so it
cannot break the offline `uv run --extra <backend>` promise UV_CACHE_DIR
exists for) and removes only cache entries nothing in the current lock
still references.

Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@kajalj22

Copy link
Copy Markdown
Contributor Author

/ok to test a8d69c9

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 22, 2026
update-uv-cache and update-uv-cache-gb200 had no status function in their
if:, so GitHub implied success(), which also requires every transitive need
to succeed. Since cicd-wait-in-queue (and later prepare-megatron-lock) became
needs of the build jobs, and both are skipped on main pushes, the refresh
has been skipped on every main run. The seed went stale, its cache key no
longer matches, and every build that reruns the dependency layer starts
from an empty uv cache.

Use the always() ... !cancelled() pattern the build jobs already use, with
explicit result checks on the jobs these depend on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@github-actions github-actions Bot added the CI Relating to CI label Oct 22, 2026
@kajalj22 kajalj22 added the CI:L1 Run doctests, unit tests, and functional tests label Oct 22, 2026
@kajalj22 kajalj22 changed the title Prune uv cache to what uv.lock still pins fix(ci): refresh uv cache seed and prune it to uv.lock Oct 22, 2026
@kajalj22

Copy link
Copy Markdown
Contributor Author

/ok to test 55dd90f

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 22, 2026 22:42
…hives

- Garbage-collect archive-v0 by scanning every remaining link under
  wheels-v*/ and sdists-v*/ (as uv cache prune does), instead of only kept
  pypi entries. The old scan deleted archives still referenced by index/,
  url/ and built-from-source entries, leaving dangling links that break
  offline installs of e.g. torch.
- Prune alternative-index trees (index/<digest>/, used for torch and
  flashinfer) by (name, version), not just pypi/.
- Prune git source builds and checkouts whose commit uv.lock no longer pins.
- Judge each cache file by its own parsed version; the old prefix glob
  removed 1.0.1's files when pruning 1.0.
- Keep the truncated form of long versions, matching uv's wheel cache keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22

Copy link
Copy Markdown
Contributor Author

/ok to test f2d2057

@kajalj22

Copy link
Copy Markdown
Contributor Author

/review

@kajalj22
kajalj22 marked this pull request as draft October 23, 2026 00:47

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all five changed files and traced the affected CI, Docker, cache, and test paths. Refreshing the seed addresses a useful problem, but the pruning integration has build-compatibility and image-size defects, plus two edge cases that delete still-pinned cache entries. There is also one non-blocking type-check coverage suggestion.

Validation was static only: tests, Docker builds, linters, and live GitHub Actions were not executed. Findings were checked against the inspected callers and regression tests; no independent teammate review was available.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread docker/Dockerfile Resolved
Comment thread docker/Dockerfile Resolved
Comment thread docker/prune_uv_cache.py Resolved
Comment thread docker/prune_uv_cache.py Resolved
Comment thread docker/prune_uv_cache.py Resolved
Deleting cache files in a later RUN only hides them behind whiteouts; the
layers that wrote them still ship every byte. The release-stage prune
therefore left the image size unchanged (61 vs 60 layers, the extra one
~0 bytes).

- Prune at the end of the dependency RUN, which seeds and fills the cache,
  so superseded versions never get committed to a layer.
- Install Dynamo with a throwaway UV_CACHE_DIR: it syncs its own Python 3.12
  lock into a --link-mode copy venv that nothing re-syncs at runtime, so its
  wheels were stored twice.
- Drop the release-stage prune RUN, and add a test that pins the prune to
  the dependency RUN and the Dynamo install to a throwaway cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22

Copy link
Copy Markdown
Contributor Author

/ok to test a6378c1

@kajalj22

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review against f2d2057, with current inspection of all five changed files.

The same-layer dependency prune addresses the earlier finding about shipping superseded seed bytes. The Dynamo installation now uses a temporary cache, consistent with its copy-mode environment.

Four prior findings remain: source-ref build compatibility, two pinned-cache deletion cases, and one non-blocking type-check coverage omission. They are detailed inline. Subsequent changes and their directly affected Docker, cache, and test paths were checked; no additional escalation-eligible regression was established.

Validation was static only: tests, Docker builds, pre-commit, and GitHub Actions were not executed. No independent teammate review was available. Suggested edits and acceptance checks require author verification.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

Comment thread docker/Dockerfile Resolved
Comment thread docker/prune_uv_cache.py Resolved
Comment thread docker/prune_uv_cache.py Resolved
Comment thread docker/prune_uv_cache.py Resolved
Pruning at the end of the dependency RUN also deleted the build-isolation
deps the syncs resolve outside uv.lock (setuptools, setuptools-scm,
poetry-core, a build-time torch 2.14.1 + CUDA wheels, ...), which the
TRT-LLM step then downloaded again. Run the prune right after the seed is
copied, so it only judges what older locks left behind.

Also, from review:
- copy the script from the recipe's context, so building an older
  NRL_GIT_REF that lacks it still works
- compare sdist version directories verbatim instead of stripping a
  metadata suffix from the version
- keep archives of a package named `src`
- type-check the script with pyrefly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22

Copy link
Copy Markdown
Contributor Author

/ok to test 0d8214e

@kajalj22

Copy link
Copy Markdown
Contributor Author

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from static review of all six changed files and their directly affected CI, Docker, cache, and test paths. No actionable findings remain.

The four findings from the latest review are addressed: the helper now comes from the recipe context, sdist version directories are compared verbatim, packages named src remain traversable, and the helper is included in type-check coverage. The pruning placement and relevant regression assertions were also inspected.

Validation was static only: tests, Docker builds, pre-commit, and live GitHub Actions were not executed. No independent teammate review was available.

Review generated by codex. Full review with RESET=true.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

main (#4340) replaced the update-uv-cache jobs with an in-build cache mount
and seed export, so this branch drops its workflow change and takes main's.

- Prune right after `uv_cache_restore force`, before any sync; the RUN's
  `uv_cache_persist prune` trap mirrors the result into the cache mount.
- Dynamo keeps its throwaway UV_CACHE_DIR, so its RUN no longer needs the
  cache mount or the restore/persist helpers.
- Placement test and docs updated for the new flow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@github-actions github-actions Bot removed the CI Relating to CI label Oct 28, 2026
@kajalj22 kajalj22 changed the title fix(ci): refresh uv cache seed and prune it to uv.lock build(docker): prune the uv cache to uv.lock and keep Dynamo out of it Oct 28, 2026
@kajalj22

Copy link
Copy Markdown
Contributor Author

/ok to test cbbf92d

kajalj22 added a commit that referenced this pull request Oct 28, 2026
main (#4340) replaced the update-uv-cache jobs, so retarget the test at
the new mechanism: both builds use the uv-cache-test tag prefix, and the
"Update uv cache image" step also runs on this PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
The uv cache mount id and the seed tag are now keyed by HYBRID_EP_MULTINODE,
and deep_ep's match-runtime torch pin is part of uv's built-wheel cache key, so
a cached DeepEP build can no longer be reused for the wrong multinode setting
or torch ABI. The clean only forced a rebuild on every build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
…ile-prune

Signed-off-by: Kajal Jain <kajalj@nvidia.com>

# Conflicts:
#	pyrefly.toml
@kajalj22

kajalj22 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 32e57ac

@kajalj22

kajalj22 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for this follow-up. I compared the changes since the previously reviewed head, inspected all six changed files, and consolidated three delegated reviews. No qualifying new regressions were established.

The restored-cache prune remains before dependency syncs and within the layer importing the cache. The persistence helper mirrors those deletions. The previously verified recipe-context helper acquisition, sdist-version and package-named-src regression cases, and type-check inclusion remain intact.

Validation was static only: tests, Docker builds, pre-commit, and live CI were not executed. External uv cache-key internals were not independently verified because their source isn't mounted.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

uv 0.13 keeps git and no-build-isolation wheels (deep-ep, deep-gemm,
mamba-ssm, ...) through `uv cache prune --ci`, so the exported seed no
longer forces those source builds on every rebuild.

With `preview = true`, 0.13 also enables the content-addressed cache:
archive files are hardlinks into files-v0/. Adapt to it:
- prune_uv_cache.py drops files-v0 objects nothing links to any more
  (link count 1, uv's own rule) and the new <sha>.ok checkout markers;
  otherwise removing an archive frees nothing.
- rsync the cache with -H so hardlinks survive the restore/persist
  round trip instead of doubling the cache.

Tests move to the 0.13 bucket layout, cover the files-v0 cleanup, and
allowlist the fake commit SHAs for the secrets detector.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22 kajalj22 changed the title build(docker): prune the uv cache to uv.lock and keep Dynamo out of it build(docker): prune the uv cache to uv.lock, keep Dynamo out of it, bump uv to 0.13.6 Nov 9, 2026
@kajalj22

kajalj22 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e8e3338

@kajalj22

kajalj22 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/review reset=true

@copy-pr-bot

copy-pr-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

/ok to test e8e3338

@kajalj22, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@kajalj22
kajalj22 marked this pull request as ready for review November 9, 2026 17:28

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed all six changed files and their directly affected Docker, cache, and test paths. No blocking correctness issue was established. C1 is one non-blocking guideline suggestion.

The recipe-context helper acquisition, same-layer pruning before dependency syncs, and sdist-version and package-named-src regressions address the earlier concrete failures.

Validation was static only: tests, Docker builds, pre-commit, and live CI were not executed. External uv cache-layout and cache-key behavior could not be independently verified from mounted upstream source. Three review topics were delegated; the pin-selection child's final report wasn't delivered, so I inspected its pin-selection and CLI scope directly.

Review generated by codex. Full review with RESET=true.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread tests/unit/tools/test_prune_uv_cache.py Resolved
balasaajay
balasaajay previously approved these changes Nov 9, 2026
The Dockerfile already explains why the prune runs where it does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22

kajalj22 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e4dce7e

@kajalj22 kajalj22 changed the title build(docker): prune the uv cache to uv.lock, keep Dynamo out of it, bump uv to 0.13.6 build(docker): prune the uv cache, keep Dynamo out of it, bump uv to 0.13.6 Nov 9, 2026
…ile-prune

Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22

kajalj22 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b8ad614

1 similar comment
@kajalj22

kajalj22 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b8ad614

@kajalj22

kajalj22 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/review-approve

@kajalj22
kajalj22 enabled auto-merge (squash) November 9, 2026 22:59

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for this follow-up. I compared the current head with the previously reviewed head, inspected all six changed files and their directly affected paths, and found no unresolved prior finding or qualifying new regression.

Prior C1 is fixed: the placement test now uses the one-line invariant, while its ordering, recipe-context, persistence, and Dynamo assertions remain intact.

The follow-up inspection covered the uv upgrade, hardlink-preserving cache transfers, checkout-marker cleanup, and content-addressed file cleanup. The updated regression cases check stale-entry removal and retained-entry preservation.

Validation was static only: tests, Docker builds, pre-commit, and live CI were not executed. Upstream uv cache-layout and upgrade behavior could not be independently verified from mounted source. Three review topics were delegated; two reports arrived. The garbage-collection child completed without delivering its report, so I inspected that scope directly.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@kajalj22
kajalj22 merged commit 5eea715 into main Nov 10, 2026
153 of 161 checks passed
@kajalj22
kajalj22 deleted the kajalj/uv-cache-lockfile-prune branch November 10, 2026 03:22
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 10, 2026
docker/Dockerfile conflicted with NVIDIA-NeMo#4863, which ran docker/dynamo/install.sh
with a throwaway uv cache. On this branch install.sh only downloads etcd and
nats-server, so the RUN keeps calling it directly. test_prune_uv_cache.py's
Dockerfile test keeps its no-cache-mount check and drops the two lines that
asserted the throwaway cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants