Repository navigation
fix(launch): install srtctl with uv's copy link mode - #4459
Merged
Merged
Conversation
The srt drivers build srtctl's venv against a uv cache that sits on shared storage (b200-nscale keeps it on the /data NFS export). When a sweep fans out, six or more jobs hardlink the same cached setuptools files into their build envs at once and the NFS LINK calls fail, so every job but the canary died at "installing srtctl". The cache entry itself is intact and a single install succeeds. Copying the few hundred files avoids hardlinks on shared storage entirely.
Contributor
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
install_srtctlnow runsuv venvanduv pip install -ewithUV_LINK_MODE=copy. The setting applies only to those two calls;run.envis unchanged.Why
On
b200-nscalethe uv cache sits on the shared/dataNFS export. When a sweep fans out after its canary, six or more jobs hardlink the same cachedsetuptoolsfiles into their build envs at once, and those NFS link calls fail:That's how sweep attempts 2–4 of #4442 failed: the c1 canary always passed alone, and every fan-out job died at "installing srtctl".
I checked on the cluster that the cache isn't corrupt. The cached entry (
archive-v0/LnndTy54RH31kiQx, 348 files, from Oct 21) is complete, every file has a link count of 1, a test hardlink to it succeeds, and a singleuv pip install setuptools==84.2.1against the shared cache with the runners' uv 0.13.5 passes. So clearing the cache wouldn't help; avoiding hardlinks on shared storage does.Cost (measured on b200-nscale, warm cache, venv on the same
/dataNFS export)srt-single.*checkouts, so expect a few GB extra on/data, which has 4.9 TB free.srtctlinstall is unchanged.Testing
py_compilepasses.test_srt_driveronly checks thatVIRTUAL_ENVstays out of later calls, which still holds.