Skip to content

fix(launch): install srtctl with uv's copy link mode - #4459

Merged
Oseltamivir merged 2 commits into
mainfrom
fix/srtctl-install-uv-copy-link-mode
Oct 24, 2026
Merged

Oseltamivir merged 2 commits into
mainfrom
fix/srtctl-install-uv-copy-link-mode

Conversation

@Oseltamivir

@Oseltamivir Oseltamivir commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

install_srtctl now runs uv venv and uv pip install -e with UV_LINK_MODE=copy. The setting applies only to those two calls; run.env is unchanged.

Why

On b200-nscale the uv cache sits on the shared /data NFS export. When a sweep fans out after its canary, six or more jobs hardlink the same cached setuptools files into their build envs at once, and those NFS link calls fail:

cause: Failed to install: setuptools-84.2.1-py3-none-any.whl (setuptools==84.2.1)
cause: failed to hardlink file from …/.cache/uv/builds-v0/.tmp…/setuptools/… to …/.cache/uv/archive-v0/LnndTy54RH31kiQx/…
LaunchError: installing srtctl failed (exit 2)

That's how sweep attempts 2–4 of #4442 failed: the c1 canary always passed alone, and every fan-out job died at "installing srtctl".

I checked on the cluster that the cache isn't corrupt. The cached entry (archive-v0/LnndTy54RH31kiQx, 348 files, from Oct 21) is complete, every file has a link count of 1, a test hardlink to it succeeds, and a single uv pip install setuptools==84.2.1 against the shared cache with the runners' uv 0.13.5 passes. So clearing the cache wouldn't help; avoiding hardlinks on shared storage does.

Cost (measured on b200-nscale, warm cache, venv on the same /data NFS export)

Link mode Install time Venv size Files
hardlink (before) 3.4 s 342 MB 6,228
copy (this PR) 11.5 s 342 MB 6,228
  • Time: about 8 s more per srt job.
  • Disk: each checkout's venv is now ~342 MB of its own data instead of hardlinks into the cache. A sampled leftover venv had files with 23 links, so they were shared before. The runner workspaces held 11 leftover srt-single.* checkouts, so expect a few GB extra on /data, which has 4.9 TB free.
  • Correctness: the same packages and versions get installed, and the editable srtctl install is unchanged.
  • Other clusters: where the cache and the checkout are on different filesystems, uv already falls back to copying, so nothing changes there.

Testing

  • py_compile passes.
  • No test asserts the install env. test_srt_driver only checks that VIRTUAL_ENV stays out of later calls, which still holds.
  • CI runs the launcher suites.

The srt drivers build srtctl's venv against a uv cache that sits on shared
storage (b200-nscale keeps it on the /data NFS export). When a sweep fans out, six
or more jobs hardlink the same cached setuptools files into their build envs at
once and the NFS LINK calls fail, so every job but the canary died at "installing
srtctl". The cache entry itself is intact and a single install succeeds. Copying
the few hundred files avoids hardlinks on shared storage entirely.
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +5 -2

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

@Oseltamivir
Oseltamivir merged commit 36f6a42 into main Oct 24, 2026
3 checks passed
@Oseltamivir
Oseltamivir deleted the fix/srtctl-install-uv-copy-link-mode branch October 24, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant