Skip to content

ci: build LMCache ROCm torch 2.10 wheels from any commit - #2388

Merged
valarLip merged 3 commits into
mainfrom
ci/lmcache-rocm-torch210-wheel
Sep 24, 2026
Merged

valarLip merged 3 commits into
mainfrom
ci/lmcache-rocm-torch210-wheel

Conversation

@yhl-amd

@yhl-amd yhl-amd commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds .github/workflows/lmcache-rocm-wheel.yaml, a manually dispatched workflow that builds an LMCache wheel for ATOM's image ABI (ROCm 7.2.4 / torch 2.10.0, gfx942 + gfx950) from any LMCache commit and can publish it as a pinned GitHub Release on this repository.

Why

ATOM's image ships AMD's torch 2.10.0+rocm7.2.4.git3d3aa833. LMCache publishes a wheel for that ABI only with tagged releases (v*-rocm-torch210). Its rolling nightly-rocm wheel is built against torch 2.11 for the upstream vLLM image. I installed today's nightly (0.5.6.dev98+rocm7.2, built from dev@05fc77a) in an ATOM image, and its native extension fails to load:

ImportError: .../lmcache/cuda_ops.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda20CUDACachingAllocator9allocatorE

So LMCache changes that ATOM depends on, such as LMCache/LMCache#5132 for #2250, cannot be used until LMCache's next tag. The nightly release is also deleted and republished every day, so its URL cannot be pinned.

How it works

  • Input: lmcache_commit must be a full 40-character SHA; branch names are rejected. publish defaults to off, which builds and validates only. runner is build-only-atom by default; linux-atom-mi35x-1 additionally runs the GPU kernel check.
  • Build: runs LMCache's own .github/scripts/build_rocm_torch210_wheel.sh from the requested commit, inside the same digest-pinned rocm/pytorch:rocm7.2.4_..._pytorch_release_2.10.0 image that LMCache's build_rocm_torch210_artifacts.yml uses. The build steps are not duplicated in ATOM.
  • Version: follows LMCache's nightly numbering (setuptools-scm guess-next-dev from the last stable tag), with the commit and ABI in the local segment, e.g. 0.5.6.dev98+g05fc77a0.rocm7.2.4.torch2.10.git3d3aa833.cxx11abi1.
  • Validation in a clean container of the build image:
    • LMCache's validate_rocm_torch210_wheel.py checks the exact torch/HIP/ROCm/CXX11 ABI and the wheel tag. On a GPU runner it also runs the HIP D2H/H2D kernels.
    • New .github/scripts/validate_lmcache_wheel_for_atom.py requires the ATOM MP adapters, execute_object_group_transfer, and the native .so modules. It also parses --null-block-id -1 --separate-object-groups through the MP server's own parser.
  • Publish runs in a separate job on ubuntu-latest, so the write token never reaches the self-hosted runner. It creates tag lmcache-v<version>-g<sha8>-rocm-torch210 as a pre-release that is not marked latest, so ATOM's own release keeps its place. It refuses to overwrite an existing tag, and the release notes include the ARG LMCACHE_WHEEL_* lines to paste into the Dockerfiles. LMCache's Apache-2.0 LICENSE is included in the wheel's dist-info.

Validation

Ran the build job's run: steps locally, read straight from the YAML, with LMCache at 05fc77a0a7ababd9a7f2e343a771bc4bbc5b65cb:

  • The build succeeded: lmcache-0.5.6.dev98+g05fc77a0.rocm7.2.4.torch2.10.git3d3aa833.cxx11abi1-cp312-cp312-manylinux_2_39_x86_64.whl
  • LMCache validator passed, including GPU kernels on gfx950: OK: ROCm torch 2.10 wheel ABI; ATOM adapters available ... hip 7.2.53211
  • ATOM validator passed: OK: lmcache 0.5.6.dev98+g05fc77a0... has the ATOM MP adapters and server flags
  • actionlint 1.7.7 passes on all workflows; black and ruff pass on the new script.

The build job also passed on GitHub, on build-only-atom, triggered by a temporary push trigger that has since been reverted: https://github.com/ROCm/ATOM/actions/runs/35982738885. That run produced lmcache-0.5.6.dev98+g05fc77a0...whl (sha256 323ed34f...7be6) and both validators passed. The publish job was skipped, and it has not run yet; it first runs on the post-merge workflow_dispatch with publish=true.

Follow-up

Once this is merged, dispatch it with 05fc77a0a7ababd9a7f2e343a771bc4bbc5b65cb and publish=true, then pin the resulting release in docker/Dockerfile and docker/atom_release.dockerfile for #2250.

🤖 Generated with Claude Code

ATOM's image ships AMD's ROCm 7.2.4 / torch 2.10 build, but LMCache only
publishes a wheel for that ABI with tagged releases. Its rolling
nightly-rocm wheel targets torch 2.11 and fails to load here
(undefined c10::cuda::CUDACachingAllocator symbol), so an LMCache fix on
dev could not reach ATOM until the next tag.

Add a manually dispatched workflow that builds LMCache at a given commit
with LMCache's own build script inside the same digest-pinned
rocm/pytorch 2.10 image LMCache uses, validates it in a clean container
(LMCache's ABI validator plus an ATOM check for the MP adapters, native
entry points and the MP server flags ATOM launches with), and optionally
publishes it as an immutable pre-release that the Dockerfiles can pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2388 --add-label <label>

Honglie Yi and others added 2 commits September 24, 2026 09:41
Temporary, to exercise the build job on GitHub before merge; revert
before merging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build job passed on build-only-atom:
https://github.com/ROCm/ATOM/actions/runs/35982738885

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@valarLip
valarLip merged commit 514f212 into main Sep 24, 2026
22 of 25 checks passed
@valarLip
valarLip deleted the ci/lmcache-rocm-torch210-wheel branch September 24, 2026 09:50
zufayu added a commit that referenced this pull request Oct 30, 2026
near_duplicates() only catches a duplicate that arrives as a whole new file. Most
of "we already have it" / "duplicated with the one in aiter" / "use new util func"
in the lead reviewer's comments is a function or class added into a file that
already exists -- a shape the file-level check cannot see, since it never looks
inside a diff hunk for a modified file.

Adds symbol_duplicates(): for each newly added top-level def/class in a modified
file's diff (whole new files stay out of scope -- near_duplicates already covers
them, and double-reporting the same evidence under two headings was not the goal),
compare its body (>=15 substantial lines) against every top-level def/class already
in the tree (same kind, whole tree -- measured at 0.16s / ~14k symbols over ATOM's
1504 .py files, cheap enough that a name-similarity prefilter would suppress exactly
the "unrelated names, near-identical bodies" case this exists to find). Same
threshold as the file-level check, 60%, and same posture: a pair and a ratio, not a
verdict.

Measured over 197 recently merged ATOM PRs (#2388-#2830) at threshold 0.60: 2 fired
(1.0%). Both spot-checked as real hits, not coincidental shape: PR#2560 modifies
atom/plugin/vllm/paged_state_image.py and its tests in place, 10 pairs against the
pre-existing paged_state_checkpoint.py sibling it mirrors (the "image vs checkpoint"
variant pair near_duplicates' own docstring describes, but file-level could not see
here because neither file is new). PR#2553 adds a test 95% identical to an existing
one, down to the mocked runner setup, with one of three asserted values changed.
No threshold raise needed -- 1.0% sits well inside the file-level check's own ~3%
territory, with no false alarms in the spot check.

Wired into the same `twins` CLI mode as a second section, after the file-level
pairs; fetch.sh's existing `triage.py twins ... | tee twins.txt` call is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants