Skip to content

[AMD] Build Miles nightly ROCm images with docker/build.py and test ROCm 7.2 - #31757

Merged
bingxche merged 3 commits into
sgl-project:mainfrom
XinyuJiangCMU:amd/miles-buildpy-ci-20260720
Jul 23, 2026
Merged

bingxche merged 3 commits into
sgl-project:mainfrom
XinyuJiangCMU:amd/miles-buildpy-ci-20260720

Conversation

@XinyuJiangCMU

@XinyuJiangCMU XinyuJiangCMU commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Build the Miles ROCm nightly images using Miles's docker/build.py instead of an inline docker build. This keeps the per-variant build arguments centralized in the Miles VARIANTS table rather than duplicating them across workflows.

build.py builds the image, and the workflow then applies today's date tag and pushes the same single miles-rocm7xx-mi3xx-<YYYYMMDD> image tag as before.

Move Miles nightly runtime coverage from ROCm 7.0 to ROCm 7.2. ROCm 7.0 images will still be built and published, but only ROCm 7.2 will be tested nightly.


CI States

Latest PR Test (Base): ✅ Run #29778896611
Latest PR Test (Extra): ❌ Run #29778896269

Build the miles ROCm nightly images by calling miles's docker/build.py
instead of an inline docker build, so the per-variant build args live in
one place (the miles VARIANTS table) instead of being duplicated in each
workflow matrix. build.py builds the image, and the workflow tags it with
today's date and pushes that single miles-rocm7xx-mi3xx dated tag, so the
published output is unchanged.

Rename the miles nightly test from rocm700 to rocm720 and point it at the
rocm720 image. The rocm700 images are still built but no longer tested.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions github-actions Bot added the amd label Jul 20, 2026

@bingxche bingxche left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Miles build path against the live upstream script and the amd-docker-scale runner.

-t rocm/sgl-dev:${dated} \
--no-cache
docker push rocm/sgl-dev:${dated}
python3 docker/build.py --variant ${{ matrix.variant }} --image-tag custom --custom-tag miles

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Install Typer before invoking this script. The checked-out Miles docker/build.py imports typer, but this workflow never installs it. A smoke run on the same amd-docker-scale pool fails here with ModuleNotFoundError: No module named 'typer': https://github.com/sgl-project/sglang/actions/runs/29752901051. Miles' own Docker workflow explicitly installs Python/pip/Typer first. Please add equivalent dependency setup before this command. The same issue applies to the corresponding invocation in the ROCm 7.2 workflow.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review! Fixed.

fetch-depth: 0

- name: Record miles commit
run: echo "Building from miles main @ $(git rev-parse HEAD)"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep this SHA pinned in the image build, rather than only printing it. The previous workflow passed this value as MILES_COMMIT and used --no-cache. Current upstream build.py does not pass MILES_COMMIT, while Dockerfile.rocm defaults it to main and clones Miles again. The image therefore is not guaranteed to contain the commit recorded here (and a cache-preserving builder can reuse an older clone layer under a new dated tag). Please have build.py derive/pass the checked-out HEAD or expose an override for it. The same issue applies to the ROCm 7.2 workflow.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @XinyuJiangCMU could you please take a look at this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @bingxche, I tested the images build successfully. Since Miles updates are relatively infrequent compared with SGLang, I think the small potential drift is acceptable here, and I’d prefer to keep this aligned with the upstream build.py flow.

@bingxche bingxche left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two additional non-blocking review notes.

@@ -1,4 +1,4 @@
name: Nightly Test Miles (AMD ROCm 7.0)
name: Nightly Test Miles (AMD ROCm 7.2)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Please confirm that dropping ROCm 7.0 runtime coverage is intentional. This rename replaces the only Miles ROCm 7.0 nightly test rather than adding a separate 7.2 workflow. Both ROCm 7.0 images will still be published, but neither will have a Miles runtime nightly after this merges. If ROCm 7.0 remains supported, please keep the existing workflow and add 7.2 separately; otherwise, documenting the coverage migration/deprecation in the PR description would make the tradeoff explicit.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is intentional. Only ROCm 7.2 will have Miles nightly runtime coverage. I’ve updated the PR description accordingly.

-t rocm/sgl-dev:${dated} \
--no-cache
docker push rocm/sgl-dev:${dated}
python3 docker/build.py --variant ${{ matrix.variant }} --image-tag custom --custom-tag miles

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3/security hardening] Minimize credentials while executing the moving external script. This now executes host-level Python from radixark/miles@main after the Docker Hub login, and actions/checkout persists the GitHub token by default. A compromised upstream commit would therefore be able to read the runner's Docker config and persisted checkout credential. Please consider persist-credentials: false plus minimal job permissions, and either run the build before the push-capable login or use separate read-only pull credentials followed by the push login. The same consideration applies to the ROCm 7.0 workflow.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

Address review on the docker/build.py migration:
- Install typer before invoking build.py (build.py imports it; a runner
  without it fails with ModuleNotFoundError).
- Do not persist the checkout credential, add a contents:read permission, and
  run the build and docker tag before the Docker Hub login so the external
  build.py never runs with push credentials present.
@bingxche

Copy link
Copy Markdown
Collaborator

Upstream end-to-end validation is running:

After miles-rocm720-mi35x-20260721 is pushed, I will trigger the ROCm 7.2 Miles nightly test workflow against that explicit tag and add the test run link here.

@bingxche

bingxche commented Jul 21, 2026 •

Copy link
Copy Markdown
Collaborator

ROCm 7.2 Miles end-to-end verification dispatched against the image built above:

@bingxche
bingxche merged commit 8ce6837 into sgl-project:main Jul 23, 2026
86 of 90 checks passed
Zhylkaaa pushed a commit to Zhylkaaa/sglang that referenced this pull request Jul 29, 2026
Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants