Repository navigation
[AMD] Build Miles nightly ROCm images with docker/build.py and test ROCm 7.2 - #31757
Conversation
Build the miles ROCm nightly images by calling miles's docker/build.py instead of an inline docker build, so the per-variant build args live in one place (the miles VARIANTS table) instead of being duplicated in each workflow matrix. build.py builds the image, and the workflow tags it with today's date and pushes that single miles-rocm7xx-mi3xx dated tag, so the published output is unchanged. Rename the miles nightly test from rocm700 to rocm720 and point it at the rocm720 image. The rocm700 images are still built but no longer tested.
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
bingxche
left a comment
There was a problem hiding this comment.
Reviewed the new Miles build path against the live upstream script and the amd-docker-scale runner.
| -t rocm/sgl-dev:${dated} \ | ||
| --no-cache | ||
| docker push rocm/sgl-dev:${dated} | ||
| python3 docker/build.py --variant ${{ matrix.variant }} --image-tag custom --custom-tag miles |
There was a problem hiding this comment.
[P1] Install Typer before invoking this script. The checked-out Miles docker/build.py imports typer, but this workflow never installs it. A smoke run on the same amd-docker-scale pool fails here with ModuleNotFoundError: No module named 'typer': https://github.com/sgl-project/sglang/actions/runs/29752901051. Miles' own Docker workflow explicitly installs Python/pip/Typer first. Please add equivalent dependency setup before this command. The same issue applies to the corresponding invocation in the ROCm 7.2 workflow.
There was a problem hiding this comment.
Thanks for the review! Fixed.
| fetch-depth: 0 | ||
|
|
||
| - name: Record miles commit | ||
| run: echo "Building from miles main @ $(git rev-parse HEAD)" |
There was a problem hiding this comment.
[P2] Keep this SHA pinned in the image build, rather than only printing it. The previous workflow passed this value as MILES_COMMIT and used --no-cache. Current upstream build.py does not pass MILES_COMMIT, while Dockerfile.rocm defaults it to main and clones Miles again. The image therefore is not guaranteed to contain the commit recorded here (and a cache-preserving builder can reuse an older clone layer under a new dated tag). Please have build.py derive/pass the checked-out HEAD or expose an override for it. The same issue applies to the ROCm 7.2 workflow.
There was a problem hiding this comment.
Hi @XinyuJiangCMU could you please take a look at this?
There was a problem hiding this comment.
Hi @bingxche, I tested the images build successfully. Since Miles updates are relatively infrequent compared with SGLang, I think the small potential drift is acceptable here, and I’d prefer to keep this aligned with the upstream build.py flow.
bingxche
left a comment
There was a problem hiding this comment.
Two additional non-blocking review notes.
| @@ -1,4 +1,4 @@ | |||
| name: Nightly Test Miles (AMD ROCm 7.0) | |||
| name: Nightly Test Miles (AMD ROCm 7.2) | |||
There was a problem hiding this comment.
[P3] Please confirm that dropping ROCm 7.0 runtime coverage is intentional. This rename replaces the only Miles ROCm 7.0 nightly test rather than adding a separate 7.2 workflow. Both ROCm 7.0 images will still be published, but neither will have a Miles runtime nightly after this merges. If ROCm 7.0 remains supported, please keep the existing workflow and add 7.2 separately; otherwise, documenting the coverage migration/deprecation in the PR description would make the tradeoff explicit.
There was a problem hiding this comment.
Yes, this is intentional. Only ROCm 7.2 will have Miles nightly runtime coverage. I’ve updated the PR description accordingly.
| -t rocm/sgl-dev:${dated} \ | ||
| --no-cache | ||
| docker push rocm/sgl-dev:${dated} | ||
| python3 docker/build.py --variant ${{ matrix.variant }} --image-tag custom --custom-tag miles |
There was a problem hiding this comment.
[P3/security hardening] Minimize credentials while executing the moving external script. This now executes host-level Python from radixark/miles@main after the Docker Hub login, and actions/checkout persists the GitHub token by default. A compromised upstream commit would therefore be able to read the runner's Docker config and persisted checkout credential. Please consider persist-credentials: false plus minimal job permissions, and either run the build before the push-capable login or use separate read-only pull credentials followed by the push login. The same consideration applies to the ROCm 7.0 workflow.
Address review on the docker/build.py migration: - Install typer before invoking build.py (build.py imports it; a runner without it fails with ModuleNotFoundError). - Do not persist the checkout credential, add a contents:read permission, and run the build and docker tag before the Docker Hub login so the external build.py never runs with push credentials present.
|
Upstream end-to-end validation is running:
After |
|
ROCm 7.2 Miles end-to-end verification dispatched against the image built above:
|
Summary
Build the Miles ROCm nightly images using Miles's
docker/build.pyinstead of an inlinedocker build. This keeps the per-variant build arguments centralized in the MilesVARIANTStable rather than duplicating them across workflows.build.pybuilds the image, and the workflow then applies today's date tag and pushes the same singlemiles-rocm7xx-mi3xx-<YYYYMMDD>image tag as before.Move Miles nightly runtime coverage from ROCm 7.0 to ROCm 7.2. ROCm 7.0 images will still be built and published, but only ROCm 7.2 will be tested nightly.
CI States
Latest PR Test (Base): ✅ Run #29778896611
Latest PR Test (Extra): ❌ Run #29778896269