Skip to content

[AMD] fix(docker): drop the obsolete ROCm Megatron fused-kernels patch - #2768

Merged
yushengsu-thu merged 1 commit into
radixark:mainfrom
michaelzhang-ai:cursor/drop-obsolete-rocm-megatron-patch-e35f
Aug 27, 2026
Merged

yushengsu-thu merged 1 commit into
radixark:mainfrom
michaelzhang-ai:cursor/drop-obsolete-rocm-megatron-patch-e35f

Conversation

@michaelzhang-ai

@michaelzhang-ai michaelzhang-ai commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Every ROCm image build fails at Dockerfile.rocm step 17:

+ git apply /tmp/amd_patch/megatron.patch
error: megatron/legacy/fused_kernels/__init__.py: No such file or directory

docker/amd_patch/latest/megatron.patch adds a ROCm guard (if not torch.version.cuda: return) to Megatron's legacy fused-kernel loader. radixark/Megatron-LM@miles-main now carries upstream NVIDIA commit 3aa73966 ("Guard non-core imports", NVIDIA/Megatron-LM#3993), which deleted the whole megatron/legacy/fused_kernels/ package and removed the fused_kernels.load(args) call from _compile_dependencies() in megatron/training/initialize.py. The patch therefore has no target file and git apply fails hard.

This takes down the nightly ROCm image releases for all three variants — rocm700-mi35x, rocm700-mi30x and rocm720-mi35x — which are published from sgl-project/sglang:

The first red nightly is Aug 25. The Aug 24 12:00 UTC run still passed, and #2734 ("docker: point MEGATRON_BRANCH back at miles-main") merged at Aug 24 20:01 UTC, right after it — miles-main had been moved onto the newer NVIDIA dev base by then, and the interim miles-main-20260819 branch has since been deleted. Dockerfile.rocm has no MEGATRON_COMMIT arg, so there is no way to pin around this from the calling workflow; it has to be fixed here.

Fix

Nothing on miles-main loads the legacy fused kernels anymore, so the guard is dead code. This removes:

  • docker/amd_patch/latest/megatron.patch, plus its COPY and the git apply in Dockerfile.rocm
  • the apply / reverse-check / error block in _run-ci-rocm.yml that ran when a Megatron-PR override was selected — with the patch file gone that git apply --check would fail too, and it no longer has anything to verify

tests/ci/test/test_run_suite.py::test_megatron_override_preserves_rocm_patch pinned that block's ordering, so it becomes test_megatron_override_installs_the_checked_out_ref_unpatched, asserting the override installs the checked-out ref with no patch step.

Result: all three ROCm release images build green

Both sglang nightly release workflows were dispatched against 9d5e695 (this branch's head) and all three variants succeeded:

Workflow Variant Result Duration Image
ROCm 7.0 run 33020346832 rocm700-mi35x success 16m59s rocm/sgl-dev:miles-rocm700-mi35x-20260826, 67.9 GB
ROCm 7.0 run 33020346832 rocm700-mi30x success 14m31s rocm/sgl-dev:miles-rocm700-mi30x-20260826, 65.6 GB
ROCm 7.2 run 33020347833 rocm720-mi35x success 11m52s rocm/sgl-dev:miles-rocm720-mi35x-20260826, 84.2 GB

Those durations are in line with the last known-good nightlies (17m for 7.0, 21m for 7.2 on Aug 24), so nothing regressed on build time.

The previously failing stage now reads [sglang 16/37] RUN rm -rf /root/Megatron-LM && git clone --recursive -b miles-main ... && pip install -e . — one stage fewer, since the patch COPY is gone — and completes. All 42 buildx stages finish and the image exports (naming to docker.io/rocm/sgl-dev:miles-rocm720-mi35x done).

How it was run

The dispatch used a throwaway sgl-project/sglang branch carrying the two real nightly workflows with exactly two edits each: the miles checkout repointed at this branch, and the Docker Hub login plus the two docker push steps replaced by docker image inspect. Runner (amd-docker-scale), prod environment, variant matrix, docker/build.py invocation and date tagging were all untouched, so this is the release path minus publication. Nothing was pushed to rocm/sgl-dev. The harness branch has since been deleted; the run logs above remain.

Also checked

  • megatron/legacy is absent from the current miles-main tree (235952df); the only surviving megatron.legacy references are the unrelated fp16_deprecated loss-scaler shims in checkpointing.py, so dropping the guard changes no live code path.
  • pytest tests/ci/test/test_run_suite.py — 89 passed.
  • _run-ci-rocm.yml still parses as YAML; no reference to megatron.patch survives anywhere in the tree.
  • docker/image_inputs.py does not hash docker/amd_patch/, so the rebuild-skip label is unaffected.
  • Not exercised: the docker push steps, which this change does not touch.

Note

docker/amd_patch/sglv0.5.10/megatron.patch is a copy of the same obsolete patch and is referenced by nothing in the repo. Left alone here to keep the change minimal; happy to delete it in this PR if preferred.

radixark/Megatron-LM@miles-main now carries upstream NVIDIA commit 3aa73966
("Guard non-core imports", radixark#3993), which deleted megatron/legacy/fused_kernels
along with the fused_kernels.load(args) call in _compile_dependencies(). The
ROCm patch guarded that loader with a torch.version.cuda check, so its only
target file no longer exists and `git apply` fails:

    error: megatron/legacy/fused_kernels/__init__.py: No such file or directory

That breaks Dockerfile.rocm step 17 for every ROCm variant, taking down the
sglang nightly ROCm image releases for rocm700-mi35x, rocm700-mi30x and
rocm720-mi35x. Nothing on miles-main loads the legacy fused kernels anymore, so
the guard is dead code: remove the patch, its COPY and `git apply`, and the
apply/verify block the ROCm PR CI ran for a Megatron-PR override.
@michaelzhang-ai
michaelzhang-ai marked this pull request as ready for review August 26, 2026 21:25

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

michaelzhang-ai added a commit to sgl-project/sglang that referenced this pull request Aug 26, 2026
Throwaway test branch: repoints the miles checkout at the fork branch carrying
radixark/miles#2768 and replaces the Docker Hub login/push steps with an image
inspect, so the release build is exercised end to end without publishing a tag.
Not for main - delete this branch once #2768 merges.

Co-authored-by: Michael <michaelzhang-ai@users.noreply.github.com>
@michaelzhang-ai

michaelzhang-ai commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor Author

@Arist12 @yushengsu-thu @guapisolo @yueming-yuan please help review. Thanks!

@XinyuJiangCMU

Copy link
Copy Markdown
Contributor

LGTM

1 similar comment
@JessicaJiang-123

Copy link
Copy Markdown
Contributor

LGTM

@michaelzhang-ai michaelzhang-ai changed the title fix(docker): drop the obsolete ROCm Megatron fused-kernels patch [AMD] fix(docker): drop the obsolete ROCm Megatron fused-kernels patch Aug 27, 2026
@yushengsu-thu
yushengsu-thu merged commit 7df95c3 into radixark:main Aug 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants