Repository navigation
[AMD] ci: expand ROCm10 nightly coverage on MI350 - #3910
Merged
guapisolo merged 11 commits intoOct 6, 2026
Merged
Conversation
Register the checkpoint test in the 2-GPU suite and FSDP disk-delta, DSV4 indexer CP balance, GLM TP2/PP2, Qwen3 Muon TP2/CP2 and Qwen3.5 Muon TP2/PP2 in the 4-GPU suite. Set runtime estimates from local MI355X ROCm10 runs. Keep test configurations, checker settings and CUDA registrations unchanged.
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
fa4 is CUDA-only, so on ROCm the test appends --sglang-attention-backend triton (the trailing flag wins), the backend scripts/amd/run_inkling.py already uses. Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
Requires the pytest entrypoints and a ROCm image with torchtitan installed. Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
- Kimi-K3 4-layer: serve the rollout in BF16 on ROCm, as the LoRA variant does. - Qwen3.8-Next 4-layer: train with FLA and prefill with Triton on ROCm; FlashQLA and FlashInfer are CUDA-only. - DeepSeek-V4-Flash 4-layer Megatron implementation: ROCm has no cuDNN DSA kernel, so run --dsa-kernel-backend none and pin NVTE_FP8_BLOCK_SCALING_FP32_SCALES=0 as scripts/amd/run_deepseek_v4.py does. - DeepSeek-V4 and GLM-4.7 multi-role session-server tests: register only. Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
…nightly - scripts/amd/run_deepseek_v4.py: accept cp_size and split the single-node actor GPUs as TP x CP with --allgather-cp. CP1 flags are unchanged. - Sparse-MLA forward kernel: run 32-head HIP tiles with one pipeline stage, as 64-head tiles already do. At TP2 each GPU holds 32 of the 64 heads, and the two-stage tile needs 168 KiB of LDS, more than gfx950's 160 KiB. Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
… nightly On ROCm the test adds two flags: - --sglang-disable-shared-experts-fusion: with the shared expert fused into the routed top-k, the R3 capture buffer is one column wider than the ids SGLang captures on ROCm, and the engine dies while capturing the decode graph. The AMD twin test already sets it. - --debug-unified-grad-fused-logprob: Megatron compiles its fused cross entropy with torch.compile into separate no-grad and grad variants, which round differently on ROCm and break the exact first-step ppo_kl check. The switch scores through the grad variant. Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
Two runs on the unpatched 2026-10-05 ROCm10 nightly image took 1026 s and 480 s. Use the longer run x 1.3, rounded up to the next hundred. Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
The ROCm image does not install torchtitan yet, so the test would fail at import. The next PR installs torchtitan in Dockerfile.rocm and enables it. Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
guapisolo
approved these changes
Oct 6, 2026
guapisolo
left a comment
Collaborator
There was a problem hiding this comment.
miles_plugins/models/deepseek_v4/ops/kernel/tilelang_sparse_mla_fwd.py need code owner double check.
Contributor
Author
|
@guapisolo The |
Collaborator
|
not nv path ,save to merge |
This was referenced Oct 20, 2026
This was referenced Oct 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expand MI350 nightly coverage with locally verified tests.
Test coverage
ROCm-only changes
Each change below applies only on ROCm; CUDA runs are unchanged.
scripts/run_deepseek_v4.pyalready does. On four GPUs the converter runs PP4, but Megatron requires all three hash-MoE layers in the embedding stage, so the nightly currently fails at conversion.--dsa-kernel-backend none(no cuDNN DSA on ROCm) and power-of-two FP8 block scales, as inscripts/amd/run_deepseek_v4.py.alltoallMegatron dispatcher (the defaultflexneeds DeepEP, which the ROCm image does not ship).cp_size(TP × CP with--allgather-cp). At TP2 each GPU holds 32 of the 64 heads, and the two-stage sparse-MLA forward tile needs 168 KiB of LDS against gfx950's 160 KiB, so 32-head HIP tiles run one pipeline stage, as 64-head tiles already did.--sglang-disable-shared-experts-fusion(fusing the shared expert makes the R3 capture buffer one column wider than the expert IDs SGLang captures on ROCm) and--debug-unified-grad-fused-logprob(the no-grad and grad compiled variants of the fused cross entropy round differently on ROCm and break the exact first-stepppo_klcheck).Validation
All sixteen tests passed locally on MI355X with ROCm10.