Skip to content

[AMD] ci: expand ROCm10 nightly coverage on MI350 - #3910

Merged
guapisolo merged 11 commits into
radixark:mainfrom
XinyuJiangCMU:amd/expand-rocm10-nightly-20261004
Oct 6, 2026
Merged

guapisolo merged 11 commits into
radixark:mainfrom
XinyuJiangCMU:amd/expand-rocm10-nightly-20261004

Conversation

@XinyuJiangCMU

@XinyuJiangCMU XinyuJiangCMU commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Expand MI350 nightly coverage with locally verified tests.

Test coverage

Test Nightly GPUs Local validation
Qwen3.5-0.8B checkpoint 2 Passed in 718 s; synchronous and asynchronous save/load with model-hash checks (zero-gradient run).
Qwen3-0.6B FSDP disk-delta 4 Passed in 353 s.
DeepSeek-V4 indexer CP balance 4 Passed in 24 s; torch top-k only, FlashInfer skipped.
GLM-4.7-Flash BF16 spec / MTP / R3, TP2/PP2 4 Passed in 1,086 s.
Qwen3-30B-A3B BF16 Muon, TP2/CP2 4 Passed in 741 s.
Qwen3.5-35B-A3B BF16 Muon, TP2/PP2 4 Passed in 934 s.
Qwen3.5-35B-A3B FP8 rollout, TP4/CP2 4 Passed in 1,192 s.
Inkling-Small 4-layer LoRA 4 Passed in 310 s.
Kimi-K3 4-layer 8 Passed in 786 s.
Qwen3.8-Next 4-layer 8 Passed in 405 s.
DeepSeek-V4-Flash 4-layer, Megatron implementation 4 Passed in 631 s.
DeepSeek-V4-Flash multi-role session server 4 Passed in 2,272 s.
GLM-4.7-Flash multi-role session server 2 Passed in 460 s.
Torch-native offload 2 Passed in 20 s with the pytest entry point from #3889.
DeepSeek-V4-Flash 4-layer, TP2/CP2 4 Passed in 536 s.
Qwen3.5-35B-A3B BF16 fully-async R3, TP1/CP1 8 Passed in 1,026 s.

ROCm-only changes

Each change below applies only on ROCm; CUDA runs are unchanged.

  • DeepSeek-V4-Flash 4-layer (already on the nightly): the AMD launcher converts the checkpoint on one GPU, as scripts/run_deepseek_v4.py already does. On four GPUs the converter runs PP4, but Megatron requires all three hash-MoE layers in the embedding stage, so the nightly currently fails at conversion.
  • Inkling-Small LoRA: Triton attention (FA4 is CUDA-only).
  • Qwen3.8-Next: FLA for training and Triton for linear-attention prefill (FlashQLA and FlashInfer are CUDA-only).
  • Kimi-K3: BF16 rollout on SGLang's Triton MoE runner (the MXFP4 rollout uses Marlin, which is CUDA-only).
  • DeepSeek-V4-Flash, Megatron implementation: --dsa-kernel-backend none (no cuDNN DSA on ROCm) and power-of-two FP8 block scales, as in scripts/amd/run_deepseek_v4.py.
  • Qwen3.5-35B-A3B FP8 rollout: alltoall Megatron dispatcher (the default flex needs DeepEP, which the ROCm image does not ship).
  • DeepSeek-V4-Flash TP2/CP2: the AMD launcher accepts cp_size (TP × CP with --allgather-cp). At TP2 each GPU holds 32 of the 64 heads, and the two-stage sparse-MLA forward tile needs 168 KiB of LDS against gfx950's 160 KiB, so 32-head HIP tiles run one pipeline stage, as 64-head tiles already did.
  • Qwen3.5-35B-A3B fully-async R3: --sglang-disable-shared-experts-fusion (fusing the shared expert makes the R3 capture buffer one column wider than the expert IDs SGLang captures on ROCm) and --debug-unified-grad-fused-logprob (the no-grad and grad compiled variants of the fused cross entropy round differently on ROCm and break the exact first-step ppo_kl check).

Validation

All sixteen tests passed locally on MI355X with ROCm10.

Register the checkpoint test in the 2-GPU suite and FSDP disk-delta,
DSV4 indexer CP balance, GLM TP2/PP2, Qwen3 Muon TP2/CP2 and Qwen3.5
Muon TP2/PP2 in the 4-GPU suite.

Set runtime estimates from local MI355X ROCm10 runs. Keep test
configurations, checker settings and CUDA registrations unchanged.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

XinyuJiangCMU and others added 10 commits October 5, 2026 18:18
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
fa4 is CUDA-only, so on ROCm the test appends --sglang-attention-backend
triton (the trailing flag wins), the backend scripts/amd/run_inkling.py
already uses.

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
Requires the pytest entrypoints and a ROCm image with torchtitan installed.

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
- Kimi-K3 4-layer: serve the rollout in BF16 on ROCm, as the LoRA variant does.
- Qwen3.8-Next 4-layer: train with FLA and prefill with Triton on ROCm; FlashQLA and FlashInfer are CUDA-only.
- DeepSeek-V4-Flash 4-layer Megatron implementation: ROCm has no cuDNN DSA kernel, so run
  --dsa-kernel-backend none and pin NVTE_FP8_BLOCK_SCALING_FP32_SCALES=0 as scripts/amd/run_deepseek_v4.py does.
- DeepSeek-V4 and GLM-4.7 multi-role session-server tests: register only.

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
…nightly

- scripts/amd/run_deepseek_v4.py: accept cp_size and split the single-node actor GPUs as TP x CP with
  --allgather-cp. CP1 flags are unchanged.
- Sparse-MLA forward kernel: run 32-head HIP tiles with one pipeline stage, as 64-head tiles already do.
  At TP2 each GPU holds 32 of the 64 heads, and the two-stage tile needs 168 KiB of LDS, more than
  gfx950's 160 KiB.

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
… nightly

On ROCm the test adds two flags:
- --sglang-disable-shared-experts-fusion: with the shared expert fused into the routed top-k, the R3
  capture buffer is one column wider than the ids SGLang captures on ROCm, and the engine dies while
  capturing the decode graph. The AMD twin test already sets it.
- --debug-unified-grad-fused-logprob: Megatron compiles its fused cross entropy with torch.compile into
  separate no-grad and grad variants, which round differently on ROCm and break the exact first-step
  ppo_kl check. The switch scores through the grad variant.

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
Two runs on the unpatched 2026-10-05 ROCm10 nightly image took 1026 s and
480 s. Use the longer run x 1.3, rounded up to the next hundred.

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
The ROCm image does not install torchtitan yet, so the test would fail at
import. The next PR installs torchtitan in Dockerfile.rocm and enables it.

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>

@guapisolo guapisolo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

miles_plugins/models/deepseek_v4/ops/kernel/tilelang_sparse_mla_fwd.py need code owner double check.

@XinyuJiangCMU

Copy link
Copy Markdown
Contributor Author

@guapisolo The tilelang_sparse_mla_fwd.py change is inside the is_hip branch, so the NV path is untouched.

@guapisolo

Copy link
Copy Markdown
Collaborator

not nv path ,save to merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants