Repository navigation
[ROCm] More gfx1250 models, di, and perf fixes - #65716
jpvillam-amd wants to merge 40 commits into
Conversation
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com> Signed-off-by: Juan Villamizar <jpvillam@amd.com>
* Enable Gluon MoE path for oai-gpt-oss * DSV4 Core dump fix Co-authored-by: Jaden Mathias <jaden.mathias@amd.com> Signed-off-by: Juan Villamizar <jpvillam@amd.com>
* Mori changes * Mori connector patch Signed-off-by: Jaden Mathias <jaden.mathias@amd.com> * changing triton pin to enable oss tp4 Signed-off-by: Daniel <danichan@amd.com> --------- Signed-off-by: Jaden Mathias <jaden.mathias@amd.com> Signed-off-by: Daniel <danichan@amd.com> Co-authored-by: jpvillam <juan.villamizar@amd.com> Co-authored-by: Daniel <danichan@amd.com>
Upgrade TORCH_VERSION/TORCHVISION_VERSION to the 2.13.0/0.28.0 ROCm 10.1 nightly builds. Signed-off-by: Jaden Mathias <jaden.mathias@amd.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Daniel <danichan@amd.com> Co-authored-by: Daniel <danichan@amd.com> Co-authored-by: Daniel Chan <Daniel.Chan@amd.com> Signed-off-by: Juan Villamizar <jpvillam@amd.com>
* flatten 3D ViT activation & dont select AITER unquantized bf16 MOE * applying 2D gate so gfx1250 fallbacks to F.linear for qwen Signed-off-by: Daniel Chan <Daniel.Chan@amd.com> --------- Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
…its() (#1360) and reading the indexer K cache in wrong layout. the indexer K cache is written preshuffled and gfx942/gfx950 read it at decode with deepgemm_fp8_paged_mqa_logits(Preshuffle=True) but gfx1250 fell through to deepgemm_fp8_paged_mqa_logits_stage1, which reads the cache row major. so decode time top-k indexes were wrong. after this fix gfx1250 also takes the preshuffle path. Co-authored-by: Muhammad Ahmed <mm.ahmed2202@gmail.com> Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com>
* changing rocm whl index and sdk version Signed-off-by: Daniel Chan <Daniel.Chan@amd.com> * bumping to newer mori commit to be compatible with sdk Signed-off-by: Daniel Chan <Daniel.Chan@amd.com> * switching to torch 2.13 Signed-off-by: Daniel Chan <Daniel.Chan@amd.com> --------- Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
…a16w16 gluon gemm race issues Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
…, flag not required anymore Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Kimi-K3 enablement on gfx1250: - Opt-in AITER FlyDSL MXFP4 SiTU MoE backend (--moe-backend aiter_flydsl_mxfp4_bf16); SiTUv2 auto resolves to a8w4 on gfx1250. - Pin the KDA chunk kernels to num_stages=1 on gfx1250. - Route gfx1250 MLA prefill to AITER's Triton flash attention. - Keep gfx1250 on the Triton KDA decode; AITER's Gluon KDA decode is not yet validated there. On gfx1250 serve with AITER_FORCE_A8W4=1 and --kda-prefill-backend triton: AITER picks a8w4 there from that flag alone (a4w4 otherwise), and the auto KDA prefill picks AITER's Gluon chunk_kda, which fails to compile there. Signed-off-by: Jaden Mathias <jaden.mathias@amd.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com> Signed-off-by: Juan Villamizar <jpvillam@amd.com>
* using 10/27 sdk Signed-off-by: Daniel Chan <Daniel.Chan@amd.com> * moving sdk to oct 30 to solve a0 issues Signed-off-by: Daniel Chan <Daniel.Chan@amd.com> --------- Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
- AiterW4A4Experts: use_mori_kernels was split upstream into use_mori_epv1_kernels / use_mori_epv2_kernels. Gate the modular expert on MoRI EPv1, which dispatches packed MXFP4 + e8m0 scales; EPv2 dispatches BF16 and remaps expert ids, and is untested with ep_sort_routing. - ruff format fixes on rebased files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Juan Villamizar <jpvillam@amd.com>
The int32 stride cast and req_idx < 0 guard in _gather_block_tables_kernel are no longer needed with the current Triton pin: DeepSeek-V4.1-Flash on gfx1250 runs gsm8k (0.92) and long-prompt stress at concurrency up to 128 with no faults on the upstream kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Juan Villamizar <jpvillam@amd.com>
|
/ci cancel |
|
/ci run |
|
✅ Requested cancellation of 1 CI build for |
|
✅ Triggered Buildkite CI #102242 for commit |
|
/ci retry |
|
✅ Queued 2 failed job(s) for retry in Buildkite CI #102242. |
|
/ci retry |
|
✅ Triggered Buildkite CI #102539 for commit |
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Juan Villamizar <jpvillam@amd.com> # Conflicts: # vllm/models/muse_spark/amd/muse_moe_runner.py # vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
|
/ci retry |
|
✅ No failed, timed-out, or expired jobs need retrying in Buildkite CI #102539. |
|
/ci run |
|
❌ This PR is 12 commits behind upstream |
|
/ci run |
|
❌ This PR is 2 commits behind upstream |
|
/ci run |
|
✅ Triggered Buildkite CI #103607 for commit |
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Upstream the gfx1250 (MI455X) work from the ROCm
gfx1250_perf2branch. Everything new is gated onon_gfx1250()or is opt-in, so gfx942/gfx950 and CUDA behavior should not change. Theone exception is the
rocm_block32_gemmfused-split gate, listed in the fixes table below.New features
--moe-backend aiter_flydsl_mxfp4_bf16(newMxfp4MoeBackend.AITER_FLYDSL_MXFP4_BF16,AiterFlydslMxfp4Experts)VLLM_ROCM_USE_AITER_MOE_SITUV2=autonow resolves toa8w4instead ofa4w4.AiterW4A4Experts, for DP+EP over MoRI EPv1 onaiter_triton_mxfp4_mxfp4ep_sort_routingsorts the dispatched global top-k onto the local experts.gate_validlets GEMM2 skip gate slots owned by other ranks._aiter_w4a4_gemms, which this class shares with the monolithic path.use_mori_epv1_kernelsis set, and does not support EPLB.MuseSparkAiterW4A4Experts, onaiter_triton_mxfp4_mxfp4)gpu_per_node=4.backend: "fabric"(UALink)FabricBackendConfigbackend.Worker._maybe_get_memory_pool_context), so MoRI can export fabric handles for cross-node transfers.gfx1250 correctness fixes
chunk_delta_hand Kimi-K3 KDArecompute_w_unum_stages=1on gfx1250. Deeper pipelines corrupt state and drop gsm8k to 0. The root cause is not established.compress_norm_rope_store_tritongemm_a8w8_blockscale_bpreshufflepreprocessing, which has no gfx1250 path. Don't fuse the inverse RoPE into sparse decode.rocm_fp8_paged_mqa_logitsFlashAttnPrefillBackendreturn_lse, then transpose the LSE into vLLM's layout.activationthrough to the weight conversion. Usescale_kwidth=8forAITER_TRITON_MXFP4_BF16with SwiGLU-OAI and 4 otherwise, so the swizzleAITER_MXFP4_BF16on gfx1250rocm_unquantized_gemm_implgemm_a16w16only for 2D activations.rocm_block32_gemmDocker (gfx1250 only)
Dockerfile.rocm_base_gfx1250:2958fce6.98c1cee, built against the system AITER. The FA stage now runs after the AITER stage.torchvision[device-gfx1250]and the gRPC/protobuf dev packages.Dockerfile.rocm_gfx1250:12225a8, which fixes an a16w16 gluon GEMM race.Test Result