Skip to content

[Feat] [AMD] srt: Add muse spark ROCm support (4/5) - #44667

Closed
Fridge003 wants to merge 1 commit into
mainfrom
codex/muse-spark-stack-04-amd
Closed

Fridge003 wants to merge 1 commit into
mainfrom
codex/muse-spark-stack-04-amd

Conversation

@Fridge003

@Fridge003 Fridge003 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Split #44530 into five review layers. This is layer 4/5, now based on main after #44666 merged.

Modifications

Adds the independent ROCm runtime, AITER patches and validation records, ROCm image and launch script, plus AMD-specific changes in shared kernels, AITER attention, environment settings, and model loading. Includes the ROCm LDS optimization in moe_align_kernel.cu.

This layer changes 44 files (+11,551 / -41) relative to its immediate base. Every split commit preserves all 32 Co-authored-by trailers from the original commit.

Validation

  • Rebased onto main at d33dd0566a1a8b77bdad83b64cbc45510307052b after PR3 merged. Stable patch ID and range diff confirm that this layer's patch is unchanged. Preserved the AMD commit's original author, author date, message and all 32 coauthor trailers.
  • Updated the ROCm text config/model names to match PR3's MuseSparkTextConfig / MuseSparkTextForCausalLM, and kept only the conditional-generation architecture as the registered model entry. The ROCm model is AST-equivalent after these mechanical name/entry updates.
  • Reviewed the range diff; syntax validation passed for all 28 Python files in this layer and pre-commit passed for affected files. No AMD hardware run was performed for this restack.

Stack

Merge/review order, from main upward:

  1. #44664 — CUDA Docker and DeepGEMM patches (merged)
  2. #44665 — Kernels and collective GEMM (merged)
  3. #44666 — SRT and registered tests (merged)
  4. #44667 — AMD / ROCm support
  5. #44668 — Helion verify megakernels (Add SSL Cert Functionality #224)

The original #44530 remains unchanged as the full-stack reference.


CI States

Latest PR Test (Base): ❌ Run #39109929535
Latest PR Test (Extra): ❌ Run #39109929490
Latest PR Test (AMD ROCm 10): ❌ Run #39109929504

@github-actions github-actions Bot added documentation Improvements or additions to documentation amd sgl-kernel blackwell SM100/SM120 jit-kernel labels Oct 16, 2026
@Fridge003 Fridge003 changed the title [Feat] [AMD] muse-spark: Add ROCm implementation and deployment [MuseSpark][4/5] Add ROCm implementation and deployment Oct 16, 2026
@Fridge003
Fridge003 force-pushed the codex/muse-spark-stack-03-srt-tests branch from cfaf2cf to 902859b Compare October 16, 2026 05:20
@Fridge003
Fridge003 force-pushed the codex/muse-spark-stack-04-amd branch from a2c37f0 to 046cd52 Compare October 16, 2026 05:20
@Fridge003
Fridge003 force-pushed the codex/muse-spark-stack-03-srt-tests branch from 902859b to f60333c Compare October 16, 2026 06:29
@Fridge003
Fridge003 force-pushed the codex/muse-spark-stack-04-amd branch from caf6b36 to cbd25ea Compare October 16, 2026 06:51
@Fridge003
Fridge003 force-pushed the codex/muse-spark-stack-03-srt-tests branch from 1e49efb to a5abcfc Compare October 16, 2026 20:50
@Fridge003
Fridge003 force-pushed the codex/muse-spark-stack-04-amd branch from cbd25ea to 2994494 Compare October 16, 2026 20:50
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@Fridge003 Fridge003 changed the title [MuseSpark][4/5] Add ROCm implementation and deployment [Feat] [AMD] srt: Add muse spark ROCm support (4/5) Oct 18, 2026
@Fridge003
Fridge003 force-pushed the codex/muse-spark-stack-04-amd branch 2 times, most recently from 813b5bd to 1057181 Compare October 18, 2026 07:57
Base automatically changed from codex/muse-spark-stack-03-srt-tests to main October 18, 2026 08:00
@mintlify

mintlify Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
sglang-doc 🟢 Ready View Preview Oct 18, 2026, 8:09 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

Split layer 4/5 from #44530.
Source commit: a55ab7d

Co-authored-by: 1am9trash <1am9trash@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ethan Che <eche@meta.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: HaiShaw <hixiao@gmail.com>
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Jimmy Shong <jimmysh341@gmail.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com>
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: ethche <ethche@gmail.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
Co-authored-by: yhyang201 <yhyang201@users.noreply.github.com>
@BBuf

BBuf commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Hi @Fridge003, thanks for the ROCm bring-up! A heads-up so this doesn't surprise you at rebase time: #45112 regroups sglang.kernels by operator, following the existing README rule that model-specific kernels live inside a logical group rather than in a model-named top-level group. As part of that, python/sglang/kernels/ops/muse_spark/ was dissolved. The files this PR touches now live at:

old ops/muse_spark/… new
fused_muse_spark_routing.py ops/moe/muse_spark_routing.py
fused_norm_gate.py ops/layernorm/muse_spark_norm_gate.py
fused_norm_residual.py ops/layernorm/muse_spark_norm_residual.py
fused_op_tuning.py ops/layernorm/_muse_spark_tuning.py (configs under ops/layernorm/configs/muse_spark/)
fused_qk_norm.py ops/attention/muse_spark_qk_norm_rope_store.py

Things that changed beyond the paths:

  • FusedPostNormResidualPreNorm now takes post_norm_is_input_scale and use_residual_gate instead of importing the srt layer classes. forward_with_ag is gone; the model calls communication.muse_spark_all_gather_norm_residual(..., **op.jit_kernel_kwargs()) directly.
  • FinalizePushWorkspace moved to srt/layers/communication/muse_spark_finalize_workspace.py.

Kernel code itself was moved byte-for-byte, so the behaviour your PR relies on is unchanged. One optional suggestion: the kernels under srt/rocm_muse_spark/ (e.g. triton_sum, the fused norm/routing variants) could sit next to their CUDA versions as HIP backends of the same ops in sglang.kernels.ops.<group>, so both platforms share one entry point. Happy to help with the rebase if that's useful. Thanks!

@Fridge003

Copy link
Copy Markdown
Collaborator Author

See #47371

@Fridge003 Fridge003 closed this Nov 4, 2026
RolaoDenthu pushed a commit that referenced this pull request Nov 6, 2026
docker/muse_spark_image/ from the image tree, adapted:

* Dockerfile: the day0 image (pinned digest) with python/sglang swapped for
  this tree, aiter patch 05 applied to /sgl-workspace/aiter (git apply, two
  Python/Triton files, nothing rebuilt), and the 16-row Muse aiter bf16 GEMM
  CSV. No ENV lines. The build checks the editable mapping, byte-compiles the
  package and the patched aiter files and verifies the manifest.
* prepare_context.sh <commit> <dir>: build context from one commit (git
  archive), plus the manifest. The two aiter files patch 05 produces are
  computed from the base image's own copies, so the manifest pins them too.
* make_manifest.py (schema 2): tree commit, base (#44667 on main d33dd05),
  files changed vs #44667, sha256 of every package file, the aiter CSV, the
  two patched aiter files and patch 05 itself. verify_manifest.py unchanged.

The CSV is force-added (*.csv is gitignored).

(cherry picked from commit 58b7898)
(cherry picked from commit 291384c26afeb19afce5d73024b78ea17d27c744)

This branch was successfully deployed

1 active deployment
staging - docs — 37197b72 Deployed Oct 18, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd blackwell SM100/SM120 documentation Improvements or additions to documentation jit-kernel sgl-kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants