Skip to content

[AMD] [GLM-5.3-Flash Day 0] Build the fused DSA k-pool top-k JIT kernel on HIP - #39339

Merged
hnyls2002 merged 6 commits into
sgl-project:mainfrom
Jacob0226:jacob/glm53-day0-kpool-topk-hip
Sep 21, 2026
Merged

hnyls2002 merged 6 commits into
sgl-project:mainfrom
Jacob0226:jacob/glm53-day0-kpool-topk-hip

Conversation

@Jacob0226

@Jacob0226 Jacob0226 commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

GLM-5.3-Flash cannot run on ROCm: the kernel that picks its sparse-attention candidates fails to build under hipcc.

fatal error: 'cuda_fp16.h' file not found
no member named 'cudaFuncSetAttribute' in the global namespace

Both come from the same file assuming CUDA. This is a hard stop rather than a slow path — GLM-5.3-Flash's pooled top-k has no other kernel to fall back to, so the indexer cannot run at all.

Scope

Two edits, both inside USE_ROCM blocks that already exist: give the fp16 include a HIP arm, and add two names to the shim's alias list. Every CUDA line in the diff is unchanged context. No kernel body, launch configuration, tile size or numeric path moves; no environment variable, no architecture gate.

The kernel has no direct test on any platform — CUDA only reaches it through a full GLM-5.3-Flash serving test, which JIT-compiles it every commit. This adds a focused one on the mi35x per-PR suite.

Test plan

Unit base rocm/sgl-dev:v0.5.18-rocm720-mi35x-20260901, the image family pr-test-amd-rocm720.yml resolves to, on xinyuan/glm-5.3-flash-support @ 515e865189 with no other PRs applied. Both builds run the same new test file; the shim aliases are the only variable. JIT caches were cleared between builds and the build hashes differ, so neither result is a stale artifact.

Build Result
Baseline 4 errors — every case dies in the JIT build, group_topk 128 and 512 alike, on 'cuda_fp16.h' file not found and no member named 'cudaFuncSetAttribute' in the global namespace
This PR 4 passed on gfx950

End to end, since no single PR here can serve the model on its own: main plus the eight open Day-0 PRs — #39338, #39339, #39340, #39341, #38545, #38546, #38547 and #39317 — which all merge cleanly onto it. rocm/sgl-dev:v0.5.19-rocm720-mi35x-20260909, MI355X TP4, decode CUDA graphs on, 1,319 GSM8K examples, one scoring pass each. Only the checkpoint differs between the two rows:

Checkpoint GSM8K
zai-org/GLM-5.3-Flash, block FP8 97.27%
amd/GLM-5.3-Flash-Quark-MXFP4 96.51%

Accuracy: not attributable to this PR in isolation, and there is no standalone baseline — GLM-5.3-Flash does not start on ROCm without the rest of the stack.


CI States

Latest PR Test (Base): ⏳ Run #35659589049
Latest PR Test (Extra): ❌ Run #35659588661
Latest PR Test (AMD ROCm 10): ⏳ Run #35659588779

jit/csrc/dsa/kpool_topk_transform.cuh includes <cuda_fp16.h> directly and
reaches ::cudaFuncSetAttribute through no alias, so hipcc cannot compile
it. On gfx950 the JIT build fails with "fatal error: 'cuda_fp16.h' file
not found", and after that with "no member named 'cudaFuncSetAttribute'
in the global namespace".

This is a hard stop rather than a slow path. GLM-5.3-Flash pools
index_topk=2048 over index_kpool=4, so group_topk is 512, and
kpool_fp8_index routes every pooled budget in (128, 160, 192, 224, 256,
512) to this JIT kernel -- the fast_topk_v2 fallback covers group_topk
2048 alone. The indexer cannot run at all on ROCm.

Give the fp16 include a USE_ROCM arm, matching
jit/csrc/elementwise/fused_metadata_copy.cuh in the same tree, and add
cudaFuncSetAttribute to the USE_ROCM alias list in sgl_kernel/utils.cuh
beside the existing cudaGetLastError and cudaDeviceGetAttribute entries.
Both CUDA lines stay untouched.

Add a focused parity test. The kernel has no direct coverage on any
platform; CUDA reaches it end to end through the GLM-5.3-Flash B200
serving test, which JIT-compiles it every commit, while AMD has no
equivalent. Registered on the mi35x per-PR suite only.

Recovers sgl-project#37563, which GitHub closed when its base branch
xinyuan/glm-5.3-flash-support was merged into main, and supersedes the
sgl-project#38542 replacement. The test-file placement under the registered-test
taxonomy is Raiden-Makoto's from sgl-project#38542.

Co-Authored-By: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
@Jacob0226
Jacob0226 requested a review from BBuf as a code owner September 15, 2026 06:32
@Jacob0226

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1am9trash pushed a commit that referenced this pull request Sep 17, 2026
…el on HIP

Squashed from #39339, which carries the same change
against main and remains open there for review.

Co-Authored-By: Jacob Chang <jacchang@amd.com>
@kevin-mii

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/utils/test_numa_utils.py::TestB200NumaTopology.test_gpu_numa_mapping

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/utils/test_numa_utils.py::TestB200NumaTopology.test_gpu_numa_mapping:

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/utils/test_numa_utils.py TestB200NumaTopology.test_gpu_numa_mapping

🚀 4-gpu-b200 (1 test): ❌ View workflow run

cd test/ && python3 registered/utils/test_numa_utils.py TestB200NumaTopology.test_gpu_numa_mapping

@kevin-mii

kevin-mii commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

MI35X stage-b run for test_kpool_topk_transform_fused.py on ab0edf2b:

https://github.com/sgl-project/sglang/actions/runs/35658461774

Passed. TestKpoolTopkTransformFused ran 4 tests in test/registered/e2e/attention/test_kpool_topk_transform_fused.py and they were OK. The stage-b-test-1-gpu-small-amd-mi35x job succeeded.

The file was an e2e test on the MI35x stage-b suite, which waits on ROCm
stage A and is not the job this kernel diff runs. JIT correctness tests
live under kernels/ops and register on jit-kernel-unit.

Co-authored-by: Cursor <cursoragent@cursor.com>
@hnyls2002
hnyls2002 merged commit 8bde82c into sgl-project:main Sep 21, 2026
97 of 133 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants