Skip to content

[Feat] glm5: Expose per-layer MLA key buffer and use_fused_topk in binding - #45765

Merged
BBuf merged 2 commits into
sgl-project:mainfrom
Torches97:glm5_mega_dev
Oct 29, 2026
Merged

BBuf merged 2 commits into
sgl-project:mainfrom
Torches97:glm5_mega_dev

Conversation

@Torches97

@Torches97 Torches97 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

This pr is the sglang modification for sgl-nvidia-mega pr sgl-project/sgl-nvidia-mega#22 to support glm5 model.

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #40493908725
Latest PR Test (Extra): ❌ Run #40493908741
Latest PR Test (AMD ROCm 10): ❌ Run #40493908705

@github-actions

This comment has been minimized.

@BBuf

BBuf commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator

Please rebase this onto the latest main and move these changes to python/sglang/srt/kernel_plugins/nvidia/binding.py, with the tests under test/registered/unit/kernel_plugins/nvidia/. The old srt/megakernels paths were removed in #45389. The cache identity and stride regression tests are worth keeping.

@BBuf

BBuf commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator

Could you update this branch with the latest main and resolve the merge conflicts? Please ping me once it's ready for another look. Thanks!

Expose native MLA KV cache tensors and fused TopK state through
the NVIDIA kernel plugin binding.

Track per-layer cache identity on first access and reject changes
to tensor addresses, shapes, dtypes or strides. Add regression tests
for native cache views, buffer replacement and stride changes.

Validation:
- GLM5 contract and binding tests passed.
- TP4/TP8 GLM5 kernel tests: 18 passed.
- GLM-5.3 TP4/TP8 serving and GSM8K: 256 prompts per TP.
- Mega-enabled 8192/1024 serving benchmarks passed at concurrency 1/2.
@github-actions

This comment has been minimized.

@Torches97

Copy link
Copy Markdown
Contributor Author

@BBuf Thanks for your advice. Currently, I have rebased the sglang PR #45765 into main. And sgl-project/sgl-nvidia-mega#22 includes the new SGLANG_READS now. All the changes have been tested.

@BBuf BBuf added the run-ci CI: run the baseline test suite on this PR label Oct 27, 2026
@BBuf BBuf changed the title [Feat] [sgl-nvidia-mega] [glm5]: Expose per-layer MLA key buffer and use_fused_topk in binding [Feat] glm5: Expose per-layer MLA key buffer and use_fused_topk in binding Oct 27, 2026
@BBuf
BBuf merged commit d39c8f9 into sgl-project:main Oct 29, 2026
375 of 428 checks passed
Yuxingwang-intel pushed a commit to Yuxingwang-intel/sglang that referenced this pull request Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants