Repository navigation
[Feat] glm5: Expose per-layer MLA key buffer and use_fused_topk in binding - #45765
Merged
Merged
Conversation
5 tasks done
This comment has been minimized.
This comment has been minimized.
Collaborator
|
Please rebase this onto the latest main and move these changes to |
Collaborator
|
Could you update this branch with the latest main and resolve the merge conflicts? Please ping me once it's ready for another look. Thanks! |
Expose native MLA KV cache tensors and fused TopK state through the NVIDIA kernel plugin binding. Track per-layer cache identity on first access and reject changes to tensor addresses, shapes, dtypes or strides. Add regression tests for native cache views, buffer replacement and stride changes. Validation: - GLM5 contract and binding tests passed. - TP4/TP8 GLM5 kernel tests: 18 passed. - GLM-5.3 TP4/TP8 serving and GSM8K: 256 prompts per TP. - Mega-enabled 8192/1024 serving benchmarks passed at concurrency 1/2.
Torches97
force-pushed
the
glm5_mega_dev
branch
from
October 26, 2026 05:52
24ca529 to
ecf63c0
Compare
This comment has been minimized.
This comment has been minimized.
Contributor
Author
|
@BBuf Thanks for your advice. Currently, I have rebased the sglang PR #45765 into main. And sgl-project/sgl-nvidia-mega#22 includes the new SGLANG_READS now. All the changes have been tested. |
BBuf
approved these changes
Oct 26, 2026
Yuxingwang-intel
pushed a commit
to Yuxingwang-intel/sglang
that referenced
this pull request
Nov 6, 2026
…nding (sgl-project#45765) Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This pr is the sglang modification for sgl-nvidia-mega pr sgl-project/sgl-nvidia-mega#22 to support glm5 model.
Modifications
Accuracy Tests
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #40493908725
Latest PR Test (Extra): ❌ Run #40493908741
Latest PR Test (AMD ROCm 10): ❌ Run #40493908705