Repository navigation
[Fix] [AMD] Quark: fix Quark block-FP8 loading in DeepSeek loader - #40776
Merged
Merged
Conversation
clintg6
requested review from
Fridge003,
ch-wan,
fzyzcjy and
ispobock
as code owners
September 22, 2026 21:05
ColinZ22
reviewed
Sep 23, 2026
ColinZ22
left a comment
Contributor
There was a problem hiding this comment.
Changes LGTM, @HaiShaw @kkHuang-amd could we get the run-ci flag for this PR? Thanks
HaiShaw
approved these changes
Sep 23, 2026
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
Contributor
Author
|
@BowenBao everything ran as expected with base z-ai/GLM-5.3. |
Contributor
Author
@HaiShaw 20%+ TTT uplift and growing with concurrency at TP4.
|
Contributor
|
@clintg6 Are you still planning to push this PR forward? It has had no activity since Oct 12 and is still waiting on code-owner review. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
hubertlu-tw
enabled auto-merge (squash)
October 22, 2026 20:57
hubertlu-tw
disabled auto-merge
October 22, 2026 21:23
hubertlu-tw
enabled auto-merge (squash)
October 22, 2026 22:15
hubertlu-tw
disabled auto-merge
October 22, 2026 22:16
hubertlu-tw
enabled auto-merge (squash)
October 23, 2026 16:56
hubertlu-tw
disabled auto-merge
October 23, 2026 17:01
This was referenced Oct 24, 2026
4 of 5 tasks
xiaofei-zheng
added a commit
to xiaofei-zheng/sglang
that referenced
this pull request
Oct 28, 2026
…d pin its digest
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fix Quark block-FP8 loading in DeepSeek loader
Summary
Enable
amd/GLM-5.3-Quark-MXFP4-AttnFP8(GlmMoeDsaForCausalLM) onSGLang/ROCm by completing the DeepSeek-family weight-loading path for Quark
checkpoints that combine:
SGLang PR #38546 already provides the shared Quark
layer_quant_configresolution and block-FP8 linear/MoE dispatch. It also handles scale names in
the
glm5_next.pyloader used by GLM-5.3-Flash.However, the non-Flash
GlmMoeDsaForCausalLMarchitecture instead loads throughDeepseekV2WeightLoaderMixin, where three compatibility gaps remained.Bug
Quark block-scale parameter name
Quark serializes block-FP8 scales as:
The shared
Fp8LinearMethodregisters the corresponding runtime parameter as:Consequences in the DeepSeek loader included missing parameters for ordinary
attention projections and fused
q_a_proj + kv_a_proj_with_mqascales.MXFP4 scales cannot be renamed unconditionally because they legitimately use
weight_scale.Reproduced on SGLang
mainwithout this change:The runtime parameter at that point is
fused_qkv_a_proj_with_mqa.weight_scale_inv; the checkpoint name has alreadybeen fused correctly, but its Quark scale suffix has not been adapted.
Fused indexer scale classification
_load_fused_indexer_wk()recognized only the_invsuffix as a scale. AQuark
weight_scaletensor could therefore be treated as a projection weight.This is a silent-corruption path rather than a guaranteed startup failure.
MLA absorption used the global configuration
post_load_weights()derivedkv_b_projblock geometry from the globalquantization configuration. For this mixed checkpoint, the global configuration
is MXFP4 and does not describe the block-FP8
kv_b_proj. The exact scheme isalready available on
kv_b_proj.quant_method.Changes
deepseek_weight_loader.py_resolve_quark_fp8_block_scale_name().weight_scaletoweight_scale_invonly when the originalparameter is absent and the
_invparameter exists.weight_scaleparameters remain unchanged.parameter-loading fallthrough.
weight_scaleandweight_scale_invin the fused indexerpath.
_resolve_layer_weight_block_size().No model-name or checkpoint-name gate is added. The compatibility behavior is
driven by the registered parameter and resolved quantization method.
Files
Production:
Tests:
Tests
The new CPU regression coverage verifies:
weight_scaletoweight_scale_invmapping;weight_scale_inv;Focused validation:
Formatting and repository checks:
MI355X validation
Configuration:
Observed:
GSM8K, 400 questions:
The measured accuracy is consistent with the checkpoint's reported result.
GLM-5.3-Flash regression validation
amd/GLM-5.3-Flash-Quark-MXFP4was also tested to verify that deriving MLAblock geometry from the resolved
kv_b_proj.quant_methoddoes not regress theFlash path.
Current
mainloaded all 62 shards successfully, then failed during decodegraph capture in the independent CUDA-only
IndexerKPoolpath:The loader change was therefore also applied to the ROCm Flash support branch
(
amd_glm5.3_flashat2fa010a4fb), which contains the ROCm k-poolimplementation used by the validated Flash deployment.
Observed on 4× MI355X:
GSM8K via
benchmark/gsm8k/bench_sglang.py, 400 questions:This benchmark invocation is not directly comparable to the model card's
thinking-enabled
sgl-evalrecipe, but it provides a focused before-mergeserving and generation regression check.
Flash performance A/B
The concurrency benchmark was repeated on the same support-branch revision,
hardware, checkpoint, and launch configuration with only this loader change
removed. Workload: random 8,192 input tokens and 1,024 output tokens.
Median ITL differed by only 0.01-0.02 ms across all three points. The observed
throughput and latency differences are below 0.3% and consistent with normal
run-to-run measurement noise; no material Flash performance regression was
observed.
Scope
This is a follow-up to the Quark mixed-precision support in #38546. It does not
replace or modify GLM-5.3-Flash's MoE/AITER layout, clamp, padding, or
model-specific loading logic. It completes the separate DeepSeek loader path
used by
GlmMoeDsaForCausalLM.CI States
Latest PR Test (Base): ✅ Run #39856212203
Latest PR Test (Extra): ❌ Run #39856211732
Latest PR Test (AMD ROCm 10): ❌ Run #39856212073