Skip to content

[Fix] [AMD] Quark: fix Quark block-FP8 loading in DeepSeek loader - #40776

Merged
HaiShaw merged 8 commits into
sgl-project:mainfrom
clintg6:quark-fix/GLM-5.3
Oct 23, 2026
Merged

HaiShaw merged 8 commits into
sgl-project:mainfrom
clintg6:quark-fix/GLM-5.3

Conversation

@clintg6

@clintg6 clintg6 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Fix Quark block-FP8 loading in DeepSeek loader

Summary

Enable amd/GLM-5.3-Quark-MXFP4-AttnFP8 (GlmMoeDsaForCausalLM) on
SGLang/ROCm by completing the DeepSeek-family weight-loading path for Quark
checkpoints that combine:

  • globally quantized OCP MXFP4 experts; and
  • explicitly configured block-FP8 attention linears.

SGLang PR #38546 already provides the shared Quark layer_quant_config
resolution and block-FP8 linear/MoE dispatch. It also handles scale names in
the glm5_next.py loader used by GLM-5.3-Flash.

However, the non-Flash GlmMoeDsaForCausalLM architecture instead loads through
DeepseekV2WeightLoaderMixin, where three compatibility gaps remained.

Bug

Quark block-scale parameter name

Quark serializes block-FP8 scales as:

*.weight_scale

The shared Fp8LinearMethod registers the corresponding runtime parameter as:

*.weight_scale_inv

Consequences in the DeepSeek loader included missing parameters for ordinary
attention projections and fused q_a_proj + kv_a_proj_with_mqa scales.
MXFP4 scales cannot be renamed unconditionally because they legitimately use
weight_scale.

Reproduced on SGLang main without this change:

Scheduler hit an exception:
  File "sglang/srt/models/deepseek_common/deepseek_weight_loader.py",
       line 464, in do_load_weights
    param = params_dict[param_name]
KeyError:
  'model.layers.0.self_attn.fused_qkv_a_proj_with_mqa.weight_scale'

The runtime parameter at that point is
fused_qkv_a_proj_with_mqa.weight_scale_inv; the checkpoint name has already
been fused correctly, but its Quark scale suffix has not been adapted.

Fused indexer scale classification

_load_fused_indexer_wk() recognized only the _inv suffix as a scale. A
Quark weight_scale tensor could therefore be treated as a projection weight.
This is a silent-corruption path rather than a guaranteed startup failure.

MLA absorption used the global configuration

post_load_weights() derived kv_b_proj block geometry from the global
quantization configuration. For this mixed checkpoint, the global configuration
is MXFP4 and does not describe the block-FP8 kv_b_proj. The exact scheme is
already available on kv_b_proj.quant_method.

Changes

deepseek_weight_loader.py

  1. Add _resolve_quark_fp8_block_scale_name().
    • Rewrite weight_scale to weight_scale_inv only when the original
      parameter is absent and the _inv parameter exists.
    • Genuine MXFP4 weight_scale parameters remain unchanged.
  2. Apply the alias after fused QKV-A name resolution and in the ordinary
    parameter-loading fallthrough.
  3. Recognize both weight_scale and weight_scale_inv in the fused indexer
    path.
  4. Add _resolve_layer_weight_block_size().
    • Prefer the constructed layer's resolved quantization method.
    • Retain the existing config-level fallback for compatibility.
  5. Use the resolved per-layer block size for MLA absorption and UE8M0 handling.

No model-name or checkpoint-name gate is added. The compatibility behavior is
driven by the registered parameter and resolved quantization method.

Files

Production:

python/sglang/srt/models/deepseek_common/deepseek_weight_loader.py

Tests:

test/registered/unit/models/test_deepseek_quark_block_fp8_loading.py

Tests

The new CPU regression coverage verifies:

  • conditional weight_scale to weight_scale_inv mapping;
  • genuine MXFP4 scale names remain unchanged;
  • fused QKV-A Quark scales concatenate and load into the registered parameter;
  • ordinary attention scales load into weight_scale_inv;
  • Quark indexer scales are not misclassified as weights; and
  • per-layer quantization geometry takes precedence over a config-level summary.

Focused validation:

24 passed
3 existing GLM tests passed, 27 subtests passed
13 existing Quark utility tests passed

Formatting and repository checks:

isort: passed
ruff: passed
ruff-format: passed
codespell: passed
registered-test validation: passed
git diff --check: passed

MI355X validation

Configuration:

sglang serve \
  --trust-remote-code \
  --model-path /models/GLM-5.3-MXFP4 \
  --tp 4 \
  --ep-size 4 \
  --kv-cache-dtype fp8_e4m3 \
  --dsa-prefill-backend triton \
  --dsa-decode-backend triton \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 5 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 6 \
  --chunked-prefill-size 131072 \
  --mem-fraction-static 0.80 \
  --watchdog-timeout 1200 \
  --host 0.0.0.0 \
  --port 8000

Observed:

  • all 141 target-model shards loaded;
  • the EAGLE/NextN model loaded;
  • target-verify, draft-decode, and draft-extend graphs captured;
  • health endpoint returned HTTP 200; and
  • no missing-parameter warnings were observed.

GSM8K, 400 questions:

Accuracy:          0.912
Invalid:           0.000
Latency:           24.621 s
Output throughput: 1569.041 token/s

The measured accuracy is consistent with the checkpoint's reported result.

GLM-5.3-Flash regression validation

amd/GLM-5.3-Flash-Quark-MXFP4 was also tested to verify that deriving MLA
block geometry from the resolved kv_b_proj.quant_method does not regress the
Flash path.

Current main loaded all 62 shards successfully, then failed during decode
graph capture in the independent CUDA-only IndexerKPool path:

NotImplementedError: kpool indexer is only supported on CUDA

The loader change was therefore also applied to the ROCm Flash support branch
(amd_glm5.3_flash at 2fa010a4fb), which contains the ROCm k-pool
implementation used by the validated Flash deployment.

Observed on 4× MI355X:

  • all 62 shards loaded;
  • decode graphs captured;
  • health endpoint returned HTTP 200;
  • internal chat-completion warm-up returned HTTP 200; and
  • no missing-scale or loader errors were observed.

GSM8K via benchmark/gsm8k/bench_sglang.py, 400 questions:

Accuracy:          0.920
Invalid:           0.000
Latency:           74.894 s
Output throughput: 497.087 token/s

This benchmark invocation is not directly comparable to the model card's
thinking-enabled sgl-eval recipe, but it provides a focused before-merge
serving and generation regression check.

Flash performance A/B

The concurrency benchmark was repeated on the same support-branch revision,
hardware, checkpoint, and launch configuration with only this loader change
removed. Workload: random 8,192 input tokens and 1,024 output tokens.

Concurrency Baseline total tok/s With fix total tok/s Delta Baseline median E2E With fix median E2E Delta
2 1,978.39 1,972.94 -0.28% 9,318.61 ms 9,325.65 ms +0.08%
4 3,631.30 3,625.18 -0.17% 10,146.22 ms 10,169.20 ms +0.23%
8 6,336.93 6,327.30 -0.15% 11,635.31 ms 11,647.49 ms +0.10%

Median ITL differed by only 0.01-0.02 ms across all three points. The observed
throughput and latency differences are below 0.3% and consistent with normal
run-to-run measurement noise; no material Flash performance regression was
observed.

Scope

This is a follow-up to the Quark mixed-precision support in #38546. It does not
replace or modify GLM-5.3-Flash's MoE/AITER layout, clamp, padding, or
model-specific loading logic. It completes the separate DeepSeek loader path
used by GlmMoeDsaForCausalLM.


CI States

Latest PR Test (Base): ✅ Run #39856212203
Latest PR Test (Extra): ❌ Run #39856211732
Latest PR Test (AMD ROCm 10): ❌ Run #39856212073

@ColinZ22 ColinZ22 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM, @HaiShaw @kkHuang-amd could we get the run-ci flag for this PR? Thanks

@1am9trash 1am9trash added amd run-ci CI: run the baseline test suite on this PR labels Sep 23, 2026
@clintg6

clintg6 commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@clintg6

clintg6 commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

@HaiShaw this is ready to be merged. Nvidia CI failing on known issue #17050 test_kimi_k3_b300.py — base-c-test-8-gpu-b300 / base-c-test-8-gpu-b300 (0)

@BowenBao BowenBao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @clintg6.

Could you please do a sanity check with h100 and mi300 to check that base fp8 flow is unaffected? Thank you!

@clintg6

clintg6 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

@BowenBao everything ran as expected with base z-ai/GLM-5.3.

@clintg6

clintg6 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author
mi355x_sglang_mtp_glm53_ttt_per_gpu-20261010-223956 @HaiShaw 20%+ TTT uplift and growing with concurrency at TP4.

@xiaofei-zheng

Copy link
Copy Markdown
Contributor

@clintg6 Are you still planning to push this PR forward? It has had no activity since Oct 12 and is still waiting on code-owner review.

@github-actions

This comment has been minimized.

@hubertlu-tw hubertlu-tw changed the title [AMD] Fix Quark block-FP8 loading in DeepSeek loader [Fix] [AMD] Fix Quark block-FP8 loading in DeepSeek loader Oct 22, 2026
@github-actions

This comment has been minimized.

@hubertlu-tw hubertlu-tw changed the title [Fix] [AMD] Fix Quark block-FP8 loading in DeepSeek loader [Fix] [AMD] Quark: Fix Quark block-FP8 loading in DeepSeek loader Oct 22, 2026
@hubertlu-tw hubertlu-tw changed the title [Fix] [AMD] Quark: Fix Quark block-FP8 loading in DeepSeek loader [Fix] [AMD] Quark: fix Quark block-FP8 loading in DeepSeek loader Oct 22, 2026
@hubertlu-tw
hubertlu-tw enabled auto-merge (squash) October 22, 2026 20:57
@hubertlu-tw
hubertlu-tw disabled auto-merge October 22, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd deepseek run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants