Skip to content

[Fix] deepseek_v2/glm4_moe_lite/nextn: Handle checkpoint KV-cache scales for MLA attention and NVFP4 draft models - #48716

Open
HollowMan6 wants to merge 4 commits into
sgl-project:mainfrom
HollowMan6:fix-mla-kv-cache-scales
Open

HollowMan6 wants to merge 4 commits into
sgl-project:mainfrom
HollowMan6:fix-mla-kv-cache-scales

Conversation

@HollowMan6

@HollowMan6 HollowMan6 commented Nov 5, 2026 •

Copy link
Copy Markdown

Motivation

Fixes #48707.
Fixes #48706.
Fixes #48725.

This PR also includes the fix for #48725 (formerly #48885), which builds on the MLA change: the last two commits give the BF16 draft layers of NVFP4 checkpoints the KV-cache method of the checkpoint.

MLA writes its latent FP8 KV cache without a per-tensor scale: prefill stores it through attn_mha, MLATokenToKVPool casts with a plain .to(fp8), and trtllm_mla quantizes with quant_scale_kv=1.0. The triton, flashmla and trtllm_mla decode kernels still dequantize that cache with attn_mqa's k_scale. DeepseekV2WeightLoaderMixin renames ModelOpt KV-cache scales (self_attn.k_proj.k_scale, self_attn.v_proj.v_scale) onto attn_mqa, so a checkpoint whose scales are not 1.0 decodes incoherently on those backends. compressed-tensors names (self_attn.k_scale, self_attn.v_scale) are dropped with one not found in params_dict warning per tensor.

NVFP4 draft models (NextN, MTP, Eagle3)

For modelopt_fp4 checkpoints, DeepseekModelNextN builds the NextN layer with quant_config=None (#7376), because NVIDIA's NVFP4 checkpoints keep that layer in BF16. This also removes the KV-cache method from the draft's RadixAttention. With --kv-cache-dtype fp8_e4m3, the draft attention then has no k_scale/v_scale. The target's attention layers have them, and so does the draft of a ModelOpt FP8 checkpoint. The GLM-4.7-Flash and GLM-OCR NextN drafts copy the override, and the GLM-5 and GLM-5.x-Flash drafts inherit it. The Qwen3.5 and Qwen4Exp MTP drafts build their MTP layer unquantized for modelopt_fp4 and MIXED_PRECISION checkpoints, and the Kimi-K2.5 Eagle3 draft does so for modelopt_fp4, with the same effect.

Modifications

  • Add skip_mla_kv_cache_scale() to deepseek_common/deepseek_weight_loader.py. It recognizes both layouts (self_attn[.N].{k_proj.k_scale, v_proj.v_scale, k_scale, v_scale}), skips the tensor, and logs one warning when a checkpoint scale other than 1.0 is ignored.
  • Use it instead of the ModelOpt rename in DeepseekV2WeightLoaderMixin, longcat_flash.py, longcat_flash_nextn.py and dots3_common/modeling.py, so attn_mqa keeps the 1.0 the cache is written with.
  • Add test/registered/unit/models/test_deepseek_mla_kv_scale_loading.py (CPU suite): 3 tests, failing on main (failures=2, errors=2) and passing with this change. The existing quark loader test still passes.
  • glm4_moe_lite.py (GLM-4.7-Flash and its NextN draft): drop the three q/k/v_proj -> qkv_proj entries from stacked_params_mapping, which the MLA attention has no parameter for and which turned k_proj.k_scale into qkqkv_proj.k_scale ([Bug] GLM-4.7-Flash ignores ModelOpt FP8 KV-cache scales (self_attn.k_proj.k_scale / v_proj.v_scale) #48706), and pass KV-cache scale names to the same helper. Add test/registered/unit/models/test_glm4_moe_lite_weight_loading.py (CPU suite) for the target model and the NextN draft. Every other tensor name of the BF16, ModelOpt FP8 and NVFP4 GLM-4.7-Flash checkpoints still maps to the same parameter as on main.

Not changed: deepseek_v4.py (different attention stack), and kimi_vl / kimi_linear / glm5_next, which already drop both layouts. Actually using the calibration would need scaled latent writes and a matching dequant in every MLA backend (the direction of #34765); this change keeps the cache write and its dequant consistent.

NVFP4 draft models (NextN, MTP, Eagle3):

  • layers/quantization/kv_cache.py: a new helper, add_kv_cache_quant_methods(module, quant_config, prefix). For every RadixAttention in a module built with quant_config=None, it sets the KV-cache method of quant_config and creates the scale parameters.
  • deepseek_nextn.py, glm4_moe_lite_nextn.py, glm_ocr_nextn.py: the NextN layer is still built unquantized, and its attention layers then get the checkpoint's KV-cache method through the helper.
  • With the MLA change above, the MLA drafts (DeepSeek, GLM-4.7-Flash, GLM-5, GLM-5.x-Flash) still skip the checkpoint's KV-cache scales and use 1.0, like their targets. The GLM-OCR draft uses standard attention and now has k_scale/v_scale, like its target.
  • qwen3_5_mtp.py: a new _mtp_kv_cache_quant_config returns the checkpoint's quant config when _mtp_quant_config builds the MTP module of a modelopt_fp4 or modelopt_mixed checkpoint unquantized; Qwen3_5ForCausalLMMTP passes it to the helper after building the MTP module. qwen4_exp_mtp.py does the same for Qwen4ExpForCausalLMMTP.
  • kimi_k25_eagle3.py: for modelopt_fp4, the draft is still built unquantized, and its attention gets the checkpoint's KV-cache method through the helper. It uses MLA, so with [Fix] deepseek_v2/glm4_moe_lite/nextn: Handle checkpoint KV-cache scales for MLA attention and NVFP4 draft models #48716 it keeps 1.0, like its target.
  • The drafts' other layers keep the quant methods they had. The Quark and NPU cases of _mtp_quant_config are not changed.

Accuracy Tests

DeepSeek-V2-Lite-Chat BF16 weights plus the 54 KV-cache scales of INC4AI/DeepSeek-V2-Lite-Chat-BF16-FP8-STATIC-FP8-KV-TEST-ONLY (k 0.058-0.127, v 0.004-0.066), stored under either naming with a KV-cache-only compressed-tensors config, FP8 KV cache, one H20. Correct answers to 11 greedy prompts (48 new tokens):

backend BF16 KV main, compressed-tensors names main, ModelOpt names this PR, either naming
fa3 8/11 8/11 8/11 8/11
triton 8/11 8/11 2/11 8/11
flashmla 8/11 8/11 2/11 8/11

The first token (prefill) was identical in all runs; only decode differed. trtllm_mla and aiter were not run; they take the same path in the code.

NVFP4 draft models (NextN, MTP, Eagle3):

Setup: 1x H20-3e (SM90), CUDA 13.0, dummy weights, --kv-cache-dtype fp8_e4m3. "Base" is main 91fd6756 with the two commits of #48716; "this PR" adds the last commit.

  • New test test/registered/unit/models/test_deepseek_nextn_kv_cache_quant.py (2 cases): it fails on base and passes with this PR.
  • KV-cache method of the NextN draft's attention, ModelOpt FP8 and NVFP4 configs:
draft class FP8, base and this PR NVFP4, base NVFP4, this PR
DeepseekV3ForCausalLMNextN ModelOptFp8KVCacheMethod none ModelOptFp8KVCacheMethod
Glm4MoeLiteForCausalLMNextN ModelOptFp8KVCacheMethod none ModelOptFp8KVCacheMethod
GlmMoeDsaForCausalLMNextN (GLM-5) ModelOptFp8KVCacheMethod none ModelOptFp8KVCacheMethod
Glm5NextForConditionalGenerationNextN (GLM-5.3-Flash) ModelOptFp8KVCacheMethod none ModelOptFp8KVCacheMethod
GlmOcrForConditionalGenerationNextN ModelOptFp8KVCacheMethod none ModelOptFp8KVCacheMethod
  • New test test/registered/unit/models/test_mtp_draft_kv_cache_quant.py (5 cases): 4 fail before the last commit and all pass with it. It checks that the BF16 MTP drafts of NVFP4 and MIXED_PRECISION configs keep unquantized weights and get ModelOptFp8KVCacheMethod, that the Qwen3.5 MTP layer's ModelOpt scales load through the draft's loader, and that a quantized FP8 draft does not change.
  • MTP and Eagle3 drafts, built from the configs of codecho/Qwen3.5-0.8B-text-only, Qwen/Qwen3.8-Flash-Next and lightseekorg/kimi-k2.5-eagle3-mla, with mtp.layers.0.self_attn.{k_proj.k_scale, v_proj.v_scale} = 0.0625 / 0.03125 in the checkpoint:
draft class FP8, before and after NVFP4, before NVFP4, after
Qwen3_5ForCausalLMMTP ModelOptFp8KVCacheMethod, 0.0625 / 0.03125 none, no scales ModelOptFp8KVCacheMethod, 0.0625 / 0.03125
Qwen4ExpForCausalLMMTP ModelOptFp8KVCacheMethod, 0.0625 / 0.03125 none, no scales ModelOptFp8KVCacheMethod, 0.0625 / 0.03125
Eagle3DeepseekV2ForCausalLM not run (the draft's post-load step raises NotImplementedError for FP8 weights, before and after) none, no scales ModelOptFp8KVCacheMethod, 1.0 / 1.0 (MLA)
  • The repro in [Bug] NextN/MTP drafts of ModelOpt NVFP4 checkpoints get no KV-cache scales, so the draft's FP8 KV cache runs at 1.0 #48725 (DeepSeek-V3 config, 4 layers) gives quant_method=None and no scales for the NVFP4 draft on base. With this PR it gives ModelOptFp8KVCacheMethod with k_scale = v_scale = 1.0, the same as the FP8 draft.
  • These related unit tests pass on base and with this PR: test_compressed_tensors_kv_cache, test_deepseek_mla_kv_scale_loading, test_deepseek_nextn_mm_embed, test_deepseek_pp_mtp_embedding, test_deepseek_quark_block_fp8_loading, test_glm4_moe_lite_weight_loading, test_glm5_next_modelopt, test_glm_nextn_moe_ptpc, test_kv_calibration_loader_layout, test_modelopt_nvfp4, test_qwen3_5_mtp_quant_config, and for the last commit also test_nemotron_h_mtp, test_qwen3_5_modelopt_fp4, test_qwen3_5_packed_weight_loader, test_qwen3_5_pipeline_parallel, test_qwen4_exp_decode_comm and test_qwen4_exp_ple_table. pre-commit passes on all commits.

Speed Tests and Profiling

No kernel or forward changes; only which checkpoint tensors the loader keeps.

Checklist

Parts of this change were drafted with AI assistance and reviewed and tested by the author.


CI States

Latest PR Test (Base): ❌ Run #43438058974
Latest PR Test (Extra): ❌ Run #43438058449
Latest PR Test (AMD ROCm 10): ❌ Run #43438058520

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@HollowMan6 HollowMan6 changed the title [Fix] deepseek_v2: stop loading checkpoint KV-cache scales into MLA attention [Fix] deepseek_v2, glm4_moe_lite: stop loading checkpoint KV-cache scales into MLA attention Nov 5, 2026
@github-actions

This comment has been minimized.

@HollowMan6 HollowMan6 changed the title [Fix] deepseek_v2, glm4_moe_lite: stop loading checkpoint KV-cache scales into MLA attention [Fix] deepseek_v2/glm4_moe_lite: stop loading checkpoint KV-cache scales into MLA attention Nov 5, 2026
@HollowMan6
HollowMan6 force-pushed the fix-mla-kv-cache-scales branch from 5e65022 to 5b2d368 Compare November 6, 2026 11:30
@HollowMan6 HollowMan6 changed the title [Fix] deepseek_v2/glm4_moe_lite: stop loading checkpoint KV-cache scales into MLA attention [Fix] MLA attention and NVFP4 draft models: handle checkpoint KV-cache scales Nov 6, 2026
@github-actions github-actions Bot added the quant LLM Quantization label Nov 6, 2026
@github-actions

This comment has been minimized.

@HollowMan6 HollowMan6 changed the title [Fix] MLA attention and NVFP4 draft models: handle checkpoint KV-cache scales [Fix] deepseek_v2/glm4_moe_lite/nextn: Handle checkpoint KV-cache scales for MLA attention and NVFP4 draft models Nov 6, 2026
@HollowMan6
HollowMan6 force-pushed the fix-mla-kv-cache-scales branch from 55ad03d to 43abdef Compare November 11, 2026 20:34
…ttention

MLA writes its latent KV cache without a per-tensor scale: the MLA pool's
set_kv_buffer / set_mla_kv_buffer cast to FP8 at scale 1.0, and trtllm_mla
quantizes with quant_scale_kv=1.0. The triton, flashmla and trtllm_mla decode
kernels still dequantize that cache with attn_mqa's k_scale. The DeepSeek
loader renamed ModelOpt scales (self_attn.k_proj.k_scale) onto attn_mqa, so a
checkpoint whose scales are not 1.0 produced incoherent decode output on those
backends, while compressed-tensors scales (self_attn.k_scale) were dropped
with one "not found in params_dict" warning per tensor.

Skip both layouts in the DeepSeek, LongCat-Flash and Dots3 loaders so the
scale stays at the 1.0 the cache is written with, and warn once when a
checkpoint scale other than 1.0 is ignored.

Signed-off-by: Hollow Man <hollowman@opensuse.org>
… them for MLA

ModelOpt names the MLA KV-cache scales self_attn.k_proj.k_scale and
self_attn.v_proj.v_scale. Glm4MoeLiteForCausalLM.load_weights mangled them
before any KV-scale handling could see them:

- stacked_params_mapping has q/k/v_proj -> qkv_proj entries, but the MLA
  attention has no qkv_proj, so the entries only rewrite these names
  (k_proj.k_scale -> qkqkv_proj.k_scale, v_proj.v_scale -> qkv_proj.v_scale);
- the KV-scale fallback then applied name.replace("_proj", "attn_mqa"), the
  rule deepseek_v2 used before sgl-project#7164.

The resulting names (qkqkvattn_mqa.k_scale, qkvattn_mqa.v_scale) match no
parameter, so the scales were dropped without a warning. The MTP draft
(Glm4MoeLiteForCausalLMNextN) uses the same loader.

Drop the qkv_proj entries, as the DeepSeek and LongCat-Flash MLA loaders do,
so the checkpoint names stay intact, and pass them to
skip_mla_kv_cache_scale() like the DeepSeek loader: MLA writes its latent KV
cache unscaled, so the scales are skipped (attn_mqa keeps 1.0) and one
warning is logged when a checkpoint scale other than 1.0 is ignored.

Signed-off-by: Hollow Man <hollowman@opensuse.org>
ModelOpt NVFP4 checkpoints store the NextN layer in BF16, so
DeepseekModelNextN builds it with quant_config=None (sgl-project#7376). That also
drops the KV-cache method of the draft's RadixAttention: with an FP8 KV
cache the draft attention has no k_scale/v_scale, unlike the target's
attention and the same draft of a ModelOpt FP8 checkpoint.

Keep building the NextN layer unquantized, but add the checkpoint's
KV-cache method to its attention layers (add_kv_cache_quant_methods). The
draft then handles KV-cache scales like its target; the MLA loaders skip
checkpoint scales, so the MLA drafts keep 1.0. GLM-5 and GLM-5.x-Flash
NextN inherit the change; the GLM-4.7-Flash and GLM-OCR NextN drafts copy
the override and get the same change.

Signed-off-by: Hollow Man <hollowman@opensuse.org>
… drafts

For ModelOpt checkpoints that keep the draft in BF16 (serialized NVFP4,
and MIXED_PRECISION without mtp.* layers), Qwen3_5ForCausalLMMTP and
Qwen4ExpForCausalLMMTP build the MTP layer with quant_config=None, and
Eagle3DeepseekV2ForCausalLM does the same for modelopt_fp4. That also
drops the KV-cache method of the draft's RadixAttention: with an FP8 KV
cache the draft attention has no k_scale/v_scale, unlike the target's
attention and the same draft of a ModelOpt FP8 checkpoint.

Keep building the draft unquantized, but add the checkpoint's KV-cache
method to its attention layers with add_kv_cache_quant_methods, as the
DeepSeek and GLM NextN drafts do. The Eagle3 draft uses MLA, whose
loaders skip checkpoint KV-cache scales, so it keeps 1.0 like its
target.

Signed-off-by: Hollow Man <hollowman@opensuse.org>
@HollowMan6
HollowMan6 force-pushed the fix-mla-kv-cache-scales branch from 43abdef to b7142a9 Compare November 13, 2026 13:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek quant LLM Quantization

Projects

None yet

1 participant