Repository navigation
[Fix] deepseek_v2/glm4_moe_lite/nextn: Handle checkpoint KV-cache scales for MLA attention and NVFP4 draft models - #48716
Open
HollowMan6 wants to merge 4 commits into
Open
HollowMan6 wants to merge 4 commits into
HollowMan6 wants to merge 4 commits into
Conversation
HollowMan6
requested review from
Fridge003,
ch-wan,
fzyzcjy and
ispobock
as code owners
November 5, 2026 20:31
3 of 5 tasks
This comment has been minimized.
This comment has been minimized.
3 of 5 tasks
HollowMan6
force-pushed
the
fix-mla-kv-cache-scales
branch
from
November 6, 2026 11:30
5e65022 to
5b2d368
Compare
4 of 5 tasks
HollowMan6
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
HaiShaw,
OrangeRedeng and
b8zhong
as code owners
November 6, 2026 14:08
This comment has been minimized.
This comment has been minimized.
HollowMan6
force-pushed
the
fix-mla-kv-cache-scales
branch
from
November 11, 2026 20:34
55ad03d to
43abdef
Compare
…ttention MLA writes its latent KV cache without a per-tensor scale: the MLA pool's set_kv_buffer / set_mla_kv_buffer cast to FP8 at scale 1.0, and trtllm_mla quantizes with quant_scale_kv=1.0. The triton, flashmla and trtllm_mla decode kernels still dequantize that cache with attn_mqa's k_scale. The DeepSeek loader renamed ModelOpt scales (self_attn.k_proj.k_scale) onto attn_mqa, so a checkpoint whose scales are not 1.0 produced incoherent decode output on those backends, while compressed-tensors scales (self_attn.k_scale) were dropped with one "not found in params_dict" warning per tensor. Skip both layouts in the DeepSeek, LongCat-Flash and Dots3 loaders so the scale stays at the 1.0 the cache is written with, and warn once when a checkpoint scale other than 1.0 is ignored. Signed-off-by: Hollow Man <hollowman@opensuse.org>
… them for MLA
ModelOpt names the MLA KV-cache scales self_attn.k_proj.k_scale and
self_attn.v_proj.v_scale. Glm4MoeLiteForCausalLM.load_weights mangled them
before any KV-scale handling could see them:
- stacked_params_mapping has q/k/v_proj -> qkv_proj entries, but the MLA
attention has no qkv_proj, so the entries only rewrite these names
(k_proj.k_scale -> qkqkv_proj.k_scale, v_proj.v_scale -> qkv_proj.v_scale);
- the KV-scale fallback then applied name.replace("_proj", "attn_mqa"), the
rule deepseek_v2 used before sgl-project#7164.
The resulting names (qkqkvattn_mqa.k_scale, qkvattn_mqa.v_scale) match no
parameter, so the scales were dropped without a warning. The MTP draft
(Glm4MoeLiteForCausalLMNextN) uses the same loader.
Drop the qkv_proj entries, as the DeepSeek and LongCat-Flash MLA loaders do,
so the checkpoint names stay intact, and pass them to
skip_mla_kv_cache_scale() like the DeepSeek loader: MLA writes its latent KV
cache unscaled, so the scales are skipped (attn_mqa keeps 1.0) and one
warning is logged when a checkpoint scale other than 1.0 is ignored.
Signed-off-by: Hollow Man <hollowman@opensuse.org>
ModelOpt NVFP4 checkpoints store the NextN layer in BF16, so DeepseekModelNextN builds it with quant_config=None (sgl-project#7376). That also drops the KV-cache method of the draft's RadixAttention: with an FP8 KV cache the draft attention has no k_scale/v_scale, unlike the target's attention and the same draft of a ModelOpt FP8 checkpoint. Keep building the NextN layer unquantized, but add the checkpoint's KV-cache method to its attention layers (add_kv_cache_quant_methods). The draft then handles KV-cache scales like its target; the MLA loaders skip checkpoint scales, so the MLA drafts keep 1.0. GLM-5 and GLM-5.x-Flash NextN inherit the change; the GLM-4.7-Flash and GLM-OCR NextN drafts copy the override and get the same change. Signed-off-by: Hollow Man <hollowman@opensuse.org>
… drafts For ModelOpt checkpoints that keep the draft in BF16 (serialized NVFP4, and MIXED_PRECISION without mtp.* layers), Qwen3_5ForCausalLMMTP and Qwen4ExpForCausalLMMTP build the MTP layer with quant_config=None, and Eagle3DeepseekV2ForCausalLM does the same for modelopt_fp4. That also drops the KV-cache method of the draft's RadixAttention: with an FP8 KV cache the draft attention has no k_scale/v_scale, unlike the target's attention and the same draft of a ModelOpt FP8 checkpoint. Keep building the draft unquantized, but add the checkpoint's KV-cache method to its attention layers with add_kv_cache_quant_methods, as the DeepSeek and GLM NextN drafts do. The Eagle3 draft uses MLA, whose loaders skip checkpoint KV-cache scales, so it keeps 1.0 like its target. Signed-off-by: Hollow Man <hollowman@opensuse.org>
HollowMan6
force-pushed
the
fix-mla-kv-cache-scales
branch
from
November 13, 2026 13:56
43abdef to
b7142a9
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Fixes #48707.
Fixes #48706.
Fixes #48725.
This PR also includes the fix for #48725 (formerly #48885), which builds on the MLA change: the last two commits give the BF16 draft layers of NVFP4 checkpoints the KV-cache method of the checkpoint.
MLA writes its latent FP8 KV cache without a per-tensor scale: prefill stores it through
attn_mha,MLATokenToKVPoolcasts with a plain.to(fp8), and trtllm_mla quantizes withquant_scale_kv=1.0. The triton, flashmla and trtllm_mla decode kernels still dequantize that cache withattn_mqa'sk_scale.DeepseekV2WeightLoaderMixinrenames ModelOpt KV-cache scales (self_attn.k_proj.k_scale,self_attn.v_proj.v_scale) ontoattn_mqa, so a checkpoint whose scales are not 1.0 decodes incoherently on those backends. compressed-tensors names (self_attn.k_scale,self_attn.v_scale) are dropped with onenot found in params_dictwarning per tensor.NVFP4 draft models (NextN, MTP, Eagle3)
For
modelopt_fp4checkpoints,DeepseekModelNextNbuilds the NextN layer withquant_config=None(#7376), because NVIDIA's NVFP4 checkpoints keep that layer in BF16. This also removes the KV-cache method from the draft'sRadixAttention. With--kv-cache-dtype fp8_e4m3, the draft attention then has nok_scale/v_scale. The target's attention layers have them, and so does the draft of a ModelOpt FP8 checkpoint. The GLM-4.7-Flash and GLM-OCR NextN drafts copy the override, and the GLM-5 and GLM-5.x-Flash drafts inherit it. The Qwen3.5 and Qwen4Exp MTP drafts build their MTP layer unquantized formodelopt_fp4andMIXED_PRECISIONcheckpoints, and the Kimi-K2.5 Eagle3 draft does so formodelopt_fp4, with the same effect.Modifications
skip_mla_kv_cache_scale()todeepseek_common/deepseek_weight_loader.py. It recognizes both layouts (self_attn[.N].{k_proj.k_scale, v_proj.v_scale, k_scale, v_scale}), skips the tensor, and logs one warning when a checkpoint scale other than 1.0 is ignored.DeepseekV2WeightLoaderMixin,longcat_flash.py,longcat_flash_nextn.pyanddots3_common/modeling.py, soattn_mqakeeps the 1.0 the cache is written with.test/registered/unit/models/test_deepseek_mla_kv_scale_loading.py(CPU suite): 3 tests, failing on main (failures=2, errors=2) and passing with this change. The existing quark loader test still passes.glm4_moe_lite.py(GLM-4.7-Flash and its NextN draft): drop the threeq/k/v_proj -> qkv_projentries fromstacked_params_mapping, which the MLA attention has no parameter for and which turnedk_proj.k_scaleintoqkqkv_proj.k_scale([Bug] GLM-4.7-Flash ignores ModelOpt FP8 KV-cache scales (self_attn.k_proj.k_scale / v_proj.v_scale) #48706), and pass KV-cache scale names to the same helper. Addtest/registered/unit/models/test_glm4_moe_lite_weight_loading.py(CPU suite) for the target model and the NextN draft. Every other tensor name of the BF16, ModelOpt FP8 and NVFP4 GLM-4.7-Flash checkpoints still maps to the same parameter as on main.Not changed:
deepseek_v4.py(different attention stack), andkimi_vl/kimi_linear/glm5_next, which already drop both layouts. Actually using the calibration would need scaled latent writes and a matching dequant in every MLA backend (the direction of #34765); this change keeps the cache write and its dequant consistent.NVFP4 draft models (NextN, MTP, Eagle3):
layers/quantization/kv_cache.py: a new helper,add_kv_cache_quant_methods(module, quant_config, prefix). For everyRadixAttentionin a module built withquant_config=None, it sets the KV-cache method ofquant_configand creates the scale parameters.deepseek_nextn.py,glm4_moe_lite_nextn.py,glm_ocr_nextn.py: the NextN layer is still built unquantized, and its attention layers then get the checkpoint's KV-cache method through the helper.k_scale/v_scale, like its target.qwen3_5_mtp.py: a new_mtp_kv_cache_quant_configreturns the checkpoint's quant config when_mtp_quant_configbuilds the MTP module of amodelopt_fp4ormodelopt_mixedcheckpoint unquantized;Qwen3_5ForCausalLMMTPpasses it to the helper after building the MTP module.qwen4_exp_mtp.pydoes the same forQwen4ExpForCausalLMMTP.kimi_k25_eagle3.py: formodelopt_fp4, the draft is still built unquantized, and its attention gets the checkpoint's KV-cache method through the helper. It uses MLA, so with [Fix] deepseek_v2/glm4_moe_lite/nextn: Handle checkpoint KV-cache scales for MLA attention and NVFP4 draft models #48716 it keeps 1.0, like its target._mtp_quant_configare not changed.Accuracy Tests
DeepSeek-V2-Lite-Chat BF16 weights plus the 54 KV-cache scales of INC4AI/DeepSeek-V2-Lite-Chat-BF16-FP8-STATIC-FP8-KV-TEST-ONLY (k 0.058-0.127, v 0.004-0.066), stored under either naming with a KV-cache-only compressed-tensors config, FP8 KV cache, one H20. Correct answers to 11 greedy prompts (48 new tokens):
The first token (prefill) was identical in all runs; only decode differed. trtllm_mla and aiter were not run; they take the same path in the code.
NVFP4 draft models (NextN, MTP, Eagle3):
Setup: 1x H20-3e (SM90), CUDA 13.0, dummy weights,
--kv-cache-dtype fp8_e4m3. "Base" is main91fd6756with the two commits of #48716; "this PR" adds the last commit.test/registered/unit/models/test_deepseek_nextn_kv_cache_quant.py(2 cases): it fails on base and passes with this PR.DeepseekV3ForCausalLMNextNModelOptFp8KVCacheMethodModelOptFp8KVCacheMethodGlm4MoeLiteForCausalLMNextNModelOptFp8KVCacheMethodModelOptFp8KVCacheMethodGlmMoeDsaForCausalLMNextN(GLM-5)ModelOptFp8KVCacheMethodModelOptFp8KVCacheMethodGlm5NextForConditionalGenerationNextN(GLM-5.3-Flash)ModelOptFp8KVCacheMethodModelOptFp8KVCacheMethodGlmOcrForConditionalGenerationNextNModelOptFp8KVCacheMethodModelOptFp8KVCacheMethodtest/registered/unit/models/test_mtp_draft_kv_cache_quant.py(5 cases): 4 fail before the last commit and all pass with it. It checks that the BF16 MTP drafts of NVFP4 andMIXED_PRECISIONconfigs keep unquantized weights and getModelOptFp8KVCacheMethod, that the Qwen3.5 MTP layer's ModelOpt scales load through the draft's loader, and that a quantized FP8 draft does not change.codecho/Qwen3.5-0.8B-text-only,Qwen/Qwen3.8-Flash-Nextandlightseekorg/kimi-k2.5-eagle3-mla, withmtp.layers.0.self_attn.{k_proj.k_scale, v_proj.v_scale}= 0.0625 / 0.03125 in the checkpoint:Qwen3_5ForCausalLMMTPModelOptFp8KVCacheMethod, 0.0625 / 0.03125ModelOptFp8KVCacheMethod, 0.0625 / 0.03125Qwen4ExpForCausalLMMTPModelOptFp8KVCacheMethod, 0.0625 / 0.03125ModelOptFp8KVCacheMethod, 0.0625 / 0.03125Eagle3DeepseekV2ForCausalLMNotImplementedErrorfor FP8 weights, before and after)ModelOptFp8KVCacheMethod, 1.0 / 1.0 (MLA)quant_method=Noneand no scales for the NVFP4 draft on base. With this PR it givesModelOptFp8KVCacheMethodwithk_scale = v_scale = 1.0, the same as the FP8 draft.test_compressed_tensors_kv_cache,test_deepseek_mla_kv_scale_loading,test_deepseek_nextn_mm_embed,test_deepseek_pp_mtp_embedding,test_deepseek_quark_block_fp8_loading,test_glm4_moe_lite_weight_loading,test_glm5_next_modelopt,test_glm_nextn_moe_ptpc,test_kv_calibration_loader_layout,test_modelopt_nvfp4,test_qwen3_5_mtp_quant_config, and for the last commit alsotest_nemotron_h_mtp,test_qwen3_5_modelopt_fp4,test_qwen3_5_packed_weight_loader,test_qwen3_5_pipeline_parallel,test_qwen4_exp_decode_commandtest_qwen4_exp_ple_table. pre-commit passes on all commits.Speed Tests and Profiling
No kernel or forward changes; only which checkpoint tensors the loader keeps.
Checklist
Parts of this change were drafted with AI assistance and reviewed and tested by the author.
CI States
Latest PR Test (Base): ❌ Run #43438058974
Latest PR Test (Extra): ❌ Run #43438058449
Latest PR Test (AMD ROCm 10): ❌ Run #43438058520
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: