Repository navigation
[CompressedTensors] DeepSeek4 CT Quantization Support - #41276
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the DeepseekV4 model implementation by adding a packed_modules_mapping for fused layers and implementing a safe initialization for scale_fmt that defaults to 'ue8m0' when the quantization configuration is missing or not a dictionary. I have no feedback to provide.
dsikka
left a comment
There was a problem hiding this comment.
this pathway is likely going to continue seeing multiple updates in the next few weeks. Would be good to add some form of smoke test
6a81e43 to
33f36d4
Compare
1e6b8a1 to
f910a73
Compare
|
FYI I'm seeing a slight accuracy loss with the model, I've ruled out output_dtype as the cause in #41533, which makes me suspect that the cause is the quantization of the indexer/compressor wkv weights. Currently working on updating the checkpoint to skip this quantization, will post accuracy evaluations. |
f5fc438 to
322ca21
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
|
Maintainer-direction question from a downstream consumer — your design here is in substantive tension with the parallel The conflict:
What canada-quant ships: Our public artifact Where we'd like maintainer guidance:
We've also been tracking a separate but related kernel-dispatch issue at Notes on your specific changes that we've validated locally:
Happy to help with any of the following downstream:
Reproducibility links: canada-quant findings docs, our test-pass validation comment on #43655. Tagging @toffee-desuwa (author of |
|
Per @pasta-paul's tag — quick context from the #43655 author side: #43655's scope is strictly the load-time Both directions can coexist: #41276 sets the recommended recipe (driven by the accuracy work in #42001), #43655 handles loader robustness for artifacts already in the wild. @kylesayrs happy to align in whichever direction makes sense — including reshaping #43655 to a warning + fallback path if you'd prefer the loader actively discourage quantization here. Defer to your call as the LLM Compressor side maintainer. |
22f6da8 to
8b92719
Compare
8b92719 to
b52a3b8
Compare
b52a3b8 to
48c02b2
Compare
48c02b2 to
cef3734
Compare
|
@toffee-desuwa I'm reducing the scope of this PR for now to only focus on DSV4 support of standard quantized parameters |
|
Thanks for the ping @kylesayrs. Does the scope cut mean the wkv/wgate loading question is open again, or is the plan still to skip quantizing those layers at the checkpoint level? Happy to pick the loader side back up if that's useful. |
## Prerequisites ## ### Accelerate/ Transformers ### - [x] huggingface/accelerate#4083 - [x] huggingface/transformers#46749 - [x] huggingface/transformers#46849 - [x] huggingface/transformers#47018 - [x] huggingface/transformers#46902 - [x] huggingface/transformers#46965 ### Compressed Tensors ### - [x] vllm-project/compressed-tensors#731 - [x] vllm-project/compressed-tensors#730 - [x] vllm-project/compressed-tensors#728 - [x] vllm-project/compressed-tensors#752 - [x] vllm-project/compressed-tensors#750 - [x] vllm-project/compressed-tensors#759 ### LLM Compressor ### - [x] #2647 - [x] #2686 - [x] #2865 ### vLLM ### - [x] vllm-project/vllm#41276 ## Testing ## ``` ========== SAMPLE GENERATION ============== Hello my name is Inigo Montoya. You killed my father. Prepare to die<|end▁of▁sentence|> ========================================== ``` --------- Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
## Prerequisites ## ### Accelerate/ Transformers ### - [x] huggingface/accelerate#4083 - [x] huggingface/transformers#46749 - [x] huggingface/transformers#46849 - [x] huggingface/transformers#47018 - [x] huggingface/transformers#46902 - [x] huggingface/transformers#46965 ### Compressed Tensors ### - [x] vllm-project/compressed-tensors#731 - [x] vllm-project/compressed-tensors#730 - [x] vllm-project/compressed-tensors#728 - [x] vllm-project/compressed-tensors#752 - [x] vllm-project/compressed-tensors#750 - [x] vllm-project/compressed-tensors#759 ### LLM Compressor ### - [x] #2647 - [x] #2686 - [x] #2865 ### vLLM ### - [x] vllm-project/vllm#41276 ## Testing ## ``` ========== SAMPLE GENERATION ============== Hello my name is Inigo Montoya. You killed my father. Prepare to die<|end▁of▁sentence|> ========================================== ``` --------- Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
## Prerequisites ## ### Accelerate/ Transformers ### - [x] huggingface/accelerate#4083 - [x] huggingface/transformers#46749 - [x] huggingface/transformers#46849 - [x] huggingface/transformers#47018 - [x] huggingface/transformers#46902 - [x] huggingface/transformers#46965 ### Compressed Tensors ### - [x] vllm-project/compressed-tensors#731 - [x] vllm-project/compressed-tensors#730 - [x] vllm-project/compressed-tensors#728 - [x] vllm-project/compressed-tensors#752 - [x] vllm-project/compressed-tensors#750 - [x] vllm-project/compressed-tensors#759 ### LLM Compressor ### - [x] #2647 - [x] #2686 - [x] #2865 ### vLLM ### - [x] vllm-project/vllm#41276 ## Testing ## ``` ========== SAMPLE GENERATION ============== Hello my name is Inigo Montoya. You killed my father. Prepare to die<|end▁of▁sentence|> ========================================== ``` --------- Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> Signed-off-by: Roderick-Wu <roderickwu2003@gmail.com>
…1276) Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
DeepSeek-V4-Flash-NVFP4-FP8
Model Optimizations
This model was obtained by using the following branch with LLM Compressor: vllm-project/llm-compressor#2647
Deployment
vllm serve RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8 --tensor-parallel-size 4 --port 8089 --kv_cache_dtype="fp8"Accuracy Evaluation
deepseek-ai/DeepSeek-V4-Pro-Basedeepseek-ai/DeepSeek-V4-ProRedHatAI/DeepSeek-V4-Pro-NVFP4-FP8Performance Evaluation
8xB200
For more details on how this model was created and run in LLM Compressor, please contact Kyle Sayers on the vLLM Slack: https://communityinviter.com/apps/vllm-dev/join-vllm-developers-slack