Skip to content

[CompressedTensors] DeepSeek4 CT Quantization Support - #41276

Merged
vllm-bot merged 1 commit into
vllm-project:mainfrom
neuralmagic:kylesayrs/deepseek-ct
Jul 24, 2026
Merged

vllm-bot merged 1 commit into
vllm-project:mainfrom
neuralmagic:kylesayrs/deepseek-ct

Conversation

@kylesayrs

@kylesayrs kylesayrs commented Apr 29, 2026 •

Copy link
Copy Markdown
Contributor

DeepSeek-V4-Flash-NVFP4-FP8

Model Optimizations

This model was obtained by using the following branch with LLM Compressor: vllm-project/llm-compressor#2647

Deployment

vllm serve RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8 --tensor-parallel-size 4 --port 8089 --kv_cache_dtype="fp8"

Accuracy Evaluation

Benchmark deepseek-ai/DeepSeek-V4-Pro-Base deepseek-ai/DeepSeek-V4-Pro RedHatAI/DeepSeek-V4-Pro-NVFP4-FP8
GPQA 90.1 0.93 (330/792 samples)
GSM8K 91.1 92.6 91.0

Performance Evaluation

8xB200

Avg prompt throughput: 155.5 tokens/s, Avg generation throughput: 3258.4 tokens/s, Running: 102 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.1%, Prefix cache hit rate: 2.4%

For more details on how this model was created and run in LLM Compressor, please contact Kyle Sayers on the vLLM Slack: https://communityinviter.com/apps/vllm-dev/join-vllm-developers-slack

@kylesayrs
kylesayrs marked this pull request as draft April 29, 2026 19:14

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the deepseek Related to DeepSeek models label Apr 29, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the DeepseekV4 model implementation by adding a packed_modules_mapping for fused layers and implementing a safe initialization for scale_fmt that defaults to 'ue8m0' when the quantization configuration is missing or not a dictionary. I have no feedback to provide.

@dsikka dsikka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this pathway is likely going to continue seeing multiple updates in the next few weeks. Would be good to add some form of smoke test

@kylesayrs

Copy link
Copy Markdown
Contributor Author

FYI I'm seeing a slight accuracy loss with the model, I've ruled out output_dtype as the cause in #41533, which makes me suspect that the cause is the quantization of the indexer/compressor wkv weights. Currently working on updating the checkpoint to skip this quantization, will post accuracy evaluations.

@mergify

mergify Bot commented May 23, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @kylesayrs.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label May 23, 2026
@pasta-paul

Copy link
Copy Markdown

Maintainer-direction question from a downstream consumer — your design here is in substantive tension with the parallel #43655 (supersedes #43515), and we'd like to know which way to align our calibration recipe.

The conflict:

Aspect This PR (#41276 — your direction) #43655 direction
compressor.fused_wkv_wgate quantized Raise NotImplementedError (line 230-234) Plumb quant_config in so it works
indexer.weights_proj quantized Raise NotImplementedError (line 1146-1147) Plumb quant_config in so it works
Ignored layers Append weights_proj, fused_wkv_wgate to ignored_layers (line 146) (no equivalent — those layers receive the model-level quant_config)
Rationale "accuracy concerns" (citing #42001) "fixes the load-time KeyError for artifacts that quantize these layers"

What canada-quant ships:

Our public artifact canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP (and the sibling NVFP4-FP8-MTP) does include FP8 block scales for compressor.fused_wkv_wgate and indexer.weights_proj — they were produced by llmcompressor.model_free_ptq running over the full DSv4 attention path. Bench numbers on these artifacts (GSM8K 93.71%, MMLU 86.88%, HumanEval 84.76%, AIME-2024 c=1=24/30 thinking-mode on a properly built venv) don't show the accuracy regression that #42001 anticipates, but we've only measured a subset of the eval matrix you'd care about for the formal upstream contract.

Where we'd like maintainer guidance:

  1. Is the answer "don't quantize these layers"? If yes, we should rebuild our calibration recipe to add weights_proj and fused_wkv_wgate to llm-compressor's ignore list and re-publish. (We'd take that hit — a ~10h H200 re-calibration per artifact.) This would make our artifacts compatible with your PR design forward.
  2. Or "these can be quantized but the loader should handle either case"? Then [Bugfix][DSV4] Plumb quant_config + route compressor GEMM through quant-aware dispatch #43655's plumbing is correct and this PR's NotImplementedError should soften to a logged warning + fall-back-to-unquantized path. We've test-passed [Bugfix][DSV4] Plumb quant_config + route compressor GEMM through quant-aware dispatch #43655's own 5-test suite locally — it's a clean fix on the load-time side, but doesn't address your accuracy concern.
  3. Or accept both — quantization permitted when configured, ignored layer when not? That's the most flexible path. Operators producing artifacts via llmcompressor.model_free_ptq would just declare these layers ignored in quantization_config themselves; the loader doesn't need new defensive code.

We've also been tracking a separate but related kernel-dispatch issue at #43564 (filed by us) where, even after quant_config is plumbed via #43655, the FP8 block scheme is routed to a per-channel kernel due to naming ambiguity at runtime. Independent scope from this PR but in the same DSv4-quant-loadability story.

Notes on your specific changes that we've validated locally:

  • ✅ The wo_scale_name getattr fallback (lines 286-296 + line 349) — we carry the equivalent patch (scripts/patch_wo_a_bf16_path.sh) and confirm it's necessary on every non-RedHat DSv4 artifact we've seen. LGTM.
  • ✅ The scale_fmt default-to-ue8m0 (lines 1010-1016) — same direction as #41791 (Dnoob). We've independently confirmed both on SM 12.0 RTX PRO 6000.
  • ❓ The ignored_layers += ["weights_proj", "fused_wkv_wgate"] (line 146) — see questions above.
  • ❓ The two NotImplementedError raises — see questions above.

Happy to help with any of the following downstream:

  • Rebuild a no-quantize-attn-compressor calibration recipe + ship as -v2 to validate the accuracy concern in [Core] [Quantization] Pass out_dtype to linear methods #42001 empirically
  • Provide bench JSONs from our current artifact at any granularity you want
  • Test this PR's branch against our artifacts on RTX PRO 6000 once the load-time path is unblocked

Reproducibility links: canada-quant findings docs, our test-pass validation comment on #43655.

Tagging @toffee-desuwa (author of #43655) since the design alignment question affects their PR too.

@toffee-desuwa

Copy link
Copy Markdown

Per @pasta-paul's tag — quick context from the #43655 author side:

#43655's scope is strictly the load-time KeyError from #43512. It doesn't recommend a quantization recipe for these layers; it just ensures that if an upstream artifact already quantizes them (as canada-quant's W4A16-FP8-MTP does), the loader doesn't crash.

Both directions can coexist: #41276 sets the recommended recipe (driven by the accuracy work in #42001), #43655 handles loader robustness for artifacts already in the wild.

@kylesayrs happy to align in whichever direction makes sense — including reshaping #43655 to a warning + fallback path if you'd prefer the loader actively discourage quantization here. Defer to your call as the LLM Compressor side maintainer.

@kylesayrs
kylesayrs force-pushed the kylesayrs/deepseek-ct branch from 22f6da8 to 8b92719 Compare June 27, 2026 19:54
@mergify mergify Bot removed the needs-rebase label Jun 27, 2026
@kylesayrs
kylesayrs force-pushed the kylesayrs/deepseek-ct branch from 8b92719 to b52a3b8 Compare June 29, 2026 19:05
@kylesayrs kylesayrs changed the title [WIP] [DSV4] Quantization Support [CompressedTensors] DeepSeek3.2/4 Quantization Support Jun 29, 2026
@kylesayrs
kylesayrs force-pushed the kylesayrs/deepseek-ct branch from b52a3b8 to 48c02b2 Compare July 23, 2026 00:00
@kylesayrs kylesayrs changed the title [CompressedTensors] DeepSeek3.2/4 Quantization Support [CompressedTensors] DeepSeek4 CT Quantization Support Jul 23, 2026
@mergify mergify Bot added the quantization label Jul 23, 2026
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
@kylesayrs
kylesayrs force-pushed the kylesayrs/deepseek-ct branch from 48c02b2 to cef3734 Compare July 23, 2026 18:26
@kylesayrs

Copy link
Copy Markdown
Contributor Author

@toffee-desuwa I'm reducing the scope of this PR for now to only focus on DSV4 support of standard quantized parameters

@kylesayrs
kylesayrs marked this pull request as ready for review July 23, 2026 20:06
@kylesayrs
kylesayrs requested a review from zyongye as a code owner July 23, 2026 20:06

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 23, 2026
@mgoin
mgoin enabled auto-merge (squash) July 23, 2026 20:13
@vllm-bot
vllm-bot merged commit bf27e34 into vllm-project:main Jul 24, 2026
58 of 61 checks passed
@toffee-desuwa

Copy link
Copy Markdown

Thanks for the ping @kylesayrs. Does the scope cut mean the wkv/wgate loading question is open again, or is the plan still to skip quantizing those layers at the checkpoint level? Happy to pick the loader side back up if that's useful.

kylesayrs added a commit to vllm-project/llm-compressor that referenced this pull request Jul 24, 2026
## Prerequisites ##
### Accelerate/ Transformers ###
- [x] huggingface/accelerate#4083
- [x] huggingface/transformers#46749
- [x] huggingface/transformers#46849
- [x] huggingface/transformers#47018
  - [x] huggingface/transformers#46902
- [x] huggingface/transformers#46965


### Compressed Tensors ###
- [x] vllm-project/compressed-tensors#731
- [x] vllm-project/compressed-tensors#730
- [x] vllm-project/compressed-tensors#728
- [x] vllm-project/compressed-tensors#752
- [x] vllm-project/compressed-tensors#750
- [x] vllm-project/compressed-tensors#759

### LLM Compressor ###
- [x] #2647
- [x] #2686
- [x] #2865

### vLLM ###
- [x] vllm-project/vllm#41276

## Testing ##
```
========== SAMPLE GENERATION ==============
Hello my name is Inigo Montoya. You killed my father. Prepare to die<|end▁of▁sentence|>
==========================================
```

---------

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
HDCharles pushed a commit to vllm-project/llm-compressor that referenced this pull request Aug 10, 2026
## Prerequisites ##
### Accelerate/ Transformers ###
- [x] huggingface/accelerate#4083
- [x] huggingface/transformers#46749
- [x] huggingface/transformers#46849
- [x] huggingface/transformers#47018
  - [x] huggingface/transformers#46902
- [x] huggingface/transformers#46965


### Compressed Tensors ###
- [x] vllm-project/compressed-tensors#731
- [x] vllm-project/compressed-tensors#730
- [x] vllm-project/compressed-tensors#728
- [x] vllm-project/compressed-tensors#752
- [x] vllm-project/compressed-tensors#750
- [x] vllm-project/compressed-tensors#759

### LLM Compressor ###
- [x] #2647
- [x] #2686
- [x] #2865

### vLLM ###
- [x] vllm-project/vllm#41276

## Testing ##
```
========== SAMPLE GENERATION ==============
Hello my name is Inigo Montoya. You killed my father. Prepare to die<|end▁of▁sentence|>
==========================================
```

---------

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Roderick-Wu pushed a commit to vllm-project/llm-compressor that referenced this pull request Aug 17, 2026
## Prerequisites ##
### Accelerate/ Transformers ###
- [x] huggingface/accelerate#4083
- [x] huggingface/transformers#46749
- [x] huggingface/transformers#46849
- [x] huggingface/transformers#47018
  - [x] huggingface/transformers#46902
- [x] huggingface/transformers#46965

### Compressed Tensors ###
- [x] vllm-project/compressed-tensors#731
- [x] vllm-project/compressed-tensors#730
- [x] vllm-project/compressed-tensors#728
- [x] vllm-project/compressed-tensors#752
- [x] vllm-project/compressed-tensors#750
- [x] vllm-project/compressed-tensors#759

### LLM Compressor ###
- [x] #2647
- [x] #2686
- [x] #2865

### vLLM ###
- [x] vllm-project/vllm#41276

## Testing ##
```
========== SAMPLE GENERATION ==============
Hello my name is Inigo Montoya. You killed my father. Prepare to die<|end▁of▁sentence|>
==========================================
```

---------

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Roderick-Wu <roderickwu2003@gmail.com>
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek Related to DeepSeek models quantization ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants