Repository navigation
Conversation
Register gguf_mul_mat_cpu and route GGUF linear layers on CPU for Q4_0 and Q8_0 packed weights. The decode path uses fused AVX512 dequantize + dot-product for single-token matmul. The prefill path dequantizes weights to bf16 with AVX512 and then uses ATen linear so the CPU backend can use the optimized bf16/BRGEMM path. Add registered CPU tests for Q8_0 and Q4_0 against dequantized reference matmul.
Map GPTQMarlinConfig and AWQMarlinConfig to the Intel AMX GPTQ/AWQ INT4 schemes when running on CPU, including MoE scheme selection. Populate the GPTQ Marlin compatibility fields needed by the CPU AMX checker so unsupported act-order configurations fail explicitly instead of via missing attributes. Add registered CPU coverage for GPTQ/AWQ scheme selection and desc_act rejection on incompatible GPTQ Marlin CPU configs.
Convert compressed-tensors WNA16 4-bit weight parameters into the qweight, qzeros, and scales layout expected by the CPU INT4 AMX kernels, then apply via int4_scaled_mm_cpu. Reject unsupported CPU cases such as activation ordering and non-4-bit weights instead of silently taking a mismatched CUDA/Marlin path. Add registered CPU coverage for the packed apply path against the direct INT4 CPU kernel.
Fail loudly for symmetric compressed-tensors WNA16 on CPU instead of synthesizing zero points for the asymmetric INT4 path. Exercise the actual process_weights_after_loading path for asymmetric WNA16 and add a symmetric rejection regression test.
Collaborator
|
Pending this feature. This is not the right way of implementing gguf here ==> dequat to bf16 is not the way that gguf handling (it's based on amx-int8 for any low bit matmul with group quantization). Aside from that Q4 and Q8 are not very valuable since we already have them in sgl kernels. |
This was referenced Nov 4, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This PR extends CPU quantization coverage by enabling INT4/GGUF inference paths on Xeon CPU backends. It adds CPU support for GGUF Q4_0/Q8_0 matmul and routes AWQ/GPTQ/compressed-tensors INT4 schemes through the existing Intel AMX backend where supported.
Changes
gguf_mul_mat_cpuin the CPU torch extension.Scope note: symmetric WNA16
This PR intentionally supports asymmetric WNA16 CPU only.
The symmetric WNA16 path is not routed through the current CPU INT4 kernel because its zero-point semantics need to be verified separately against the compressed-tensors/Marlin path. Implementing symmetric INT4 CPU packing in this PR would risk introducing an unverified silent wrong-result path.
Instead, the CPU path now fails loudly for
symmetric=TruewithNotImplementedError. This keeps the current PR scoped to the validated asymmetric path while preventing silent incorrect behavior. A rejection test was also added for the symmetric case, and the asymmetric test now exercisesprocess_weights_after_loadingbeforeapply_weights.Testing
Added/updated registered CPU tests covering:
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #36517104045
Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.Latest PR Test (AMD ROCm 10): ❌ Run #36517104001