Skip to content

[CPU] Add GGUF Q4/Q8 and AMX quantization support - #41331

Draft
yanbing-j wants to merge 5 commits into
sgl-project:mainfrom
yanbing-j:yanbing/cpu-int4-gguf-quant-support
Draft

yanbing-j wants to merge 5 commits into
sgl-project:mainfrom
yanbing-j:yanbing/cpu-int4-gguf-quant-support

Conversation

@yanbing-j

@yanbing-j yanbing-j commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

This PR extends CPU quantization coverage by enabling INT4/GGUF inference paths on Xeon CPU backends. It adds CPU support for GGUF Q4_0/Q8_0 matmul and routes AWQ/GPTQ/compressed-tensors INT4 schemes through the existing Intel AMX backend where supported.

Changes

  • Add a CPU GGUF matmul kernel for Q4_0 and Q8_0 weights.
  • Register gguf_mul_mat_cpu in the CPU torch extension.
  • Route GGUF quantized linear layers to the CPU kernel when running on CPU.
  • Enable AWQ and GPTQ configs to select Intel AMX CPU INT4 schemes.
  • Add CPU support for asymmetric compressed-tensors WNA16 4-bit weights through the existing AMX INT4 kernel.
  • Add registered CPU tests for GGUF Q4_0/Q8_0, AWQ/GPTQ CPU scheme selection, and compressed-tensors WNA16 CPU behavior.

Scope note: symmetric WNA16

This PR intentionally supports asymmetric WNA16 CPU only.

The symmetric WNA16 path is not routed through the current CPU INT4 kernel because its zero-point semantics need to be verified separately against the compressed-tensors/Marlin path. Implementing symmetric INT4 CPU packing in this PR would risk introducing an unverified silent wrong-result path.

Instead, the CPU path now fails loudly for symmetric=True with NotImplementedError. This keeps the current PR scoped to the validated asymmetric path while preventing silent incorrect behavior. A rejection test was also added for the symmetric case, and the asymmetric test now exercises process_weights_after_loading before apply_weights.

Testing

Added/updated registered CPU tests covering:

  • GGUF Q8_0 CPU matmul
  • GGUF Q4_0 CPU matmul
  • AWQ/GPTQ CPU INT4 scheme selection
  • GPTQ CPU rejection for unsupported activation ordering
  • compressed-tensors WNA16 asymmetric CPU apply path
  • compressed-tensors WNA16 symmetric CPU rejection

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #36517104045
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ❌ Run #36517104001

@github-actions github-actions Bot added quant LLM Quantization sgl-kernel labels Sep 26, 2026
Register gguf_mul_mat_cpu and route GGUF linear layers on CPU for Q4_0 and
Q8_0 packed weights.

The decode path uses fused AVX512 dequantize + dot-product for single-token
matmul. The prefill path dequantizes weights to bf16 with AVX512 and then uses
ATen linear so the CPU backend can use the optimized bf16/BRGEMM path.

Add registered CPU tests for Q8_0 and Q4_0 against dequantized reference
matmul.
Map GPTQMarlinConfig and AWQMarlinConfig to the Intel AMX GPTQ/AWQ INT4
schemes when running on CPU, including MoE scheme selection. Populate the GPTQ
Marlin compatibility fields needed by the CPU AMX checker so unsupported
act-order configurations fail explicitly instead of via missing attributes.

Add registered CPU coverage for GPTQ/AWQ scheme selection and desc_act rejection
on incompatible GPTQ Marlin CPU configs.
Convert compressed-tensors WNA16 4-bit weight parameters into the qweight,
qzeros, and scales layout expected by the CPU INT4 AMX kernels, then apply via
int4_scaled_mm_cpu.

Reject unsupported CPU cases such as activation ordering and non-4-bit weights
instead of silently taking a mismatched CUDA/Marlin path. Add registered CPU
coverage for the packed apply path against the direct INT4 CPU kernel.
@yanbing-j yanbing-j changed the title Yanbing/cpu int4 gguf quant support Add CPU INT4 GGUF and AMX quantization support Sep 26, 2026
Fail loudly for symmetric compressed-tensors WNA16 on CPU instead of synthesizing zero points for the asymmetric INT4 path.

Exercise the actual process_weights_after_loading path for asymmetric WNA16 and add a symmetric rejection regression test.
@mingfeima

Copy link
Copy Markdown
Collaborator

Pending this feature.

This is not the right way of implementing gguf here ==> dequat to bf16 is not the way that gguf handling (it's based on amx-int8 for any low bit matmul with group quantization).

Aside from that Q4 and Q8 are not very valuable since we already have them in sgl kernels.
<=3 bit is somewhat worth doing but definitely not in this way.

@mingfeima mingfeima added intel cpu cpu backend performance optimization labels Sep 29, 2026
@mingfeima mingfeima changed the title Add CPU INT4 GGUF and AMX quantization support [CPU] Add GGUF Q4/Q8 and AMX quantization support Sep 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cpu cpu backend performance optimization intel quant LLM Quantization sgl-kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants