Skip to content

[Feat] [CPU] quantization: Add CPU FP8/INT8 quant ops and compressed-tensors support - #41329

Merged
mingfeima merged 14 commits into
sgl-project:mainfrom
yanbing-j:yanbing/cpu-fp8-quant-support
Nov 18, 2026
Merged

mingfeima merged 14 commits into
sgl-project:mainfrom
yanbing-j:yanbing/cpu-fp8-quant-support

Conversation

@yanbing-j

@yanbing-j yanbing-j commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Add CPU (Xeon / AMX) support for the FP8 and INT8 quantization utilities and the compressed-tensors linear schemes that use them, so CPU execution goes through in-tree sgl_kernel CPU kernels (or torch-native code where not on the hot path) instead of falling into CUDA/Triton-only paths.

This PR supersedes #41330 (CPU INT8), merged here per review.

Modifications

FP8 quant ops (python/sglang/kernels/aot/csrc/cpu/fp8_quant.cpp)

New CPU ops, named after their CUDA counterparts (<cuda_op>_cpu):

CPU op CUDA counterpart Notes
sgl_per_token_group_quant_8bit_cpu(input, group_size, eps) sgl_per_token_group_quant_8bit FP8 (e4m3) output, bf16 input
sgl_per_token_quant_fp8_cpu(input) sgl_per_token_quant_fp8 dynamic per-token
sgl_per_tensor_quant_fp8_cpu(input, scale?) sgl_per_tensor_quant_fp8 scale given → static, otherwise dynamic per-tensor
mxfp8_group_quantize_cpu(input) — (Triton on CUDA) group 32, UE8M0 scales
  • Schemas are functional (return tensors), matching existing CPU ops; outputs never alias inputs.
  • Kernel structure: at::parallel_for on the outer loop, shared row helpers (absmax_row / quantize_row_fp8); #if CPU_CAPABILITY_AVX512 only wraps the 16-wide vector body, and the scalar loop handles the remainder (and is the full fallback without AVX512).
  • bf16 / fp16 / fp32 inputs supported for per-token, per-tensor and MXFP8; results are bit-exact with the torch reference clamp(x / scale).to(fp8).
  • Fake impls registered in cpu_graph_runner.py (register_cpu_compile_fake) for CPU graph / torch.compile.

Python dispatch

  • fp8_kernel.py: CPU branches for per_token_group_quant_fp8, static_quant_fp8 (ndim >= 2 inputs are flattened and restored), and scaled_fp8_quant (num_token_padding handled in Python, as in the CUDA path).
  • fp8_utils.py: CPU branch for mxfp8_group_quantize.
  • int8_kernel.py: CPU per_token_quant_int8 uses a torch-native implementation (it is not on the CPU hot path, since the AMX GEMMs quantize activations internally), so it has no K-alignment constraint.

compressed-tensors schemes on CPU

  • W8A16 FP8: routed to the existing fp8_per_tensor_scaled_mm_cpu AMX kernel instead of the Marlin path (apply_fp8_marlin_linear depends on GPU/Marlin packing). Per-tensor single-scale weights are supported; fused layers with multiple per-shard scales are rejected explicitly for now.
  • W8A8 FP8: explicitly raises NotImplementedError on CPU (the CPU FP8 GEMM here is W8A16).
  • W8A8 INT8: per-tensor scales are requantized to a single max scale, per-channel scales are used as-is, weights are packed via _amx_process_weight_after_loading, and the linear runs through int8_scaled_mm_with_quant. Static input scales, asymmetric activations, and AMX-unpackable weights are rejected explicitly instead of silently changing semantics.

Tests

test/registered/cpu/test_quant_ops.py, registered under stage-a-test-cpu-intel:

  • FP8 ops: bit-exact vs torch reference over bf16/fp16/fp32 and non-16-aligned K (1x7, 5x129, 33x4096); group quant with a remainder (group 40); MXFP8; static quant with 3D input; num_token_padding.
  • torch.library.opcheck (schema + fake tensor) for all new ops.
  • compressed-tensors W8A16 FP8 CPU path, multi-scale rejection, W8A8 FP8 unsupported.
  • per_token_quant_int8 CPU contract (dtype / shape / scale / cal_sum, including K=33).
  • compressed-tensors W8A8 INT8 AMX path (per-channel and per-tensor scales), plus rejection of unpacked weights, static input, and asymmetric input.
python -m pytest test/registered/cpu/test_quant_ops.py
# 16 passed, 19 subtests passed

Speed Tests and Profiling

No performance claim; these ops are not on the CPU decode/prefill hot path.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #44068336533
Latest PR Test (Extra): ⚠️ Not enabled -- add run-ci-nv-extra label to opt in.
Latest PR Test (AMD ROCm 10): ❌ Run #44068336499

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@yanbing-j
yanbing-j force-pushed the yanbing/cpu-fp8-quant-support branch from c353243 to 9820af6 Compare September 26, 2026 08:23
@yanbing-j yanbing-j changed the title Yanbing/cpu fp8 quant support Add CPU FP8 quantization support for compressed-tensors paths Sep 26, 2026

@mingfeima mingfeima left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put changes in gemm_fp8.cpp into a new file fp8_quant.cpp

Comment on lines +1430 to +1431
template <typename scalar_t>
void quantize_groups_to_fp8_kernel(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the only useful dtype is bfloat16 now?
provide template specification for bfloat16 ONLY with CPU_CAPABILITY_AVX512 is more clean.

going to save a few lines and you can generally get rid of load_fp8_quant_input_avx512

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the current CPU FP8 path, the only intended and tested input dtype for per_token_group_quant_fp8_cpu is bf16 activation. Other dtypes are not needed for the serving path, so the CPU op can explicitly restrict to bf16.

I updated the implementation to make per_token_group_quant_fp8_cpu accept bf16 input only, and removed the unused scalar_t dispatch from quantize_groups_to_fp8_kernel. The CPU quantization tests were also updated to use bf16 inputs for the per-token dynamic path.

Comment thread test/registered/cpu/test_quant_ops.py Outdated
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.test_utils import CustomTestCase

register_cpu_ci(est_time=5, suite="base-b-test-cpu")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"base-b-test-cpu" is not a CPU suite in test/run_suite.py. Xeon runs stage-a-test-cpu-intel / stage-b-test-cpu-intel.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to stage-a-test-cpu-intel.

@mingfeima mingfeima added intel cpu cpu backend performance optimization run-ci CI: run the baseline test suite on this PR labels Sep 29, 2026
@mingfeima mingfeima changed the title Add CPU FP8 quantization support for compressed-tensors paths [CPU] Add FP8 quantization support for compressed-tensors paths Sep 29, 2026
@yanbing-j
yanbing-j force-pushed the yanbing/cpu-fp8-quant-support branch from 9b5953d to ff4dc7b Compare October 15, 2026 06:47
@yanbing-j
yanbing-j requested a review from mingfeima October 15, 2026 06:47

@mingfeima mingfeima left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we want to align cpu kernel names with CUDA counterparts, align the func names.

inside the kernels, put at::parallel_for in outer loop and #if defined(CPU_CAPABILITY_AVX512) for remainder handling to remove duplicated code.

@mingfeima mingfeima left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CPU quantization kernels are a reasonable direction, but compressed-tensors CPU enabling still needs the capability-selection and bias-preparation fixes below. Please also move FP8 weight packing out of the forward path and add coverage through normal scheme selection/loading. Relevant CPU quantization and CUDA kernel CI coverage remains unverified on this head; the observed Mooncake JIT failure also occurs on main and is not evidence of a regression introduced by this PR.

# So if we have a fused module (QKV, MLP) with per tensor scales,
# we expand each scale to its shard's channels.
def process_weights_after_loading(self, layer) -> None:
if _is_cpu:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new CPU schemes are still unreachable through normal loading: CompressedTensorsConfig.get_linear_scheme() (compressed_tensors.py:1147-1148) exempts NPU but not CPU from _check_scheme_supported(), which calls torch.cuda.get_device_capability() and raises on CPU-only PyTorch. Please add CPU capability handling and test the normal config-to-scheme selection path; the current tests construct schemes directly and bypass this failure.

layer.weight_scale = torch.nn.Parameter(
layer.weight_scale.reshape(()), requires_grad=False
)
return

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This CPU preparation leaves BF16/FP16 bias unchanged, but fp8_per_tensor_scaled_mm_cpu calls get_bias_data(), which requires FP32 via data_ptr<float>(). Please normalize bias during loading and add a test starting with a model-dtype bias; the current test supplies FP32 bias manually and misses this failure.

layer.weight_scale,
bias,
x.dtype,
False,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing is_vnni=False makes fp8_per_tensor_scaled_mm_cpu call convert_weight_packed() on every forward, including each decode step. Please reuse load-time AMX weight packing and pass the packed-layout flag, rather than repacking the full weight in the inference hot path.

@yanbing-j
yanbing-j force-pushed the yanbing/cpu-fp8-quant-support branch from 0743f9e to 73f421f Compare November 6, 2026 05:57
@yanbing-j
yanbing-j requested a review from mingfeima November 6, 2026 06:08
@mingfeima
mingfeima force-pushed the yanbing/cpu-fp8-quant-support branch from 73f421f to 60f0535 Compare November 9, 2026 06:07
@github-actions

This comment has been minimized.

@mingfeima

Copy link
Copy Markdown
Collaborator

@yanbing-j rebase.

@yanbing-j
yanbing-j force-pushed the yanbing/cpu-fp8-quant-support branch from 27171e2 to 9e8ab0f Compare November 17, 2026 10:49
Register and route CPU implementations for:
- per_token_group_quant_fp8_cpu
- scaled_fp8_quant_cpu
- mxfp8_group_quantize_cpu

The kernels cover the raw row-major CPU-supported FP8 quantization modes and
use local parallel CPU loops instead of high-level ATen reductions/casts. The
Python dispatch keeps CUDA-only layouts and fused variants out of the CPU path
unless the CPU kernel implements matching semantics.

Add registered CPU unit coverage for per-group FP8 quantization, dynamic/static
scaled FP8 quantization, padding behavior, and MXFP8 UE8M0 scales.
Route compressed-tensors W8A16 FP8 per-tensor weights to the CPU FP8 W8A16
matmul kernel. Keep true compressed-tensors W8A8 FP8 linear unsupported on CPU
because the GNR CPU path does not provide CUDA-equivalent FP8 activation x FP8
weight GEMM semantics.

Add registered CPU coverage for the W8A16 apply path and for the W8A8 FP8
NotImplementedError so the CPU path cannot silently use a W8A16 fallback under a
W8A8 API.
Make per_token_group_quant_fp8_cpu accept only bfloat16 input and remove the
unused scalar_t dispatch from quantize_groups_to_fp8_kernel. Update the CPU
quantization tests to use bf16 inputs and compare dequantized fp32 values
accordingly.
Flatten CPU static_quant_fp8 inputs to 2D before calling the CPU scaled FP8
quantization op, then restore the original quantized tensor shape. This keeps
the existing static_quant_fp8 contract for ndim >= 2 tensors while still using
the 2D-only CPU kernel underneath.

Add a 3D CPU regression test to cover the shape-preserving static quantization
path.
@mingfeima
mingfeima merged commit bff44f5 into sgl-project:main Nov 18, 2026
169 of 179 checks passed
Shunkangz added a commit to Shunkangz/sglang that referenced this pull request Nov 18, 2026
… suite

sgl-project#50874 renamed the Xeon suites stage-a/b-test-cpu-intel to
extra-test-cpu-intel. sgl-project#41329 merged afterwards still registering
test_quant_ops.py on stage-a-test-cpu-intel, which no workflow runs, so
check-registered-tests fails on main and on every PR based on it. Move it
to extra-test-cpu-intel, the suite pr-test-xeon.yml runs and every other
Intel CPU kernel test under test/registered/cpu uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cpu cpu backend performance optimization intel jit-kernel quant LLM Quantization run-ci CI: run the baseline test suite on this PR sgl-kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants