Repository navigation
[Feat] [CPU] quantization: Add CPU FP8/INT8 quant ops and compressed-tensors support - #41329
Conversation
c353243 to
9820af6
Compare
mingfeima
left a comment
There was a problem hiding this comment.
put changes in gemm_fp8.cpp into a new file fp8_quant.cpp
| template <typename scalar_t> | ||
| void quantize_groups_to_fp8_kernel( |
There was a problem hiding this comment.
the only useful dtype is bfloat16 now?
provide template specification for bfloat16 ONLY with CPU_CAPABILITY_AVX512 is more clean.
going to save a few lines and you can generally get rid of load_fp8_quant_input_avx512
There was a problem hiding this comment.
For the current CPU FP8 path, the only intended and tested input dtype for per_token_group_quant_fp8_cpu is bf16 activation. Other dtypes are not needed for the serving path, so the CPU op can explicitly restrict to bf16.
I updated the implementation to make per_token_group_quant_fp8_cpu accept bf16 input only, and removed the unused scalar_t dispatch from quantize_groups_to_fp8_kernel. The CPU quantization tests were also updated to use bf16 inputs for the per-token dynamic path.
| from sglang.test.ci.ci_register import register_cpu_ci | ||
| from sglang.test.test_utils import CustomTestCase | ||
|
|
||
| register_cpu_ci(est_time=5, suite="base-b-test-cpu") |
There was a problem hiding this comment.
"base-b-test-cpu" is not a CPU suite in test/run_suite.py. Xeon runs stage-a-test-cpu-intel / stage-b-test-cpu-intel.
There was a problem hiding this comment.
Updated to stage-a-test-cpu-intel.
9b5953d to
ff4dc7b
Compare
mingfeima
left a comment
There was a problem hiding this comment.
we want to align cpu kernel names with CUDA counterparts, align the func names.
inside the kernels, put at::parallel_for in outer loop and #if defined(CPU_CAPABILITY_AVX512) for remainder handling to remove duplicated code.
mingfeima
left a comment
There was a problem hiding this comment.
The CPU quantization kernels are a reasonable direction, but compressed-tensors CPU enabling still needs the capability-selection and bias-preparation fixes below. Please also move FP8 weight packing out of the forward path and add coverage through normal scheme selection/loading. Relevant CPU quantization and CUDA kernel CI coverage remains unverified on this head; the observed Mooncake JIT failure also occurs on main and is not evidence of a regression introduced by this PR.
| # So if we have a fused module (QKV, MLP) with per tensor scales, | ||
| # we expand each scale to its shard's channels. | ||
| def process_weights_after_loading(self, layer) -> None: | ||
| if _is_cpu: |
There was a problem hiding this comment.
The new CPU schemes are still unreachable through normal loading: CompressedTensorsConfig.get_linear_scheme() (compressed_tensors.py:1147-1148) exempts NPU but not CPU from _check_scheme_supported(), which calls torch.cuda.get_device_capability() and raises on CPU-only PyTorch. Please add CPU capability handling and test the normal config-to-scheme selection path; the current tests construct schemes directly and bypass this failure.
| layer.weight_scale = torch.nn.Parameter( | ||
| layer.weight_scale.reshape(()), requires_grad=False | ||
| ) | ||
| return |
There was a problem hiding this comment.
This CPU preparation leaves BF16/FP16 bias unchanged, but fp8_per_tensor_scaled_mm_cpu calls get_bias_data(), which requires FP32 via data_ptr<float>(). Please normalize bias during loading and add a test starting with a model-dtype bias; the current test supplies FP32 bias manually and misses this failure.
| layer.weight_scale, | ||
| bias, | ||
| x.dtype, | ||
| False, |
There was a problem hiding this comment.
Passing is_vnni=False makes fp8_per_tensor_scaled_mm_cpu call convert_weight_packed() on every forward, including each decode step. Please reuse load-time AMX weight packing and pass the packed-layout flag, rather than repacking the full weight in the inference hot path.
0743f9e to
73f421f
Compare
73f421f to
60f0535
Compare
This comment has been minimized.
This comment has been minimized.
|
@yanbing-j rebase. |
27171e2 to
9e8ab0f
Compare
Register and route CPU implementations for: - per_token_group_quant_fp8_cpu - scaled_fp8_quant_cpu - mxfp8_group_quantize_cpu The kernels cover the raw row-major CPU-supported FP8 quantization modes and use local parallel CPU loops instead of high-level ATen reductions/casts. The Python dispatch keeps CUDA-only layouts and fused variants out of the CPU path unless the CPU kernel implements matching semantics. Add registered CPU unit coverage for per-group FP8 quantization, dynamic/static scaled FP8 quantization, padding behavior, and MXFP8 UE8M0 scales.
Route compressed-tensors W8A16 FP8 per-tensor weights to the CPU FP8 W8A16 matmul kernel. Keep true compressed-tensors W8A8 FP8 linear unsupported on CPU because the GNR CPU path does not provide CUDA-equivalent FP8 activation x FP8 weight GEMM semantics. Add registered CPU coverage for the W8A16 apply path and for the W8A8 FP8 NotImplementedError so the CPU path cannot silently use a W8A16 fallback under a W8A8 API.
Make per_token_group_quant_fp8_cpu accept only bfloat16 input and remove the unused scalar_t dispatch from quantize_groups_to_fp8_kernel. Update the CPU quantization tests to use bf16 inputs and compare dequantized fp32 values accordingly.
Flatten CPU static_quant_fp8 inputs to 2D before calling the CPU scaled FP8 quantization op, then restore the original quantized tensor shape. This keeps the existing static_quant_fp8 contract for ndim >= 2 tensors while still using the 2D-only CPU kernel underneath. Add a 3D CPU regression test to cover the shape-preserving static quantization path.
…epacking every forward
…stead of rejecting them
…ough normal scheme selection
… suite sgl-project#50874 renamed the Xeon suites stage-a/b-test-cpu-intel to extra-test-cpu-intel. sgl-project#41329 merged afterwards still registering test_quant_ops.py on stage-a-test-cpu-intel, which no workflow runs, so check-registered-tests fails on main and on every PR based on it. Move it to extra-test-cpu-intel, the suite pr-test-xeon.yml runs and every other Intel CPU kernel test under test/registered/cpu uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Motivation
Add CPU (Xeon / AMX) support for the FP8 and INT8 quantization utilities and the compressed-tensors linear schemes that use them, so CPU execution goes through in-tree
sgl_kernelCPU kernels (or torch-native code where not on the hot path) instead of falling into CUDA/Triton-only paths.This PR supersedes #41330 (CPU INT8), merged here per review.
Modifications
FP8 quant ops (
python/sglang/kernels/aot/csrc/cpu/fp8_quant.cpp)New CPU ops, named after their CUDA counterparts (
<cuda_op>_cpu):sgl_per_token_group_quant_8bit_cpu(input, group_size, eps)sgl_per_token_group_quant_8bitsgl_per_token_quant_fp8_cpu(input)sgl_per_token_quant_fp8sgl_per_tensor_quant_fp8_cpu(input, scale?)sgl_per_tensor_quant_fp8scalegiven → static, otherwise dynamic per-tensormxfp8_group_quantize_cpu(input)at::parallel_foron the outer loop, shared row helpers (absmax_row/quantize_row_fp8);#if CPU_CAPABILITY_AVX512only wraps the 16-wide vector body, and the scalar loop handles the remainder (and is the full fallback without AVX512).clamp(x / scale).to(fp8).cpu_graph_runner.py(register_cpu_compile_fake) for CPU graph / torch.compile.Python dispatch
fp8_kernel.py: CPU branches forper_token_group_quant_fp8,static_quant_fp8(ndim >= 2 inputs are flattened and restored), andscaled_fp8_quant(num_token_paddinghandled in Python, as in the CUDA path).fp8_utils.py: CPU branch formxfp8_group_quantize.int8_kernel.py: CPUper_token_quant_int8uses a torch-native implementation (it is not on the CPU hot path, since the AMX GEMMs quantize activations internally), so it has no K-alignment constraint.compressed-tensors schemes on CPU
fp8_per_tensor_scaled_mm_cpuAMX kernel instead of the Marlin path (apply_fp8_marlin_lineardepends on GPU/Marlin packing). Per-tensor single-scale weights are supported; fused layers with multiple per-shard scales are rejected explicitly for now.NotImplementedErroron CPU (the CPU FP8 GEMM here is W8A16)._amx_process_weight_after_loading, and the linear runs throughint8_scaled_mm_with_quant. Static input scales, asymmetric activations, and AMX-unpackable weights are rejected explicitly instead of silently changing semantics.Tests
test/registered/cpu/test_quant_ops.py, registered understage-a-test-cpu-intel:num_token_padding.torch.library.opcheck(schema + fake tensor) for all new ops.per_token_quant_int8CPU contract (dtype / shape / scale /cal_sum, including K=33).python -m pytest test/registered/cpu/test_quant_ops.py # 16 passed, 19 subtests passedSpeed Tests and Profiling
No performance claim; these ops are not on the CPU decode/prefill hot path.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #44068336533⚠️ Not enabled -- add
Latest PR Test (Extra):
run-ci-nv-extralabel to opt in.Latest PR Test (AMD ROCm 10): ❌ Run #44068336499
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: