Skip to content

[Quantization][Autoround][Toolkit] Add W4A16 Support - #39778

Merged
jikunshang merged 19 commits into
vllm-project:mainfrom
yiliu30:zhenzhong/w4a16_ark
May 14, 2026
Merged

jikunshang merged 19 commits into
vllm-project:mainfrom
yiliu30:zhenzhong/w4a16_ark

Conversation

@Zhenzhong1

@Zhenzhong1 Zhenzhong1 commented Apr 14, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

We want to introduce robust native XPU/CPU support for the INT4/FP8 AutoRound format via Intel AutoRound Toolkit, enabling efficient and stable inference for AutoRound-quantized LLMs.

Plan

This PR is part of RFC: Intel Quantization Support Roadmap and RFC: Support Intel Quantization Toolkit AutoRound on Intel Platforms

This PR supports: XPU & CPU: Linear (W4A16) - Weight-Only Integer Quantization.

There are many stages to introduce the AutoRound Toolkit.

  • Stage 1: enable the code via AutoRound Toolkit;
  • Stage 2: collect the API feedback

Documents

For Installation and API usage, please refer to README, RFC.

Test

python3 examples/basic/offline_inference/generate.py  \
--model OPEA/Qwen2.5-0.5B-Instruct-int4-sym-inc       \
--block-size 64 --enforce-eager --max-model-len 4096  --gpu-memory-utilization  0.8
image

@mergify mergify Bot added the intel-gpu Related to Intel GPU label Apr 14, 2026
@Zhenzhong1 Zhenzhong1 changed the title [Quantization][Autoround][XPU] Add W4A16 Support [Quantization][Autoround][ARK] Add W4A16 Support Apr 14, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the INCXPULinearARKMethod to support INC quantization on XPU using the ARK backend, featuring weight repacking and a fallback mechanism. Feedback focuses on improving code cleanliness by removing debug log statements and redundant commented-out code.

@bigPYJ1151 bigPYJ1151 added the verified Run pre-commit for new contributors without triggering other tests label Apr 14, 2026
@mergify

mergify Bot commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Hi @Zhenzhong1, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?
mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:
# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

@yiliu30 yiliu30 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember 2-bit is supported by ARK — does this PR target it?

@mergify

mergify Bot commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Hi @Zhenzhong1, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?
mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:
# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

@Zhenzhong1

Zhenzhong1 commented Apr 15, 2026 •

Copy link
Copy Markdown
Contributor Author

@lvliang-intel @luoyu-intel @thuang6 Please help to review.

@Zhenzhong1
Zhenzhong1 force-pushed the zhenzhong/w4a16_ark branch from a805bc5 to d0b61bc Compare April 15, 2026 02:35
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>

@yiliu30 yiliu30 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments and please add test back.

Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
@mergify

mergify Bot commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

Hi @Zhenzhong1, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?
mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:
# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
@bigPYJ1151

Copy link
Copy Markdown
Member

@jikunshang Please take a review

@bigPYJ1151
bigPYJ1151 requested a review from jikunshang April 16, 2026 09:47
@jikunshang

Copy link
Copy Markdown
Member

I think rocm's mori support PR is a good example to introduce a new third party dependency into vllm. I don't want merge something looks like a block box to developer and user.

…inc_weights method

Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>

@yiliu30 yiliu30 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please keep the naming consistent; others lgtm.

@Zhenzhong1

Zhenzhong1 commented May 11, 2026 •

Copy link
Copy Markdown
Contributor Author

@jikunshang Hi, The Intel-CI failed on test_parallel_sampling[False] for the XPU backend with an AssertionError: 18 unique completions; expected 19. error output

This seems to be a seed collision issue on the XPU backend and is unrelated to my changes in inc.py. Could you please take a look?

@jikunshang

Copy link
Copy Markdown
Member

@jikunshang Hi, The Intel-CI failed on test_parallel_sampling[False] for the XPU backend with an AssertionError: 18 unique completions; expected 19. error output

This seems to be a seed collision issue on the XPU backend and is unrelated to my changes in inc.py. Could you please take a look?

you are right, we will prioritize to fix it on main.

@Zhenzhong1
Zhenzhong1 force-pushed the zhenzhong/w4a16_ark branch from 1f874e8 to 335e961 Compare May 12, 2026 07:40
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Comment thread requirements/xpu.txt Resolved
@Zhenzhong1
Zhenzhong1 force-pushed the zhenzhong/w4a16_ark branch from 4549b0c to 335e961 Compare May 13, 2026 02:12
Signed-off-by: Zhenzhong Xu <zhenzhong.xu@intel.com>
Comment thread requirements/xpu.txt Resolved
@Zhenzhong1
Zhenzhong1 force-pushed the zhenzhong/w4a16_ark branch from b91de6e to 708dcb9 Compare May 14, 2026 02:00
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
@Zhenzhong1
Zhenzhong1 requested a review from jikunshang May 14, 2026 07:38
@jikunshang jikunshang added the ready ONLY add when PR is ready to merge/full CI is needed label May 14, 2026
@jikunshang
jikunshang merged commit 1ea9401 into vllm-project:main May 14, 2026
73 checks passed
mfylcek pushed a commit to mfylcek/vllm that referenced this pull request May 19, 2026
)

Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Signed-off-by: Zhenzhong Xu <zhenzhong.xu@intel.com>
h1t35h pushed a commit to h1t35h/vllm that referenced this pull request May 21, 2026
)

Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Signed-off-by: Zhenzhong Xu <zhenzhong.xu@intel.com>
Liuweixiong0118 pushed a commit to Liuweixiong0118/vllm that referenced this pull request Jun 1, 2026
)

Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Signed-off-by: Zhenzhong Xu <zhenzhong.xu@intel.com>
Signed-off-by: Liuweixiong0118 <lwx34158427@gmail.com>
philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026
)

Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Signed-off-by: Zhenzhong Xu <zhenzhong.xu@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build intel-gpu Related to Intel GPU ready ONLY add when PR is ready to merge/full CI is needed verified Run pre-commit for new contributors without triggering other tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants