Repository navigation
[Quantization][Autoround][Toolkit] Add W4A16 Support - #39778
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the INCXPULinearARKMethod to support INC quantization on XPU using the ARK backend, featuring weight repacking and a fallback mechanism. Feedback focuses on improving code cleanliness by removing debug log statements and redundant commented-out code.
|
Hi @Zhenzhong1, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, Tip Is
|
yiliu30
left a comment
There was a problem hiding this comment.
I remember 2-bit is supported by ARK — does this PR target it?
|
Hi @Zhenzhong1, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, Tip Is
|
|
@lvliang-intel @luoyu-intel @thuang6 Please help to review. |
a805bc5 to
d0b61bc
Compare
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
yiliu30
left a comment
There was a problem hiding this comment.
Left a few comments and please add test back.
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
|
Hi @Zhenzhong1, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, Tip Is
|
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
|
@jikunshang Please take a review |
|
I think rocm's mori support PR is a good example to introduce a new third party dependency into vllm. I don't want merge something looks like a block box to developer and user. |
…inc_weights method Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
yiliu30
left a comment
There was a problem hiding this comment.
Please keep the naming consistent; others lgtm.
|
@jikunshang Hi, The Intel-CI failed on test_parallel_sampling[False] for the XPU backend with an AssertionError: 18 unique completions; expected 19. error output This seems to be a seed collision issue on the XPU backend and is unrelated to my changes in inc.py. Could you please take a look? |
you are right, we will prioritize to fix it on main. |
1f874e8 to
335e961
Compare
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
4549b0c to
335e961
Compare
Signed-off-by: Zhenzhong Xu <zhenzhong.xu@intel.com>
b91de6e to
708dcb9
Compare
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com>
Motivation
We want to introduce robust native XPU/CPU support for the INT4/FP8 AutoRound format via Intel AutoRound Toolkit, enabling efficient and stable inference for AutoRound-quantized LLMs.
Plan
This PR is part of RFC: Intel Quantization Support Roadmap and RFC: Support Intel Quantization Toolkit AutoRound on Intel Platforms
This PR supports: XPU & CPU: Linear (W4A16) - Weight-Only Integer Quantization.
There are many stages to introduce the AutoRound Toolkit.
Documents
For Installation and API usage, please refer to README, RFC.
Test