Repository navigation
[WIP, Kernel] (3/N) Machete W4A8 - #8046
Closed
LucasWilkinson wants to merge 18 commits into
Closed
LucasWilkinson wants to merge 18 commits into
LucasWilkinson wants to merge 18 commits into
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge). To run full CI, you can do one of these:
🚀 |
LucasWilkinson
changed the title
Contributor
|
Hi @LucasWilkinson, I ran the w4a8 benchmark in the PR, the gemm perf from machete is 15-20% slower than the marline kernels. Is that expected? Thanks. |
Contributor
Author
|
@cli99 Thanks for your interest in the kernels! For starters I will preface by saying this is work-in-progress PR so I still need to do some performance tuning (the updated heuristic in #7701 will likely help this PR once its merged). We do expect that for an M dim (batch-size * seq-len) <= 64 for Marlin to outperform this current PR, but you should see speedups for M > 64 and larger speedups at M >= 128 for most shapes. What were the shapes that you tested? |
LucasWilkinson
force-pushed
the
lwilkinson/machete-w4a8
branch
from
October 2, 2024 02:11
358a67a to
76a5c3d
Compare
LucasWilkinson
force-pushed
the
lwilkinson/machete-w4a8
branch
from
October 7, 2024 04:18
261b1c2 to
02fb2c0
Compare
|
Hi, @LucasWilkinson, thank you for your amazing work! I wanted to install machete-w4a8 for testing, and the installation was successful. However, when I ran benchmark_machete.py, I encountered the following issue: File "~/vllm-0.6.3-machete-w4a8/./benchmarks/kernels/benchmark_machete.py", line 255, in <lambda>
return lambda: ops.machete_mm(
File "~/vllm-0.6.3-machete-w4a8/vllm/_custom_ops.py", line 43, in wrapper
raise NotImplementedError(msg % (fn.__name__, e)) from e
NotImplementedError: Error in calling custom op machete_mm: Could not run '_C::machete_mm' with arguments from the 'CUDA' backend. This could be because the
operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee usi
ng PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. '_C::machete_mm' is only available for these backends: [Meta, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy, AutogradMeta, Tracer, AutocastCPU, AutocastXPU, AutocastCUDA, FuncTorchBatched, Batched
NestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].Environment configuration:
Could you please provide some suggestions? |
Contributor
Author
|
Superseded by: #9855 |
wenhow
pushed a commit
to wenhow/vllm
that referenced
this pull request
Aug 5, 2026
…cture in xlite backend (vllm-project#8046) This PR is the result of the last two commits on `feat/xlite-qwen3-vl-moe`: 1. Refactor `LlamaXliteModel` so the shared `initialize` path uses `config.rope_head_dim` instead of duplicating subclass-specific setup. 2. Add `Qwen3VLMoeForConditionalGeneration` to the xlite strategy map and route it through `QwenMoeXliteModel`. The net effect is that Qwen3-VL MoE models can reuse the existing xlite initialization flow while still applying the MoE-specific config and weight wiring in the subclass. ### What this PR does / why we need it? This PR extends xlite backend coverage to the `Qwen3VLMoeForConditionalGeneration` architecture. The previous implementation only handled `Qwen3MoeForCausalLM` and related (non-)MoE variants, so Qwen3-VL MoE models were not routed into the xlite path. At the same time, the shared rotary embedding precomputation was normalized to use `config.rope_head_dim`, which keeps the base `initialize` implementation generic and avoids duplicated subclass-specific logic. ### Does this PR introduce _any_ user-facing change? Yes. Users running models with the `Qwen3VLMoeForConditionalGeneration` architecture can now use the xlite backend. ### How was this patch tested? First, `Qwen3-VL-235B-A22B-Instruct`->`QwenMoeXliteModel` was tested and the vLLM server started successfully with the xlite backend, including launching the worker process and processing requests without crashing (see the script below). Previous models (`Qwen3-32B`->`LlamaXliteModel`, `Qwen3-235B-A22B`->`QwenMoeXliteModel`, `GLM-4.7`->`Glm4MoeXliteModel`) were also re-tested to confirm no regressions. ```bash # Script to start the server and test a single request # using Docker image for Ascend A3 export VLLM_USE_MODELSCOPE=true export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True # server launch command (with xlite enabled) vllm serve /path/to/model \ --host 0.0.0.0 \ --port 8000 \ --api-server-count 1 \ --data-parallel-size 1 \ --data-parallel-size-local 1 \ --tensor-parallel-size 16 \ --served-model-name mymodel \ --max-num-seqs 16 \ --max-model-len 40960 \ --max-num-batched-tokens 4096 \ --enable-expert-parallel \ --trust-remote-code \ --async-scheduling \ --gpu-memory-utilization 0.9 \ --block-size 128 \ --allowed-local-media-path /path/to/media \ --additional-config='{"xlite_graph_config": {"enabled": true, "full_mode": true}}' \ # single request example curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "mymodel", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Tell me how to sleep well at night."} ], "max_tokens": 128, "temperature": "0.0" }' ``` Further tests were performed to confirm the accuracy of the outputs using `ais_benchmark==3.1.0`. For `Qwen3-VL-235B-A22B-Instruct`, multiple datasets were evaluated, including a multimodal dataset `mmmu`. The results are summarized in the table below. ```bash # command to run the evaluation for `Qwen3-VL-235B-A22B-Instruct` with xlite enabled ais_bench --models vllm_api_general_chat --datasets \ aime2024_gen_0_shot_chat_prompt \ ceval_gen_0_shot_cot_chat_prompt \ gpqa_gen_0_shot_cot_chat_prompt \ gsm8k_gen_0_shot_cot_chat_prompt \ math_prm800k_500_0shot_cot_gen \ mmlu_gen_0_shot_cot_chat_prompt \ mmmu_gen_cot \ --work-dir "outputs/Qwen3-VL-235B-A22B-Instruct-xlite" \ --mode all \ --dump-eval-details --merge-ds \ --max-num-workers 128 ``` For the `vllm_api_general_chat` task type, the corresponding `ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py` was modified as: ```python # ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py models = [ dict( attr="service", type=VLLMCustomAPIChat, abbr="vllm-api-general-chat", path="", model="mymodel", stream=False, request_rate=0, use_timestamp=False, retry=2, api_key="", host_ip="localhost", host_port=8000, url="", max_out_len=32768, batch_size=512, trust_remote_code=False, generation_kwargs=dict( temperature=0.01, top_k=10, top_p=0.95, seed=None, repetition_penalty=1.03, ignore_eos=False, ), pred_postprocessor=dict(type=extract_non_reasoning_content), ) ] ``` | dataset | version | metric | mode | w/ xlite | w/o xlite | Qwen official | | ----- | ----- | ----- | ----- | ----- | ----- | ----- | | aime2024 | 544e9a | accuracy | gen | 83.33 | _N.A._ | _74.7 (aime2025)_ | | ceval | - | accuracy-weighted | gen | 90.42 | 90.19 | _N.A._ | | gpqa | 5d9994 | accuracy | gen | 70.20 | _N.A._ | _N.A._ | | gsm8k | 271d0b | accuracy | gen | 96.51 | _N.A._ | _N.A._ | | livecodebench | 270e7b | pass@1 | gen | 52.61 | _N.A._ | _54.3 (LCBV6)_ | | math | 9eff90 | accuracy | gen | 94.40 | _N.A._ | _N.A._ | | mmlu | - | accuracy-weighted | gen | 89.94 | _N.A._ | 88.8 | | mmmu* | 14da4b | [validation]: Overall | gen | 71.11 | _N.A._ | 78.7 | | mmmu* | 14da4b | [dev]: Overall | gen | 69.33 | _N.A._ | _N.A._ | > *Multimodal datasets that include image inputs.\ > Qwen official: <https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct>\ > Bug fix from <AISBench/benchmark#238> were also merged to ensure the evaluation correctness for the `mmmu` dataset. For other previously supported models, their accuracies were also validated on the `ceval` dataset, with the results below: | model | dataset | version | metric | mode | w/ xlite | w/o xlite | | ----- | ----- | ----- | ----- | ----- | ----- | ----- | | Qwen3-32B | ceval | - | accuracy-weighted | gen | 88.26 | 88.48 | | Qwen3-235B-A22B | ceval | - | accuracy-weighted | gen | 91.01 | _N.A._ | | GLM-4.7 | ceval | - | accuracy-weighted | gen | 90.79 | _N.A._ | These results confirm that the xlite backend is functioning correctly for the `Qwen3VLMoeForConditionalGeneration` architecture, with no observed alterations to previous models. - vLLM version: v0.18.0 - vLLM main: vllm-project@29e4870 --------- Signed-off-by: Sijie Fu <fusijie@huawei.com> Co-authored-by: Sijie Fu <fusijie@huawei.com>
wenhow
pushed a commit
to wenhow/vllm
that referenced
this pull request
Aug 5, 2026
…troduce shared utils (vllm-project#8431) ### What this PR does / why we need it? This PR combines the last two commits in the xlite integration area to improve maintainability, type safety, and adapter extensibility. Main changes: - Refactors xlite model adapter structure in `vllm_ascend/xlite/xlite.py`. - Introduces a shared utility helper in `vllm_ascend/xlite/utils.py`: - adds _get_nested_attr to simplify safe nested attribute lookup. - Modernizes adapter interfaces and typing: - makes XliteModel an abstract base class with clearer extension points. - standardizes initialization return typing through XliteInitResult. - improves method signatures and removes redundant type-ignore usage where not needed. - Improves documentation quality: - expands and clarifies class and method docstrings across xlite model/wrapper code paths. - Keeps behavior aligned while making architecture-specific adapter logic easier to extend and review. Why needed: - The previous implementation had tighter coupling and repeated nested attribute access patterns, which made model-specific extension harder and increased maintenance risk. - Stronger typing and cleaner contracts improve readability, reduce ambiguity for future contributors, and help static analysis catch regressions earlier. Commit breakdown: - 1st: focused type-hint fixes in `xlite.py`. - 2nd: major xlite refactor + utils introduction + typing/doc cleanup. ### Does this PR introduce _any_ user-facing change? No direct user-facing API or behavior change is intended. This is primarily an internal refactor and code-quality improvement in xlite integration code paths. ### How was this patch tested? - Static type checking with mypy to ensure type correctness and catch any typing issues. - Manual code review to verify that refactor maintains intended behavior and improves code clarity. - Accuracy tests on xlite's previously supported models to confirm no regressions in output. The accuracy tests were conducted similarly to PR vllm-project#8046, using `ais_bench==0.3.1` with the `ceval` dataset. 10 models were tested, covering five archietectures and all three existing xlite model classes in `xlite.py`. No significant accuracy regression was observed across the board, with some minor fluctuations within expected variance. ```bash # command to run the evaluation for `Qwen3-VL-235B-A22B-Instruct` with xlite enabled ais_bench --models vllm_api_general_chat \ --datasets ceval_gen_0_shot_cot_chat_prompt \ --work-dir "outputs/Qwen3-VL-235B-A22B-Instruct-xlite" \ --mode all \ --dump-eval-details --merge-ds \ --max-num-workers 64 ``` | Model | Architecture | PyClass | metric | mode | w/ xlite | w/o xlite | | --- | --- | --- | --- | --- | --- | --- | | Qwen3-0.6B | Qwen3ForCausalLM | LlamaXliteModel | accuracy-weighted | gen | 50.97 | 50.52 | | Qwen3-32B | Qwen3ForCausalLM | LlamaXliteModel | accuracy-weighted | gen | 88.93 | 89.08 | | Qwen3-VL-8B-Instruct | Qwen3VLForConditionalGeneration | LlamaXliteModel | accuracy-weighted | gen | 83.58 | 83.36 | | Qwen3-VL-32B-Instruct | Qwen3VLForConditionalGeneration | LlamaXliteModel | accuracy-weighted | gen | 87.52 | 88.26 | | Qwen3-30B-A3B | Qwen3MoeForCausalLM | QwenMoeXliteModel | accuracy-weighted | gen | 86.48 | 86.26 | | Qwen3-30B-A3B-Instruct-2507 | Qwen3MoeForCausalLM | QwenMoeXliteModel | accuracy-weighted | gen | 86.18 | 86.48 | | Qwen3-235B-A22B | Qwen3MoeForCausalLM | QwenMoeXliteModel | accuracy-weighted | gen | 90.04 | 90.71 | | Qwen3-VL-30B-A3B-Instruct | Qwen3VLMoeForConditionalGeneration | QwenMoeXliteModel | accuracy-weighted | gen | 85.88 | 86.85 | | Qwen3-VL-235B-A22B-Instruct | Qwen3VLMoeForConditionalGeneration | QwenMoeXliteModel | accuracy-weighted | gen | 90.12 | 91.08 | | GLM-4.7 | Glm4MoeForCausalLM | Glm4MoeXliteModel | accuracy-weighted | gen | 91.75 | 90.27 | - vLLM version: v0.19.0 - vLLM main: vllm-project@6f786f2 --------- Signed-off-by: Sijie Fu <fusijie@huawei.com> Co-authored-by: Sijie Fu <fusijie@huawei.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.